An endoscopic vision autonomous control method, system and medium

Abstract

The present invention discloses a method, system, and medium for autonomously controlling the field of view of an endoscope. The method comprises: obtaining visual feedback from an endoscope camera and the actual length of a rope-driven manipulator; calculating image clarity based on the visual feedback, and segmenting and positioning the surgical instrument tip; establishing a rope-end kinematic model and position-level RCM constraint equations based on the actual rope length; establishing an optimization model for adjusting the surgical field of view based on the above-obtained information; calculating the desired speed of the endoscope camera using a multi-objective optimization-based intelligent adjustment method for the surgical field of view; establishing a speed-level RCM constraint equation and an end-end-rope inverse kinematic model, solving for the rope length, and thereby achieving motion control of a scope-holding robot. By using the present invention, rapid, accurate, and effective tracking of the instrument tip can be achieved, hand-eye coordination can be reduced, image clarity can be improved, and the safety and stability of intelligent field of view adjustment can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical auxiliary control, and in particular to a method, system and medium for autonomous control of the visual field of an endoscope. Background Art

[0002] With the increasing popularity of minimally invasive surgery and the application of robotic technology, research on laparoscopic surgical robots has become increasingly active. Unlike master-slave control systems, the collaborative operation of assistive mirror-holding robots and the primary surgeon is more reliable, and the development cost of assistive mirror-holding robots is far lower than that of master-slave control systems. In laparoscopic surgery, intelligent assistive mirror-holding robots offer advantages such as fast response, strong stability, and high precision, enabling them to assist surgeons in intelligently adjusting the surgical field of view. Intelligent assistive mirror-holding robots use visual feedback to rapidly locate the tip of surgical instruments, allowing them to autonomously adjust the surgical field of view and assist surgeons in completing surgical operations. These intelligent robots hold significant research significance and face significant market demand. However, traditional servo control methods, mostly based on master-slave laparoscopic robots, not only have a single optimization objective but also lack control of the endoscope's posture and the evaluation and optimization of visual feedback quality. Summary of the Invention

[0003] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to provide a method, system and medium for autonomous control of the endoscope field of view, which can achieve fast, accurate and effective tracking of the instrument tip, reduce hand-eye coordination, improve image clarity and the safety and stability of intelligent field of view adjustment.

[0004] To achieve the above object, the present invention provides a method for autonomously controlling the field of view of an endoscope, comprising:

[0005] Obtain visual feedback from the endoscope camera and the actual rope length of the rope-driven manipulator;

[0006] Calculate image clarity based on visual feedback, segment and locate the tip of the surgical instrument; establish a rope-end kinematic model and position-level RCM constraint equations based on the actual rope length;

[0007] Establish an optimization model for surgical field adjustment;

[0008] The expected speed of the endoscopic camera is calculated using an intelligent adjustment method of the surgical field of view based on multi-objective optimization;

[0009] The velocity-level RCM constraint equation and the terminal-rope inverse kinematics model are established, the rope length is solved, and the motion of the mirror-holding robot is controlled.

[0010] Furthermore, the segmenting and positioning of the surgical instrument tip includes:

[0011] (1) Extract ROI image;

[0012] (2) converting the ROI image into a grayscale image of the instrument tip;

[0013] (3) Perform median filtering on the grayscale image to obtain a filtered image;

[0014] (4) converting the filtered image into a binary image of the instrument tip;

[0015] (5) Extract the outline of the instrument tip in the binary image;

[0016] (6) Screening the outline of the instrument tip in the binary image;

[0017] (7) Calculate the center of gravity of the instrument tip;

[0018] (8) Calculate system tracking points.

[0019] Furthermore, the optimization model for adjusting the surgical field of view includes: a tracking model for the image features of the instrument tip, a hand-eye coordination model for directional errors, an optimization model for image clarity, and safety constraint models at the position and speed levels.

[0020] Furthermore, the tracking model of the instrument tip image feature is specifically:

[0021]

[0022] Where,

[0023]

[0024]

[0025]

[0026] Where t represents time, f u 、f v , u0 and v0 represent the internal parameters of the endoscope camera; u tips ,v tips Represent the pixel coordinates of the end of the instrument; K p is a constant diagonal matrix; s des ,s tips Represent the pixel coordinates of the desired tracking point and the actual instrument end tracking point respectively; K p,track and K d,track They are all normal numbers. The actual instrument end tracking point s at time t and time t-1 respectively tips and the expected tracking point s des The pixel distance between lap ) # For Jlap Pseudo-reversal.

[0027] Furthermore, the hand-eye coordination model of the direction error is specifically:

[0028]

[0029] Among them, β k (k=t,t-1) represents the direction error:

[0030] β=atan 2[- lap A0(2,2),- lap A0(1,2)] (6)

[0031] Where K p,coor and K d,coor They are all normal numbers. lap A0(2,2), lap A0(1,2) represents the matrix lap The elements in the second row and second column and the first row and second column of A0.

[0032] Furthermore, the optimization model of the image clarity is specifically as follows:

[0033]

[0034]

[0035] in, con A lap is the rotation matrix from the endoscope connector to the endoscope camera; are the image clarity at time t-1 and time t respectively; The endoscope camera tracks the linear velocity of the instrument tip in the z direction at time t-1; for The symbolic function has the property that when hour, when hour, when hour, Indicates the rate of change of image clarity; η0 represents the minimum tolerable change rate, K def is a positive constant, and g(η q ) also satisfies the following properties:

[0036] When η q When <0, g(η q )<0; when η q = 0, g(η q )=0; when η q >0, g(ηq )>0;

[0037] When η q ≠0, |η q The larger the | q ) is smaller.

[0038] Furthermore, the position-level and speed-level safety constraint models are specifically as follows:

[0039]

[0040]

[0041] in, Represents the actual insertion distance of the device at time t-1 and time t respectively; d in,d Indicates the desired insertion distance; K p,safe and K d,safe They are all normal numbers.

[0042] The present invention also provides an endoscope field of view autonomous control system, comprising: a feedback data receiving module, an image preprocessing module, a kinematics and RCM constraint modeling module, a surgical field of view intelligent adjustment module and a motion control module, wherein:

[0043] A feedback data receiving module is used to obtain the image captured by the endoscope camera and the actual rope length of the rope-driven manipulator;

[0044] Image preprocessing module, used to calculate image clarity and segment and locate the surgical instrument tip;

[0045] Kinematics and RCM constraint modeling module, used to establish the kinematic model from active rope space to operation space and position-level RCM constraint equations of the rope-driven flexible arm;

[0046] The intelligent surgical field adjustment module is used to establish an optimization model for surgical field adjustment and then calculate the expected speed of the endoscopic camera using an intelligent surgical field adjustment method based on multi-objective optimization;

[0047] The motion control module is used to establish the speed-level RCM constraint equations and the inverse kinematics model from the rope-driven flexible arm operation space to the active rope space, solve the rope length, and then realize the motion control of the mirror-holding robot.

[0048] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0049] The beneficial effects of the present invention are:

[0050] (1) The present invention establishes an optimization model for adjusting the surgical field of view, which mainly includes a tracking model that considers the image characteristics of the instrument tip, a hand-eye coordination model that considers directional errors, an optimization model that considers image clarity, and a safety constraint model at the position and speed levels. Among them:

[0051] The tracking model that considers the image features of the instrument tip can ensure that the instrument tip does not exceed the surgical field of view and maintain the system tracking point in the center of the surgical image.

[0052] The hand-eye coordination model considering directional error can optimize the hand-eye coordination problem and reduce the impact of directional error on the surgeon.

[0053] The optimization model that takes image clarity into consideration optimizes the quality of visual feedback by adjusting the depth of the endoscopic camera, thereby providing the surgeon with clearer visual feedback.

[0054] The position-level and speed-level safety constraint models can avoid collisions between the endoscope and the human body or instruments, and improve the safety of surgical field adjustment.

[0055] (2) The method of the present invention takes into account the optimization problems of multiple objectives including instrument tip tracking, hand-eye coordination, image clarity, and safety constraints. It can achieve fast, accurate, and effective tracking of the instrument tip, reduce hand-eye coordination, improve image clarity, and the safety and stability of intelligent field of view adjustment.

[0056] (3) The present invention establishes the RCM position-level and velocity-level equations of the mirror-holding robot, which decouples the motion of the end of the rope-driven flexible arm and the tracking motion of the endoscope camera.

[0057] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of a scenario of a robotic surgical-assisted operation according to the present invention.

[0059] Figure 2 It is a flow chart of the method of the present invention.

[0060] Figure 3 It is a system principle block diagram of the present invention.

[0061] Figure 4 This is the algorithm flow chart of the auxiliary mirror-holding robot system of the present invention.

[0062] Figure 5 It is the DH coordinate system diagram of the auxiliary mirror-holding robot system of the present invention.

[0063] Figure 6 1 is a flow chart of the segmentation and positioning of the instrument tip of the present invention.

[0064] Figure 7 It is a trajectory diagram of the instrument tip in the visual feedback of the present invention.

[0065] Figure 8 It is a tracking distance variation curve diagram of the present invention.

[0066] Figure 9 It is a direction error change curve diagram of the present invention.

[0067] Figure 10 It is a graph showing the change in image clarity of the present invention.

[0068] Figure 11 It is a curve diagram of the change of the endoscope insertion distance of the present invention. DETAILED DESCRIPTION

[0069] like Figure 1 As shown in the figure, the assisted endoscope-holding robot system primarily consists of a computer with control software, a tethered flexible arm, an endoscope connector, an endoscope, and a monitor. The computer, connected to the tethered flexible arm, the endoscope, and the monitor, serves as the control center for the entire assisted endoscope-holding robot system. The endoscope connector, mounted at the end of the tethered flexible arm, secures the endoscope. During laparoscopic surgery, the assisted endoscope-holding robot system adjusts the position of the endoscopic camera through active or passive control modes to provide the surgeon with accurate and stable visual feedback, assisting the surgeon in completing the surgical procedure.

[0070] like Figure 2 As shown, the present invention provides an endoscope field of view autonomous control method, comprising:

[0071] Obtain visual feedback from the endoscope camera and the actual rope length of the rope-driven manipulator;

[0072] Calculate image clarity based on visual feedback, segment and locate the tip of the surgical instrument; establish a rope-end kinematic model and position-level RCM constraint equations based on the actual rope length;

[0073] Establish an optimization model for surgical field adjustment;

[0074] The expected speed of the endoscopic camera is calculated using an intelligent adjustment method of the surgical field of view based on multi-objective optimization;

[0075] The velocity-level RCM constraint equation and the terminal-rope inverse kinematics model are established, the rope length is solved, and the motion of the mirror-holding robot is controlled.

[0076] like Figure 3As shown in the figure, the overall framework of the intelligent control system of the scope-holding robot mainly consists of a feedback data receiving module, an image preprocessing module, a kinematics and RCM constraint modeling module, an intelligent surgical field adjustment module, and a motion control module. The feedback data receiving module is used to obtain images captured by the endoscope camera and the actual rope length of the rope-driven manipulator arm. The image preprocessing module is used to calculate image clarity and segment and locate the tip of the surgical instrument. The kinematics and RCM constraint modeling module is used to establish the kinematic model from the active rope space to the operating space of the rope-driven flexible arm and the position-level RCM constraint equations. The intelligent surgical field adjustment module is used to establish an optimization model for surgical field adjustment and then calculate the desired speed of the endoscope camera using an intelligent surgical field adjustment method based on multi-objective optimization. The motion control module is used to establish the speed-level RCM constraint equations and the inverse kinematic model from the operating space to the active rope space of the rope-driven flexible arm, solve the rope length, and then realize the motion control of the scope-holding robot.

[0077] like Figure 4 As shown in Figure 1, the corresponding algorithm for the assisted scope-holding robot system primarily includes feedback data reception, image preprocessing, kinematic and RCM constraint modeling, intelligent surgical field adjustment, and motion control. Feedback data reception is the first step in the algorithm, primarily used to obtain images captured by the endoscope camera and the actual cable length of the tether-driven robotic arm. Next, image preprocessing and kinematic and RCM constraint modeling are implemented based on the acquired feedback data. Image preprocessing primarily involves calculating image clarity and segmenting and positioning the surgical instrument tip. Kinematic and RCM constraint modeling primarily involves establishing a kinematic model from the active cable space to the operating space of the tether-driven flexible arm and the position-level RCM constraint equations. Next, an optimization model for surgical field adjustment is established based on the information obtained. This model primarily includes a tracking model that considers the image characteristics of the instrument tip, a hand-eye coordination model that considers directional errors, an optimization model that considers image clarity, and safety constraint models at the position and velocity levels. Finally, a multi-objective optimization-based intelligent surgical field adjustment method is used to calculate the desired velocity of the endoscopic camera. Finally, the velocity-level RCM constraint equations and the inverse kinematics model from the rope-driven flexible arm operation space to the active rope space are established, and the rope length is solved to realize the motion control of the mirror-holding robot.

[0078] like Figure 5 As shown, assuming {f0}-{f 2m} represents the DH coordinate system of the rope-driven flexible arm (where {f0} is the base coordinate system), {a i ,d i |i=1,2,...,2m} represents the DH parameter. The endoscope connector is installed at the end of the rope-driven flexible arm, which is used to fix the endoscope. During laparoscopic surgery, the endoscope axis always passes through a fixed point. 0 p RCM Assume {fcon} represents the endoscope connector coordinate system, {f RCM} represents the RCM coordinate system, {f lap} represents the endoscope camera coordinate system. Assume that the axial length of the endoscope connector is represented by d con , the axial length of the endoscope is denoted as d lap The length of the endoscope inserted into the abdominal cavity is d in =|| 0 p RCM - 0 p c ||.

[0079] Assume l=[l1,l2,…,l 3m ] T and θ=[θ1,θ2,…,θ 2m ] T represents the rope length and joint angle of the 2m-DOF rope-driven flexible arm, J L represents the Jacobian matrix from joint space to active rope space, J e Represents the Jacobian matrix from joint space to action space.

[0080] Establish the DH coordinate system of the mirror-holding robot, and the homogeneous transformation matrix of the adjacent coordinate systems is as follows:

[0081]

[0082] Where cθ i represents cosθ i , sθ i represents sinθ i , i=1,2,...,2m.

[0083] According to the chain rule, the kinematic model of the mirror-holding robot is established, namely:

[0084]

[0085] 0 T 2m = 0 T1 1 T2… 2m-1 T 2m =fkine(θ) (3)

[0086] in, It's J L Pseudo-reversal.

[0087] The velocity-level inverse kinematics numerical model of the mirror-holding robot can be defined as:

[0088]

[0089] in, Indicates J e Pseudo-reversal.

[0090] Taking {f0} as the reference coordinate system, {f con} and {f lap The homogeneous transformation matrix of} can be described as:

[0091]

[0092]

[0093] in, 0 A con and 0 A lap Respectively represent {f com} and {f lap}'s rotation matrix; 0 p con and 0 p lap Respectively represent {f com} and {f lap}'s position vector; m T con and con T lap They represent the homogeneous transformation matrices from the end of the mirror-holding robotic arm to the endoscope connector and from the endoscope connector to the endoscope camera, respectively.

[0094] Assume η in =d in / d lap represents the ratio between the insertion distance and the axial length of the endoscope. Since the endoscope must always pass through the RCM, the constraints of the position-level RCM can be expressed as:

[0095]

[0096] For the convenience of expression, {f con} as the reference coordinate system, assuming that the linear velocity and angular velocity of the endoscope connector are con v con =[ con v conx con v cony con v conz ] T and con ω con =[ con ω conα con ω conβ con ωconγ ] T ; The linear velocity and angular velocity of the endoscope at RCM are con v RCM =[ con v RCMx con v RCMy con v RCMz ] T and con ω RCM =[ con ω RCMα con ω RCMβ con ω RCMγ ] T ; The linear velocity and angular velocity of the endoscope camera are con v lap =[ con v lapx con v lapy con v lapz ] T and con ω c =[ con ω lapα con ω lapβ con ω lapγ ] T .

[0097] {f con}、{f RCM} and {f lap} are defined on the endoscope, according to theoretical mechanics:

[0098]

[0099] According to the characteristics of RCM constraints, the endoscope cannot achieve translation perpendicular to the axis, and loses two degrees of freedom. Therefore, the velocity of the endoscope perpendicular to the axis in the velocity level RCM constraint is always equal to 0, that is:

[0100]

[0101] According to the law of speed level RCM constraint, the relationship between the speed of the endoscope camera and the speed of the endoscope connector can be established as follows:

[0102]

[0103] The positioning of the surgical instrument tip tracking point provides effective and important feedback for assisting the intelligent field of view adjustment of the scope-holding robot. To accurately extract the system's tracking points, a deep learning method is used to segment and locate the surgical instrument tip, which can reduce the positioning error of the instrument tip. First, a real-time object detection model is used to extract the ROI region (the instrument tip area) for visual feedback. Then, the instrument tip is segmented within the ROI region, and the center of gravity of the instrument tip is extracted. Finally, the weighted average of the centers of gravity of all instrument tips is calculated as the system's tracking point.

[0104] The flow chart of the segmentation and positioning of the instrument tip is as follows Figure 6 As shown, it mainly includes the following steps: (1) in I S Cut out the ROI image of the kth instrument tip (2) Converted to a grayscale image of the instrument tip (3) According to ksize Perform median filtering to obtain the filtered image (4) According to the threshold [thresh min ,thresh max ]Will Converted to a binary image of the instrument tip (5) Extraction Q contours in ; (6) Calculate the qth (q=1,2,…,Q) contour q area q , further calculate the contour q exist The area ratio of When the ratio is met min <η area <ratio max When the contour q is the contour of the kth instrument tip; (7) according to contour q Calculate the center of gravity of the kth instrument tip

[0105] The center of gravity of the instrument tip segmented region in the image is calculated using the first-order moment. The (i+j)-order moment m of the instrument tip contour in the k-th ROI image is ij Expressed as:

[0106]

[0107] Then, the center of gravity of the kth instrument tip can be calculated for:

[0108]

[0109] Therefore, a system tracepoint can be expressed as:

[0110]

[0111] Among them, η k represents the tracking weight of the kth instrument tip center of gravity (i.e., the importance of this type of instrument), and

[0112] The optimization model for surgical field adjustment mentioned above includes: a tracking model for the image features of the instrument tip, a hand-eye coordination model for directional error, an optimization model for image clarity, and safety constraint models at the position and velocity levels. The details are as follows:

[0113] (1) Tracking model of instrument tip image features

[0114] The center of gravity of each instrument tip is extracted by the above-mentioned tip segmentation and positioning method, and the system tracking point s is further calculated. tips Assume that des =[w s / 2h s / 2] T Represents the center of the image, the system tracks point s tips and image center s des The pixel distance between them is d px =||s tips -s des ||.

[0115] The endoscope camera tracks the tip of the instrument with a minimum pixel distance d px The expected pixel distance is 0. Therefore, the following tracking model can be established:

[0116]

[0117] Where,

[0118]

[0119]

[0120]

[0121] Where t represents time, f u 、f v , u0 and v0 represent the internal parameters of the endoscope camera; u tips ,v tips Represent the pixel coordinates of the end of the instrument; K p is a constant diagonal matrix; s des,s tips Represent the pixel coordinates of the desired tracking point and the actual instrument end tracking point respectively; K p,track and K d,track They are all normal numbers. The actual instrument end tracking point s at time t and time t-1 respectively tips and the expected tracking point s des The pixel distance between lap ) # For J lap Pseudo-reversal.

[0122] (2) Hand-eye coordination model of directional error

[0123] Due to the characteristics of RCM constraints, the linear and angular velocities of the endoscope camera are coupled. Therefore, when the endoscope camera tracks the tip, the coupled rotational motion can cause directional errors. This means that the tip's direction of motion as seen in the visual feedback deviates from the surgeon's intended direction. This can affect the surgeon's hand-eye coordination, reducing comfort and surgical efficiency.

[0124] Assume β k (k=t,t-1) represents the direction error, that is:

[0125] β k =atan 2[- lap A0(2,2),- lap A0(1,2)] (19)

[0126] Similarly, improving hand-eye coordination is actually a matter of minimizing |β k |, the desired direction error is 0. Therefore, the following hand-eye coordination model can be established:

[0127]

[0128] Where K p,coor and K d,coor They are all normal numbers. lap A0(2,2), lap A0(1,2) represents the matrix lap The elements in the second row and second column and the first row and second column of A0.

[0129] (3) Image clarity optimization model

[0130] Image clarity evaluation metrics based on image edge texture information are highly sensitive, unimodal, and monotonic on both sides of the peak. Image gradient operators are best suited to reflect image edge texture information. Therefore, a clarity evaluation metric without a reference image is used to assess the quality of surgical images.

[0131] Assume q I Indicates image clarity (q I The larger the value, the clearer the image). The calculation formula is:

[0132] q I =ID(I s ) (twenty one)

[0133] Where ID(·) represents the no-reference image clarity evaluation method based on the gradient operator.

[0134] Assume d obj Indicates the object distance. According to the camera imaging principle, there is an object distance d obj,foc This makes the endoscope camera focus the best and the image the clearest. Therefore, in the same scene, the functional relationship between image clarity and object distance is q I (d obj ) has the following properties:

[0135] (1) When d obj <d obj,foc or d obj >d obj,foc When q I (d obj )<q I (d obj,foc );

[0136] (2) When d obj,1 <d obj,2 <d obj,foc hour, When d obj,2 >d obj,1 >d obj,foc hour,

[0137] The key to adjusting image clarity is to assist the scope-holding robot to intelligently adjust the distance between the endoscope camera and the surgical instruments and the human body. In order to maximize image clarity, according to the above image clarity q I (d obj ) and object distance d obj The nonlinear functional relationship between them can be used to establish the following optimization model:

[0138]

[0139]

[0140] in, con A lap is the rotation matrix from the endoscope connector to the endoscope camera; are the image clarity at time t-1 and time t respectively; The endoscope camera tracks the linear velocity of the instrument tip in the z direction at time t-1; for The symbolic function has the property that when hour, when hour, when hour, Indicates the rate of change of image clarity; η0 represents the minimum tolerable change rate, K def is a positive constant, and g (η q ) also satisfies the following properties:

[0141] When η q When <0, g(η q )<0; when η q = 0, g(η q )=0; when η q >0, g(η q )>0;

[0142] When η q ≠0, |η q The larger the | q ) is smaller.

[0143] (4) Safety constraint model at position level and speed level

[0144] In order to ensure the safety of the auxiliary endoscope holding robot, the position level and speed level surgical field adjustment safety constraints are set. At the position level, in order to avoid the endoscope being inserted too short or too long and causing collision with the human body and instruments, the insertion distance d of the endoscope is set to 100mm. in Constraints are specifically stated as follows:

[0145] (1) Define a placement distance safety zone d in ∈[d safe,min ,d safe,max ]. When the endoscope is inserted at a distance d in When in the safe zone, the movement of the endoscope is ensured to be safe, so there is no need to in Take control.

[0146] (2) Define two distance warning zones d in ∈[d alart,min ,d safe,min ] and d in ∈[d safe,max ,d alart,max ]. For safety reasons, when the endoscope is placed at a distance d in When in the warning zone, you need toin Control it to enter the safe zone. Therefore, the following safety constraint model can be established:

[0147]

[0148]

[0149] in, Represents the actual insertion distance of the device at time t-1 and time t respectively; d in,d Indicates the desired insertion distance; K p,safe and K d,safe They are all normal numbers.

[0150] At the speed level, in order to avoid the endoscope camera moving too fast and causing loss of control and collision with the human body and equipment, the maximum linear speed of the endoscope camera is constrained. max and the maximum axial angular velocity ω of the endoscope max .

[0151] The generalized velocity of the endoscope camera can be calculated based on multi-objective optimization of instrument tip tracking, hand-eye coordination, image clarity, and safety constraints. for:

[0152]

[0153] Among them, η track ,η coor ,η def and η safe Represent the weight coefficients of the three optimization objectives and location-level safety constraints respectively.

[0154] In addition, when the surgeon operates on a key part, the visual feedback will capture the vibration of the instrument tip in the local area, causing the auxiliary mirror-holding robot to produce a small movement. In order to prevent the above situation from causing the vibration of the surgical field, a s des As the center, with R dead When the tracking point enters the dead zone, the auxiliary endoscope robot stops moving, providing stable visual feedback to the surgeon, unless the insertion distance of the endoscope enters the warning zone.

[0155] To verify the correctness and effectiveness of the proposed method, two sets of instrument tip tracking simulation experiments were designed (i.e., one instrument, two instruments, and three instruments). To verify the anti-jitter capability of the proposed method, it was assumed that the jitter error of the instrument trajectory conforms to a uniform distribution of ±1mm.

[0156] For the case of one instrument, the tip of the surgical instrument moves along a circular trajectory. For the case of two instruments, the tip of the surgical instrument moves along a rectangular trajectory and a circular trajectory respectively. For the case of three instruments, the tip of the surgical instrument moves along a fixed point, a rectangular trajectory, and a circular trajectory respectively. Based on the initial state and the simulation results, the trajectory of the instrument tip in the visual feedback is as follows Figure 7 As shown in Figure 2, in the initial state, when the endoscope camera does not track the tip of the surgical instrument, the tip of the surgical instrument will exceed the surgical field of view. The tracking distance change curve is shown in Figure 2. Figure 8 As shown, the direction error change curve is as follows Figure 9 As shown, the image clarity change curve is as follows Figure 10 As shown in the figure, the endoscope insertion distance change curve is as follows Figure 11 shown.

[0157] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An endoscope field of view autonomous control system, characterized in that: include: Feedback data receiving module, image preprocessing module, kinematics and RCM constraint modeling module, surgical field of view intelligent adjustment module and motion control module, including: A feedback data receiving module is used to obtain the image captured by the endoscope camera and the actual rope length of the rope-driven flexible arm; Image preprocessing module, used to calculate image clarity and segment and locate the surgical instrument tip; Kinematics and RCM constraint modeling module, used to establish the kinematic model from active rope space to operation space and position-level RCM constraint equations of the rope-driven flexible arm; The intelligent surgical field adjustment module is used to establish an optimization model for surgical field adjustment and then calculate the expected speed of the endoscopic camera using an intelligent surgical field adjustment method based on multi-objective optimization; The motion control module is used to establish the velocity-level RCM constraint equations and the inverse kinematics model from the rope-driven flexible arm operation space to the active rope space, and solve the rope length to realize the motion control of the mirror-holding robot; The optimization model for surgical field adjustment established includes: a tracking model for the image features of the instrument tip, a hand-eye coordination model for directional errors, an optimization model for image clarity, and safety constraint models at the position and speed levels. The tracking model of the instrument tip image feature is specifically: Where, Where t represents time, f u 、f v , u0 and v0 represent the internal parameters of the endoscope camera; u tips 、v tips Represent the pixel coordinates of the instrument tip; K p is a constant diagonal matrix; s des 、s tips represent the pixel coordinates of the desired tracking point and the actual instrument tip tracking point respectively; K p,track and K d,track They are all normal numbers. The actual instrument tip tracking point s at time t and time t-1 respectively tips and the expected tracking point s des The pixel distance between lap ) # For J lap of pseudo-rebellion; The hand-eye coordination model of the direction error is specifically: Among them, β k (k=t,t-1) represents the direction error: β k =atan2[- lap A0(2,2),- lap A0(1,2)] (6) Where K p,coor and K d,coor They are all normal numbers. lap A0(2,2), lap A0(1,2) represents the matrix lap The elements in the second row and second column and the first row and second column of A0; The optimization model of image clarity is specifically: in, con A lap is the rotation matrix from the endoscope connector to the endoscope camera; are the image clarity at time t-1 and time t respectively; is the linear velocity of the endoscope camera tracking the tip of the instrument in the z direction at time t-1; for The symbolic function has the property that when hour, when hour, when hour, Indicates the rate of change of image clarity; η0 represents the minimum tolerable change rate, K def is a positive constant, and g(η q ) also satisfies the following properties: When q <0, g(η q )<0;applicable q When =0, g(η q )=0;when q When >0, g(η q )>0; When η q ≠0, |η q The larger the | q ) is smaller; The safety constraint models at the position level and speed level are specifically as follows: in, Represents the actual insertion distance of the device at time t-1 and time t respectively; d in,d Indicates the desired insertion distance; K p,safe and K d,safe They are all normal numbers.

2. The endoscope field of view autonomous control system according to claim 1, characterized in that: Segmentation and positioning of surgical instrument tips include: (1) Extract ROI image; (2) converting the ROI image into a grayscale image of the instrument tip; (3) Perform median filtering on the grayscale image to obtain a filtered image; (4) converting the filtered image into a binary image of the instrument tip; (5) extracting the outline of the instrument tip from the binary image; (6) screening the outline of the instrument tip in the binary image; (7) Calculate the center of gravity of the instrument tip; (8) Calculate system tracking points.

Images