Optimal visual servoing based ureteroscopy robot system and method for kidney stone automatic tracking

Abstract

The application discloses a kidney stone automatic tracking system and method of a ureter flexible mirror surgery robot based on optimal visual servoing, and the system comprises a target identification module, a feature extraction module, a pose solving module and a pose adjusting module; the target identification module is used for acquiring image data of a ureter cavity and a kidney stone, and processing the image data to obtain a stone target region; the feature extraction module is used for performing feature extraction on the stone target region to obtain actual image features and an actual interaction matrix of an actual image pose, and is also used for calculating expected image features and an expected interaction matrix of an expected image pose; the pose solving module is used for calculating an optimal solution of a motion pose of each joint of the robot based on the actual image features, the actual interaction matrix, the expected image features and the expected interaction matrix by using an optimal control algorithm and a control barrier function; and the pose adjusting module is used for controlling the pose of the robot based on the optimal solution of the motion pose.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical robots, and particularly relates to a kidney stone automatic tracking system and method of a ureteroscopy surgery robot based on optimal visual servoing. BACKGROUND

[0002] In recent years, the number of kidney stone patients is increasing. The ureteroscopy surgery robot applied to kidney stones is gradually applied to surgery. However, endoscopic surgery often requires the surgeon to pay close attention, and frequent surgical operations can easily cause mental fatigue, thereby reducing the accuracy of the operation. Even the observation target may be lost during the operation, causing the operation to be delayed or failed.

[0003] The ureteroscopy surgery robot can be applied to ureteroscopy lithotripsy. The kidney stone automatic tracking method and system applied to the ureteroscopy surgery robot can achieve the function of automatically tracking kidney stone lesions, improve the success rate of kidney stone crushing surgery and the efficiency of the operator. This new technology should be able to reduce the energy consumed by the doctor during the ureteroscopy examination, while maintaining reliability in various complex situations. In addition, the technology also designs a visibility constraint for the kidney stone target, thereby reducing the risk of kidney stone target loss in the endoscopic field of view during the operation, reducing the operation risk.

[0004] For ureteroscopy surgery, the following defects exist. First, the traditional ureteroscopy surgery has a great requirement for the doctor's attention, and frequent surgery can easily cause the doctor to feel tired, which endangers the safety and efficiency of the operation. Second, the automatic tracking of kidney stones by general ureter robots requires high-integrity images, and the cavity environment often cannot provide good image information. Finally, due to the existence of disturbance, the ureteroscopy robot may lose the kidney stone target out of the camera field of view when tracking the kidney stone. SUMMARY

[0005] The application aims to solve the problems of the prior art, and provides the following scheme:

[0006] The kidney stone automatic tracking system of the ureteroscopy surgery robot based on optimal visual servoing comprises a target recognition module, a feature extraction module, a pose solving module and a pose adjustment module.

[0007] The target recognition module is used to obtain image data of the ureteral cavity and kidney stones, and process the image data to obtain a stone target area.

[0008] The feature extraction module is configured to perform feature extraction on the stone target region to obtain actual image features and an actual interaction matrix of an actual image pose, and is further configured to calculate expected image features and an expected interaction matrix of an expected image pose.

[0009] The pose solving module is configured to calculate an optimal solution of a motion pose of each joint of the robot based on the actual image features, the actual interaction matrix, the expected image features and the expected interaction matrix, by using an optimal control algorithm and a control barrier function.

[0010] The pose adjusting module is configured to control the pose of the robot based on the optimal solution of the motion pose.

[0011] Preferably, the target recognition module comprises an image acquisition unit and an image processing unit.

[0012] The image acquisition unit is configured to acquire the image data of the ureteral tract and the kidney stone.

[0013] The image processing unit is configured to recognize the contour of the kidney stone in the image data by using a neural network algorithm, perform binaryzation processing on the image data by using an adaptive threshold method to obtain a binary image, mark the contour in the binary image, and obtain the stone target region.

[0014] Preferably, the feature extraction module comprises an actual image extraction unit and an expected image extraction unit.

[0015] The actual image extraction unit is configured to perform feature extraction on the stone target region after binaryzation to obtain the actual image features and the actual interaction matrix of the actual image pose.

[0016] s1 = [x n1 y n1 a n1 s x1 s y1 α1] T ,

[0017]

[0018] wherein s1 represents actual image features, L1 represents an actual interaction matrix, x n1 represents actual image features for controlling X-direction translational freedom, y n1 represents actual image features for controlling Y-direction translational freedom, a n1 represents actual image features for controlling Z-direction translational freedom, s x1 represents actual features for rotation around the X-axis, and s y1represents the actual feature of rotation around the Y axis, a1 represents the actual orientation angle of the target region, and T represents the transpose of a matrix;

[0019] The desired image extraction unit calculates the desired image feature and the desired interaction matrix of the desired image pose based on a preset desired image:

[0020] s0 = [x n0 y n0 a n0 s x0 s y0 α0] T ,

[0021]

[0022] wherein s0 represents the desired image feature, L0 represents the desired interaction matrix, x n0 represents the desired image feature of the control X direction translation degree of freedom, y n0 represents the desired image feature of the control Y direction translation degree of freedom, a n0 represents the desired image feature of the control Z direction translation degree of freedom, s x0 represents the desired feature of rotation around the X axis, s y0 represents the desired feature of rotation around the Y axis, and a0 represents the desired orientation angle of the target region.

[0023] Preferably, the pose solving module comprises a camera desired speed calculation unit, an obstacle function setting unit and a robot pose calculation unit;

[0024] The camera desired speed calculation unit calculates the camera desired speed based on the actual image feature, the actual interaction matrix, the desired image feature and the desired interaction matrix:

[0025]

[0026] e = s1 - s0,

[0027]

[0028] wherein v d represents the camera desired speed, λ represents a constant, e represents the image feature error, L s represents the interaction matrix, represents the inverse matrix of the interaction matrix;

[0029] The obstacle function setting unit is used for setting the obstacle function:

[0030]

[0031] wherein B represents the obstacle function, represents the image feature difference that controls the translational freedom in the X direction, represents the image feature difference that controls the Y-direction translation freedom, and ρ represents the visibility constraint radius;

[0032] The robot posture calculation unit is used to calculate the optimal solution of the robot's motion posture based on the optimal control algorithm and in combination with the obstacle function:

[0033] B (1) ≥-k b B,

[0034] min M=||v d -v c || 2 ,

[0035] stv c =J robot Δq,

[0036]

[0037] Among them, B (1) represents the first-order derivative of the barrier function, k b represents the proportional constant, J robot represents the Jacobian matrix of the robot, v c represents the actual speed of the camera, Δq represents the optimal solution of the motion posture, represents the rotation speed of the robot, represents the bending speed of the robot, Indicates the feed rate of the robot.

[0038] The present invention also provides a method for automatically tracking kidney stones using a ureteroscopic surgical robot based on optimal visual servoing. The method is applied to any of the above-mentioned systems and includes the following steps:

[0039] Acquiring image data of the ureteral cavity and kidney stones, and processing the image data to obtain a target area of ​​the stone;

[0040] Performing feature extraction on the target stone area to obtain actual image features and an actual interaction matrix of the actual image posture, which are also used to calculate expected image features and an expected interaction matrix of the expected image posture;

[0041] Calculating an optimal solution for the motion posture of each joint of the robot using an optimal control algorithm and a control obstacle function based on the actual image features, the actual interaction matrix, the expected image features, and the expected interaction matrix;

[0042] The posture of the robot is controlled based on the optimal solution of the motion posture.

[0043] Preferably, the method for obtaining the stone target region comprises:

[0044] obtaining the image data of the ureteral cavity and the kidney stone;

[0045] recognizing the contour of the kidney stone in the image data by using a neural network algorithm, performing binaryzation processing on the image data by using an adaptive threshold method to obtain a binaryzation image, and then marking the contour in the binaryzation image to obtain the stone target region.

[0046] Preferably, the method for performing the feature extraction comprises:

[0047] performing feature extraction on the stone target region after binaryzation to obtain the actual image feature of the actual image pose and the actual interaction matrix:

[0048] s1=[x n1 y n1 a n1 s x1 s y1 α1] T ,

[0049]

[0050] wherein s1 represents the actual image feature, L1 represents the actual interaction matrix, x n1 represents the actual image feature for controlling the X-direction translational degree of freedom, y n1 represents the actual image feature for controlling the Y-direction translational degree of freedom, a n1 represents the actual image feature for controlling the Z-direction translational degree of freedom, s x1 represents the actual feature for rotating around the X-axis, s y1 represents the actual feature for rotating around the Y-axis, and α1 represents the actual direction angle of the target region, and T represents the transpose of a matrix.

[0051] based on a preset expected image, calculating the expected image feature of the expected image pose and the expected interaction matrix:

[0052] s0=[x n0 y n0 a n0 s x0 s y0 α0] T ,

[0053]

[0054] wherein s0 represents the expected image feature, L0 represents the expected interaction matrix, x n0represents a desired image feature for controlling X-direction translational freedom, y n0 represents a desired image feature for controlling Y-direction translational freedom, a n0 represents a desired image feature for controlling Z-direction translational freedom, s x0 represents a desired feature for rotation around X-axis, s y0 represents a desired feature for rotation around Y-axis, a0represents a desired orientation angle of the target region.

[0055] Preferably, the method for obtaining the motion posture optimal solution comprises:

[0056] The camera desired velocity calculation unit calculates a camera desired velocity based on the actual image feature, the actual interaction matrix, the desired image feature and the desired interaction matrix:

[0057]

[0058] e = s1- s0,

[0059]

[0060] wherein v d represents a camera desired velocity, λ represents a constant, e represents an image feature error, L s represents an interaction matrix, represents an inverse matrix of the interaction matrix;

[0061] The barrier function setting unit is configured to set a barrier function:

[0062]

[0063] wherein B represents a barrier function, represents an image feature difference for controlling X-direction translational freedom, represents an image feature difference for controlling Y-direction translational freedom, ρ represents a visibility constraint radius;

[0064] The robot posture calculation unit is configured to calculate the motion posture optimal solution of the robot based on the optimal control algorithm and in combination with the barrier function:

[0065] B (1) ≥ -k b B,

[0066] min M = ||v d -v c || 2 ,

[0067] s.t.v c = J robot Δq,

[0068]

[0069] wherein, B (1) represents the first derivative of the barrier function, k b represents a proportional constant, J robot represents the Jacobian matrix of the robot, v c represents the actual speed of the camera, Δq represents the optimal solution of the motion posture, represents the rotational speed of the robot, represents the bending speed of the robot, represents the feed speed of the robot.

[0070] Compared with the prior art, the beneficial effects of the present application are:

[0071] The technical scheme of the present application innovatively proposes to calculate the moment features and the interaction matrix of the kidney stone image, compare the expected image with the actual image, and solve the optimal joint speed by using a nonlinear solver, so as to realize the automatic tracking function of the ureteroscopy robot for kidney stones. When solving the optimal joint speed, the visibility constraint based on the control barrier function is set to ensure that the kidney stone target is always kept in the field of view of the endoscope, preventing the loss of the target. The present application can realize the control of the ureteroscopy robot under the condition of incomplete image, reduce the risk of losing the tracking target, and improve the safety and efficiency of the ureteroscopy robot in use. BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical scheme of the present application, the following briefly introduces the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0073] Figure 1 The system structure schematic diagram of the embodiment of the present application is shown in the figure.

[0074] Figure 2 The method flowchart of the embodiment of the present application is shown in the figure.

[0075] Figure 3 The method flowchart of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present application will be described clearly and completely below with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0077] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with the accompanying drawings and specific embodiments.

[0078] Embodiment one

[0079] In this embodiment, as shown in the figure, Figure 1 The kidney stone automatic tracking system of the ureter flexible mirror surgery robot based on optimal visual servoing includes a target identification module, a feature extraction module, a pose solving module and a pose adjusting module.

[0080] The target identification module is used to acquire image data of the ureteral cavity and kidney stones, and process the image data to obtain a stone target region.

[0081] The target identification module includes an image acquisition unit and an image processing unit. The image acquisition unit is used to acquire image data of the ureteral cavity and kidney stones; the image processing unit identifies the contour of the kidney stones in the image data by using a neural network algorithm, and adopts an adaptive threshold method to perform binaryzation processing on the image data to obtain a binary image, and then marks the contour in the binary image to obtain the stone target region.

[0082] The feature extraction module is used to extract features of the stone target region to obtain actual image features and an actual interaction matrix of an actual image pose, and is also used to calculate expected image features and an expected interaction matrix of an expected image pose.

[0083] The feature extraction module includes an actual image extraction unit and an expected image extraction unit. The actual image extraction unit is used to extract features of the binaryzation stone target region to obtain actual image features and an actual interaction matrix of an actual image pose:

[0084] s1=[x n1 y n1 a n1 s x1 s y1 α1] T ,

[0085]

[0086] wherein s1 represents actual image features, L1 represents an actual interaction matrix, xn1 actual image feature representing the translational freedom degree in the X direction, y n1 actual image feature representing the translational freedom degree in the Y direction, a n1 actual image feature representing the translational freedom degree in the Z direction, s x1 actual feature representing the rotation around the X axis, s y1 actual feature representing the rotation around the Y axis, a1 represents the actual orientation angle of the target region, T represents the transpose of a matrix; the expected image extraction unit calculates the expected image feature and the expected interaction matrix of the expected image pose based on the preset expected image:

[0087] s0 = [x n0 y n0 a n0 s x0 s y0 a0] T ,

[0088]

[0089] wherein s0 represents the expected image feature, L0 represents the expected interaction matrix, x n0 expected image feature representing the translational freedom degree in the X direction, y n0 expected image feature representing the translational freedom degree in the Y direction, a n0 expected image feature representing the translational freedom degree in the Z direction, s x0 expected feature representing the rotation around the X axis, s y0 expected feature representing the rotation around the Y axis, a0 represents the expected orientation angle of the target region.

[0090] In the embodiment, the specific process of calculating the actual image feature and the expected image feature is as follows: first, the actual image feature and the expected image feature are both represented as a feature s:

[0091] s = [x n y n a n s x s y a] T ;

[0092] wherein x n image feature representing the translational freedom degree in the X direction, y n image feature representing the translational freedom degree in the Y direction, a n image feature representing the translational freedom degree in the Z direction, s x feature representing the rotation around the X axis, s ycharacteristic representing rotation around Y axis, and a represents the direction angle of the target region; then, the center of moment (x g , y g ) of the image region is calculated:

[0093] x g = m 10 / m 00 ,

[0094] y g = m 01 / m 00 ,

[0095] wherein m 00 is the area of the target region, and the image moment m ij and the center moment μ ij are respectively defined as follows:

[0096] m ij =∫∫ o x i y j f(x, y)dxdy,

[0097] μ ij =∫∫ o (x-x g ) i (y-y g ) j f(x, y)dxdy,

[0098]

[0099] wherein i and j represent natural numbers; f() represents the gray scale of a two-dimensional graph, x represents the horizontal coordinate of a pixel in the two-dimensional graph, and y represents the vertical coordinate of a pixel in the two-dimensional graph; in order to reduce the coupling of motion, the characteristic is normalized, and the processing mode is as follows:

[0100]

[0101] x n = a n x g ,

[0102] y n = a n y g ,

[0103] wherein z* represents the expected depth, a represents the image area of the actual image region, a* represents the image area of the expected image region, and the area calculation formula of the image area of the actual image region is as follows:

[0104] a = m 00 =∫∫ of(x, y)dxdy,

[0105] The formula for calculating the direction angle a of the target object is:

[0106]

[0107] The formula for calculating the feature rotating around the X axis and the feature rotating around the Y axis is:

[0108] s x = (c2c3 + s2s3) / K,

[0109] s y = (s2c3 - c2s3) / K,

[0110] In the formula, there are:

[0111]

[0112] In this embodiment, the specific process of calculating the actual interaction matrix and the expected interaction matrix is as follows: first, the actual interaction matrix and the expected interaction matrix are both expressed as matrix L:

[0113]

[0114] In the formula, there are:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133] n ij = μ ij / m 00 ,

[0134] The interaction matrix in the above equation and is:

[0135]

[0136]

[0137] where:

[0138] m vx = -i(Am ij +Bm i-1，j+1 +Cm i-1，j )-Am ij ,

[0139] m vy = -j(Am i+1，j +Bm ij +Cm i，j-1 )-Bm ij ,

[0140] m vz = (i+j+3)(Am i+1，j +Bm i，j+1 +Cm ij )-Cm ij ,

[0141] m wx = (i+j+3)m i，j+1 +jm i，j-1 ,

[0142] m wy = -(i+j+3)m i+1，j -im i-1，j ,

[0143] m wz = im i-1，j+1 -jm i+1，j-1 ,

[0144] μ vx = -(i+1)Aμ ij -iBμ i-1，j+1 ,

[0145] μ vy = -jAμ i+1，j-1 -(j+1)Bμ ij ,

[0146]

[0147]

[0148] μ wz = iμ i-1，j+1 -jμ i+1，j-1 ,

[0149] wherein A, B and C are constants.

[0150] The pose solving module is configured to calculate an optimal solution of a motion pose of each joint of the robot based on actual image features, an actual interaction matrix, expected image features and an expected interaction matrix by using an optimal control algorithm and a control barrier function.

[0151] The pose solving module comprises a camera expected speed calculation unit, a barrier function setting unit and a robot pose calculation unit. The camera expected speed calculation unit calculates a camera expected speed based on actual image features, an actual interaction matrix, expected image features and an expected interaction matrix:

[0152] v d = [v dx v dy v dz ω dx ω dy ω dz ] T ,

[0153]

[0154] e = S1 - S0,

[0155]

[0156] wherein v d represents the camera expected speed, λ represents a constant, e represents an image feature error, L s represents an interaction matrix, represents an inverse matrix of the interaction matrix; the barrier function setting unit is configured to set a barrier function:

[0157]

[0158]

[0159]

[0160] wherein B represents a barrier function, represents an image feature difference for controlling an X-direction translation degree of freedom, represents an image feature difference for controlling a Y-direction translation degree of freedom, and p represents a visibility constraint radius; the robot pose calculation unit is configured to calculate an optimal solution of a motion pose of the robot based on an optimal control algorithm in combination with the barrier function:

[0161] B (1) ≥-k b B,

[0162] min M = ||v d -v c || 2 ,

[0163] s.t.v c = J robot Δq,

[0164]

[0165]

[0166]

[0167]

[0168] wherein B (1) represents a first-order derivative of the barrier function, k b represents a proportional constant, j robot represents a Jacobian matrix of the robot, v c represents an actual speed of the camera, and Δq represents the optimal solution of the motion pose, represents a rotational speed of the robot, represents a bending speed of the robot, represents a feeding speed of the robot, and l represents a length of a flexible segment, represents an angle of rotation, and θ represents an angle of bending.

[0169] The pose adjustment module is configured to control a pose of the robot based on the optimal solution of the motion pose.

[0170] In this embodiment, the calculated rotational speed, bending speed and feeding speed of the ureter robot are converted to obtain a motion speed control amount required by a motor, and the control amount is sent to the ureter flexible mirror surgery robot to drive the motor, so as to adjust the pose of the ureter flexible mirror end.

[0171] Embodiment Two

[0172] In this embodiment, as shown in Figure 2 、 Figure 3 the kidney stone automatic tracking method of the ureter flexible mirror surgery robot based on optimal visual servoing includes the following steps:

[0173] S1. Obtain image data of the ureteral cavity and kidney stones, and process the image data to obtain a stone target region.

[0174] The method for obtaining the stone target region includes: obtaining image data of the ureteral cavity and kidney stones; using a neural network algorithm to identify the contour of the kidney stones in the image data, and using an adaptive threshold method to perform binaryzation processing on the image data to obtain a binaryzation image, then marking the contour in the binaryzation image to obtain the stone target region.

[0175] S2. Feature extraction is performed on the stone target region to obtain actual image features and actual interaction matrices of the actual image pose, which are also used to calculate expected image features and expected interaction matrices of the expected image pose.

[0176] The method for performing feature extraction includes: performing feature extraction on the binaryzation stone target region to obtain actual image features and actual interaction matrices of the actual image pose:

[0177] s1=[x n1 y n1 a n1 s x1 s y1 α1] T ,

[0178]

[0179] wherein s1 represents the actual image features, L1 represents the actual interaction matrices, x n1 represents the actual image features controlling the X-direction translational degree of freedom, y n1 represents the actual image features controlling the Y-direction translational degree of freedom, a n1 represents the actual image features controlling the Z-direction translational degree of freedom, s x1 represents the actual features rotating around the X-axis, s y1 represents the actual features rotating around the Y-axis, and α1 represents the actual direction angle of the target region, and T represents the transpose of the matrix; based on a preset expected image, the expected image features and the expected interaction matrices of the expected image pose are calculated:

[0180] s0=[x n0 y n0 a n0s x0 s y0 α0] T ,

[0181]

[0182] wherein s0 represents a desired image feature, L0 represents a desired interaction matrix, x n0 represents a desired image feature for controlling X-direction translational freedom, y n0 represents a desired image feature for controlling Y-direction translational freedom, a n0 represents a desired image feature for controlling Z-direction translational freedom, s x0 represents a desired feature for rotation around the X-axis, s y0 represents a desired feature for rotation around the Y-axis, and a0 represents a desired orientation angle of the target region.

[0183] S3. Based on the actual image feature, the actual interaction matrix, the desired image feature and the desired interaction matrix, an optimal solution of a motion posture of the robot is calculated by using an optimal control algorithm and a control barrier function.

[0184] The method for obtaining the optimal solution of the motion posture comprises: a camera desired speed calculation unit calculates a camera desired speed based on the actual image feature, the actual interaction matrix, the desired image feature and the desired interaction matrix:

[0185]

[0186] e = S1 - s0,

[0187]

[0188] wherein v d represents the camera desired speed, λ represents a constant, e represents an image feature error, L s represents an interaction matrix, represents an inverse matrix of the interaction matrix; and a barrier function setting unit is configured to set a barrier function:

[0189]

[0190] wherein B represents the barrier function, represents an image feature difference for controlling X-direction translational freedom, represents an image feature difference for controlling Y-direction translational freedom, and p represents a visibility constraint radius; and a robot posture calculation unit is configured to calculate an optimal solution of a motion posture of the robot based on an optimal control algorithm and in combination with the barrier function:

[0191] B (1) ≥ -k b B,

[0192] min M = ||v d -v c || 2 ,

[0193] s.t.v c = J robot Δq,

[0194]

[0195] wherein B (1) represents the first derivative of the barrier function, k b represents a proportional constant, J robot represents the Jacobian matrix of the robot, v c represents the actual speed of the camera, and Δq represents the optimal solution of the motion posture, represents the rotational speed of the robot, represents the bending speed of the robot, represents the feed speed of the robot.

[0196] S4. Controlling the posture of the robot based on the optimal solution of the motion posture.

[0197] The calculated rotational speed, bending speed and feed speed of the ureter robot are converted to obtain the motion speed control amount required by the motor, and the control amount is sent to the ureteroscopy robot for motor driving, so as to adjust the posture of the ureteroscopy end.

[0198] The above-described embodiments are only descriptions of the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. An automatic tracking system for kidney stones in a ureteroscopic surgical robot based on optimal visual servoing, characterized by: include: Target recognition module, feature extraction module, attitude solution module and attitude adjustment module; The target recognition module is used to obtain image data of the ureteral cavity and kidney stones, and process the image data to obtain a target area of ​​the stone; The feature extraction module is used to extract features of the stone target area to obtain actual image features and actual interaction matrix of actual image posture, and is also used to calculate expected image features and expected interaction matrix of expected image posture; The posture solving module is used to calculate the optimal solution of the motion posture of each joint of the robot using the optimal control algorithm and the control obstacle function based on the actual image features, the actual interaction matrix, the expected image features and the expected interaction matrix; The posture adjustment module is used to control the posture of the robot based on the optimal solution of the motion posture; The feature extraction module includes: an actual image extraction unit and an expected image extraction unit; The actual image extraction unit is used to extract features of the binarized target stone area to obtain the actual image features of the actual image posture and the actual interaction matrix: s1=[x n1 y n1 a n1 s x1 s y1 α1] T , Among them, s1 represents the actual image feature, L1 represents the actual interaction matrix, and x n1 represents the actual image feature that controls the translational freedom in the X direction, y n1 represents the actual image feature that controls the Y-direction translational freedom, a n1 represents the actual image feature that controls the Z-direction translational freedom, s x1 Indicates the actual characteristics of rotation around the X axis, s y1 represents the actual feature rotated around the Y axis, α1 represents the actual direction angle of the target area, and T represents the transpose of the matrix; The expected image extraction unit calculates the expected image features and the expected interaction matrix of the expected image pose based on the preset expected image: s0=[x n0 y n0 a n0 s x0 s y0 α0] T , Among them, s0 represents the expected image features, L0 represents the expected interaction matrix, and x n0 represents the desired image feature that controls the translational freedom in the X direction, y n0 represents the desired image feature that controls the Y-direction translational freedom, a n0 represents the desired image feature that controls the Z-direction translational freedom, s x0 represents the desired characteristics of rotation around the X axis, s y0 represents the desired feature of rotation around the Y axis, and α0 represents the desired direction angle of the target area; The posture solving module includes: a camera expected speed calculation unit, an obstacle function setting unit and a robot posture calculation unit; The camera expected speed calculation unit calculates the camera expected speed based on the actual image features, the actual interaction matrix, the expected image features, and the expected interaction matrix: e=s1-s0, Among them, v d represents the expected speed of the camera, λ represents a constant, e represents the image feature error, L s represents the interaction matrix, represents the inverse matrix of the interaction matrix; The barrier function setting unit is used to set the barrier function: Where B represents the barrier function, represents the image feature difference that controls the translational freedom in the X direction, represents the image feature difference that controls the Y-direction translation freedom, and ρ represents the visibility constraint radius; The robot posture calculation unit is used to calculate the optimal solution of the robot's motion posture based on the optimal control algorithm and in combination with the obstacle function: B (1) ≥-k b B, min M=||v d -v c || 2 , s.t.v c =J robot Δ q , Among them, B (1) represents the first-order derivative of the barrier function, k b represents the proportional constant, J robot represents the Jacobian matrix of the robot, v c represents the actual speed of the camera, Δq represents the optimal solution of the motion posture, represents the rotation speed of the robot, represents the bending speed of the robot, Indicates the feed rate of the robot.

2. The kidney stone automatic tracking system of the ureteroscopic surgical robot based on optimal visual servoing according to claim 1 is characterized in that: The target recognition module includes: an image acquisition unit and an image processing unit; The image acquisition unit is used to obtain the image data of the ureteral cavity and kidney stones; The image processing unit uses a neural network algorithm to identify the outline of the kidney stone in the image data, and uses an adaptive threshold method to binarize the image data to obtain a binary image, and then marks the outline in the binary image to obtain the stone target area.

3. A method for automatically tracking kidney stones using a ureteroscopic surgical robot based on optimal visual servoing, the method being applied to the system according to any one of claims 1 to 2, characterized in that: The following steps are involved: Acquiring image data of the ureteral cavity and kidney stones, and processing the image data to obtain a target area of ​​the stone; Performing feature extraction on the target stone area to obtain actual image features and an actual interaction matrix of the actual image posture, which are also used to calculate expected image features and an expected interaction matrix of the expected image posture; Calculating an optimal solution for the motion posture of each joint of the robot using an optimal control algorithm and a control obstacle function based on the actual image features, the actual interaction matrix, the expected image features, and the expected interaction matrix; The posture of the robot is controlled based on the optimal solution of the motion posture.

4. The method for automatically tracking kidney stones using a ureteroscopic surgical robot based on optimal visual servoing according to claim 3, characterized in that: The method for obtaining the target area of ​​the stone includes: Acquiring the image data of the ureteral cavity and kidney stones; A neural network algorithm is used to identify the outline of the kidney stone in the image data, and an adaptive threshold method is used to binarize the image data to obtain a binary image. The outline is then marked in the binary image to obtain the stone target area.

5. The method for automatically tracking kidney stones using a ureteroscopic surgical robot based on optimal visual servoing according to claim 3, characterized in that: The method for performing the feature extraction includes: Feature extraction is performed on the binarized target stone area to obtain the actual image features and the actual interaction matrix of the actual image posture: s1=[x n1 y n1 a n1 s x1 s y1 α1] T , Among them, s1 represents the actual image feature, L1 represents the actual interaction matrix, and x n1 represents the actual image feature that controls the translational freedom in the X direction, y n1 represents the actual image feature that controls the Y-direction translational freedom, a n1 represents the actual image feature that controls the Z-direction translational freedom, s x1 Indicates the actual characteristics of rotation around the X axis, s y1 represents the actual feature rotated around the Y axis, α1 represents the actual direction angle of the target area, and T represents the transpose of the matrix; Based on the preset expected image, the expected image features and the expected interaction matrix of the expected image pose are calculated: s0=[x n0 y n0 a n0 s x0 s y0 α0] T , Among them, s0 represents the expected image features, L0 represents the expected interaction matrix, and x n0 represents the desired image feature that controls the translational freedom in the X direction, y n0 represents the desired image feature that controls the Y-direction translational freedom, a n0 represents the desired image feature that controls the Z-direction translational freedom, s x0 represents the desired characteristics of rotation around the X axis, s y0 represents the desired feature of rotation around the Y axis, and α0 represents the desired direction angle of the target area.

6. The method for automatically tracking kidney stones using a ureteroscopic surgical robot based on optimal visual servoing according to claim 5, characterized in that: The method for obtaining the optimal solution of the motion posture includes: The camera expected speed calculation unit calculates the camera expected speed based on the actual image features, the actual interaction matrix, the expected image features, and the expected interaction matrix: e=s1-s0, Among them, v d represents the expected speed of the camera, λ represents a constant, e represents the image feature error, L s represents the interaction matrix, represents the inverse matrix of the interaction matrix; The barrier function setting unit is used to set the barrier function: Where B represents the barrier function, represents the image feature difference that controls the translational freedom in the X direction, represents the image feature difference that controls the Y-direction translation freedom, and ρ represents the visibility constraint radius; The robot posture calculation unit is used to calculate the optimal solution of the robot's motion posture based on the optimal control algorithm and in combination with the obstacle function: B (1) ≥-k b B, min M=||v d -v c || 2 , s.t.v c =J robot Δq, Among them, B (1) represents the first-order derivative of the barrier function, k b represents the proportional constant, J robot represents the Jacobian matrix of the robot, v c represents the actual speed of the camera, Δq represents the optimal solution of the motion posture, represents the rotation speed of the robot, represents the bending speed of the robot, Indicates the feed rate of the robot.

Images