Endoscope robot vision servo and optimization control method, system and robot under RCM constraint

Abstract

The application discloses an endoscope robot visual servoing and optimization control method and system under RCM constraint and a robot, and comprises the following steps: step one, completing a target detection task in a surgical field and extracting a current state of the target; step two, establishing a constraint optimization control scheme according to a specific robot model; step three, based on a projection operator, equivalently converting the constraint optimization problem into a nonlinear equation set; step four, constructing an error function by using the nonlinear equation set obtained in step three, designing a recurrent neural network (RNN) solver based on Dini derivative, and obtaining an optimal solution of the constraint optimization problem; and step five, sending joint angles or joint speeds obtained in step four to a lower computer to drive the robot to move and automatically adjust the surgical field. Compared with other methods, in the new algorithm, iterative calculation is not needed, training is not needed, and partial coefficients and elements are allowed to be continuous but non-differentiable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to an endoscope robot visual servoing and optimization control method and system under RCM constraint, and a robot. BACKGROUND

[0002] In traditional minimally invasive surgery, manual operation has many unstable factors. For example, long-time operation of an endoscope will cause the doctor's hand to shake, resulting in unstable endoscope vision; in addition, there are disadvantages of communication and operation between doctors and assistants. In recent years, with the development of robot technology and the improvement of medical level, medical robots have been successfully applied in the field of minimally invasive surgery. The robot-assisted minimally invasive surgery (RAMIS) system can effectively remove the influence of hand shaking when the doctor operates, improve the stability of the operation, reduce the configuration of surgical personnel, alleviate the fatigue of the operator, and reduce the labor cost of the operation; the master-slave separation type operation mode can realize remote surgery and sharing of medical resources. In the research of minimally invasive surgery robots, how to make the surgical instrument integrated at the end of the robot reliably move with the small incision on the body surface as the fulcrum without expanding the incision trauma has become one of the research hotspots of surgical robots.

[0003] For an institution, if a part or a point of the institution always passes through a fixed point away from the institution itself during the movement of the institution, and the point has no actual physical constraint, the institution is called a "telecentric mechanism" (RCM mechanism). This feature of the telecentric mechanism meets the operation requirements of minimally invasive surgery, so the telecentric mechanism is widely used in minimally invasive surgery robots, and the movement performance and stability of the telecentric mechanism directly affect the operation performance of the entire minimally invasive surgery robot. The telecentric mechanism on the machine usually has only one fixed telecentric point, which often needs to be set for specific surgical operations, and lacks flexibility. In view of this, the existing literature proposes a software-based telecentric motion generation algorithm, which can be adjusted on the software level as needed, has higher flexibility, and has a wider range of applications. In the surgical process, it is crucial to provide stable and timely visual information for the doctor. Robot visual servoing control is a method of capturing images in real time through a vision sensor, extracting feature information through image processing algorithms, and using these feature information as feedback signals to drive the robot to operate.

[0004] The visual servoing and optimal control of the robot can be abstractly represented as an optimization problem with multiple constraints in mathematics, such as a quadratic programming (QP) problem. Due to the characteristics of parallel computing, adaptability, and hardware implementability, a neural network has become one of the effective methods for solving optimization problems. Existing QP solving algorithms mainly include a recursive neural network (RNN) based on gradient and a zeroing neural network (ZNN), wherein the RNN is suitable for solving time-invariant problems, and has problems of a large number of iterations and a long solving time. Although the ZNN has the advantage of not requiring iterative calculation, it requires that relevant elements and coefficients are derivable everywhere and cannot directly process double-end constraints. Based on the above analysis, a new QP solving algorithm Dini-RNN is proposed, in which iterative calculation is not required, double-end constraints can be directly processed, and some coefficients and elements are allowed to be continuous but non-derivable. SUMMARY

[0005] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and provide an endoscopic robot visual servoing and optimal control method, system and robot under RCM constraint. The present application can process the kinematic control problem of the redundant manipulator for automatic tracking of the target according to the visual feedback information under the RCM constraint, and can realize joint limit avoidance, and ensure the safety of the robot during task execution.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] In the first aspect, the present application provides an endoscopic robot visual servoing and optimal control method under RCM constraint, comprising the following steps:

[0008] Step one, complete the target detection task in the surgical field, and extract the current state r of the target;

[0009] Step two, establish the corresponding constraint optimization control scheme according to the specific endoscopic robot, and design the performance index as The equality constraint is And And the double-end constraint is q - ≤q≤q + And Wherein W represents a weighted matrix, q and Respectively represent the driving joint angle and the angular rate of change, q - And q + Respectively represent the upper and lower limits of the driving joint angle, And Respectively represent the upper and lower limits of the driving joint angular rate of change, J rcm Indicates the Jacobian matrix at the RCM point p rcm , J system Is the Jacobian matrix of the endoscopic robot, And respectively, the velocity of the RCM point and the velocity of the feature point under the camera plane; two double-end constraints on the driving joint angle and its rate of change are arranged into one double-end constraint through an equivalent transformation method wherein and β>0, and by introducing a matrix n>5 and The constraint optimization control scheme is uniformly represented as a quadratic programming problem with a general form, wherein the performance index is The constraint conditions are

[0010] Step three, by introducing a projection operator wherein the specific expression of the projection operator is defined as

[0011] and based on the KKT condition, the constraint optimization problem is equivalent to a nonlinear equation group g(t,x,μ)=0;

[0012] Step four, inspired by the design idea of the zeroization neural network, a Dini derivative is introduced to obtain a new RNN solver wherein γ is a convergence parameter, and Φ(·) represents an activation function array,

[0013]

[0014]

[0015] is a diagonal matrix,

[0016] z=Cx+μ,

[0017]

[0018] is the Dini right upper derivative operator with respect to z i , the optimal solution of the constraint optimization problem is obtained by solving the RNN solver, and the rate of change of the driving joint angle of the endoscopic robot is obtained

[0019] Step five, the result obtained in step four or the driving joint angle q is sent to the lower computer, and the robot is driven to move to automatically adjust the surgical field.

[0020] As a preferred technical scheme, the endoscopic robot comprises a fixed platform, an end effector, an endoscope, and n driving joints q1-q n n>5, and the velocity layer kinematics equation of the mechanical arm of the endoscopic robot is wherein J represents a Jacobian matrix, represents the camera plane feature point velocity and the linear velocity at the RCM point.

[0021] As a preferred technical solution, the Jacobian matrix is specifically:

[0022]

[0023] wherein J system = J image J camera , J image is an image Jacobian matrix,

[0024]

[0025] J task is an endoscope tip Jacobian matrix relative to the base coordinate system, 0 J n and 0 J n+1 are the Jacobian matrices of the nth coordinate system and the n+1 coordinate system relative to the base coordinate system respectively, 0 R n+1 is the rotation matrix of the n+1 coordinate system relative to the base coordinate system, and λ∈(0,1).

[0026] As a preferred technical solution, in the robot velocity layer kinematics equation, the design optimization performance index and considering the upper and lower limit constraints of the driving joint angle and its rate of change, a constrained optimization control scheme of the endoscope robot is established;

[0027] After equivalent arrangement, it is mathematically represented as a quadratic programming problem with a general form, wherein the performance index is the constraint condition is

[0028] Based on the projection operator and the KKT condition, the quadratic programming problem is equivalent to a nonlinear equation group g(t,x,μ)=0.

[0029] As a preferred technical solution, the projection operator is continuous everywhere.

[0030] As a preferred technical solution, inspired by the design idea of the annihilating neural network, a brand new RNN solver is obtained wherein:

[0031]

[0032]

[0033] is a diagonal matrix,

[0034]

[0035] z = Cx + μ,

[0036]

[0037] As a preferred technical solution, the RNN solver does not need training and iterative calculation, can efficiently solve the optimal solution of the quadratic programming problem, and further obtains the angle change rate of the driving joint of the endoscope robot By sending the driving joint angle q obtained by the integral of the driving joint angle change rate to the lower computer, the robot movement is driven, and the automatic adjustment of the surgical field is performed.

[0038] In a second aspect, the application also provides an endoscope robot visual servoing and optimization control system under RCM constraint, which is applied to the endoscope robot visual servoing and optimization control method under RCM constraint, and includes a target detection module, a kinematic control scheme construction module, an equivalent conversion module, an optimal problem solving module and a driving module.

[0039] The target detection module is used to constrain the motion trajectory, i.e., the ideal speed of the feature point under the camera plane of the endoscope robot is set.

[0040] The kinematic control scheme construction module is used to establish the corresponding constraint optimization control scheme of the endoscope robot according to the specific endoscope robot, and the performance index is designed as The equality constraint is and and the double-end constraint is q - ≤q≤q + and wherein W represents a weighted matrix, q and respectively represent the driving joint angle and the angle change rate, q - and q + respectively represent the upper and lower limits of the driving joint angle, and respectively represent the upper and lower limits of the driving joint angle change rate, J rcm represents the Jacobian matrix at the RCM point p rcm , J system is the Jacobian matrix of the endoscope robot, and respectively represent the linear velocity of the RCM point and the feature point velocity under the camera plane; the two double-end constraints on the driving joint angle and the angle change rate are arranged into one double-end constraint by the equivalent conversion method wherein and β>0, and by introducing the matrix n>5 and The constraint optimization control scheme is uniformly represented as a quadratic programming problem with a general form, wherein a performance index is The constraint condition is

[0041] The equivalent conversion module is configured to convert the constraint optimization problem into a nonlinear equation group g(t, x, mu) = 0 based on a projection operator

[0042]

[0043] The constraint optimization problem is equivalent to the nonlinear equation group g(t, x, mu) = 0 by using the KKT condition;

[0044] The optimal solution module is configured to define an error monitoring function e(t): = g(t, x, mu), wherein the error monitoring function is the nonlinear equation group obtained in step three, inspired by the homotopy neural network design idea, a Dini derivative is introduced, and a new RNN solver is obtained wherein gamma is a convergence parameter, and Phi(·) represents an activation function array,

[0045]

[0046]

[0047] is a diagonal matrix,

[0048] z = Cx + mu,

[0049]

[0050] is a Dini right upper derivative operator with respect to z i The optimal solution of the constraint optimization problem is obtained by using the RNN solver, and then the angular rate of change of the driving joint of the endoscopic robot is obtained.

[0051] The driving module is configured to send the obtained result or the driving joint angle q to a lower computer, drive the robot to move, and automatically adjust the surgical field.

[0052] In a third aspect, the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the endoscopic robot visual servoing and optimization control method under the RCM constraint.

[0053] In a fourth aspect, the present application further provides an endoscopic robot, which comprises:

[0054] At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores computer program instructions executable by the at least one processor, the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the endoscopic robot visual servoing and optimization control method under the RCM constraint.

[0055] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0056] The present application can effectively overcome the deficiencies of the prior art. Prior to this, the ZNN solver requires the derivative information of all relevant elements and coefficients, so the equality constraint and the inequality constraint need to have a boundary that is differentiable everywhere. In addition, when considering the double-end constraint, the ZNN solver needs to convert the double-end constraint into an inequality constraint for processing, which increases the number of neurons and the computational complexity of the neural network controller. The present application uses a projection operator to process the inequality constraint and the equivalent transformation of the quadratic programming problem, introduces the Dini derivative, and combines the evolution rule and the activation function to obtain a new RNN solver. The RNN solver can obtain the optimal solution of the quadratic programming problem. After obtaining the optimal solution of the quadratic programming problem, the endoscopic robot can be driven to move under the RCM constraint, and the target automatic tracking task can be realized. The Dini-RNN solver algorithm has the advantages of no iteration, direct processing of double-end constraints, and allowing partial coefficients and elements to be continuous but non-differentiable. The present application provides a control method for endoscopic robot visual servoing automatic tracking of a target under the RCM constraint, which is easy to operate, does not require training and iterative calculation, and is standard. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0058] Figure 1 The flow chart of the visual servoing and optimization control of the endoscopic robot under the RCM constraint of the embodiment of the present application;

[0059] Figure 2 The model diagram of the simulation Franka Emika Panda mechanical arm of the embodiment of the present application.

[0060] Figure 3 The RCM scheme model diagram of the embodiment of the present application.

[0061] Figure 4Trajectory of feature points on the camera plane of the simulated Franka Emika Panda robot arm in this embodiment of the application, starting from any of the 6 initial positions, eventually reaching the center of the camera plane.

[0062] Figure 5 RCM error map of the simulated Franka Emika Panda robot arm in this embodiment of the application.

[0063] Figure 6 Joint angle variation map of the simulated Franka Emika Panda robot arm in this embodiment of the application.

[0064] Figure 7 Joint angle rate of change map of the simulated Franka Emika Panda robot arm in this embodiment of the application.

[0065] Figure 8 is a structural schematic diagram of a system for constraint motion planning and control of an endoscopic robot under RCM constraints in an embodiment of the application;

[0066] Figure 9 is a structural schematic diagram of a robot in an embodiment of the application. DETAILED DESCRIPTION

[0067] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme of the present application will be described clearly and completely below in conjunction with the embodiments in the present application and the drawings, and it should be understood that the drawings are only used for illustrative description and cannot be understood as a limitation on the present patent. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0068] In the present application, the phrase “embodiment” means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0069] EMBODIMENT

[0070] As shown in Figure 1 , the present embodiment is a method for vision servoing and optimal control of an endoscopic robot under RCM constraints, which comprises the following steps:

[0071] Step one, complete the target detection task in the surgical field, extract the current state of the target r;

[0072] Step two, establish the corresponding constraint optimization control scheme according to the specific endoscopic robot, and the performance index in the finally obtained constraint optimization control scheme is The equality constraint is The double-end constraint is

[0073] In the step two, through the equality constraint The feature point velocity in the camera plane can be associated with the RCM point velocity And its joint angular velocity So that the automatic tracking target control under the RCM constraint can be achieved. Through the inequality constraint The joint limit constraint of the joint angle of the endoscopic robot can be processed.

[0074] Step three: based on the projection operator The constraint optimization problem is equivalent to a nonlinear equation set.

[0075] In the step three, by defining the corresponding Lagrange function for the constraint optimization problem, on this basis, the KKT condition is obtained by derivation, and the KKT condition is processed using

[0076]

[0077] The constraint optimization problem can be equivalent to a nonlinear equation set.

[0078] Step four: for the nonlinear equation set in step three, construct an error function and design an RNN solver to obtain the optimal solution of the nonlinear equation set and the constraint optimization problem, and then obtain the speed of each joint of the endoscopic robot.

[0079] Based on the nonlinear equation set in step three, an error monitoring function e(t): = g(t,x,μ) is constructed, inspired by the idea of zeroization neural network, Dini derivative is introduced, and a new RNN solver Where γ is the convergence parameter, Φ(·) represents the activation function array,

[0080]

[0081]

[0082] is a diagonal matrix,

[0083] z=Cx+μ,

[0084]

[0085] is the Dini right upper derivative operator of z i , the solution of nonlinear equations and the optimal solution of constrained optimization problems can be obtained simultaneously by the final RNN solver The endoscopic robot driving angle rate of change can be obtained

[0086] Step five: send the results obtained in step four or the driving joint angle q to the lower computer, drive the endoscopic robot to move, and automatically adjust the surgical field.

[0087] As shown in Figure 2 , in the embodiment, the endoscopic robot includes a fixed platform, an end effector, an endoscope, and seven rotating joints, which are O1,..., O7 and the endoscope O8, respectively. The kinematic relationship of the robot velocity layer under the RCM constraint is Where represents the camera plane feature point velocity and the RCM point velocity.

[0088] As shown in Figure 3 , the endoscopic link enters the body through the insertion point, and the RCM point cannot be laterally displaced.

[0089] As shown in Figure 4 , randomly initialize the detection target to any position on the camera plane 6 times, and finally the camera controls the movement of the mechanical arm through visual servoing. As can be seen from the figure, the detection target moves to the center of the camera plane, indicating that the method can realize automatic tracking of the target under the RCM constraint, and deviation rarely occurs.

[0090] As shown in Figure 5 , where the solid line e x represents the error of the simulation Panda mechanical arm RCM point in the X direction, the dotted line e y represents the error of the simulation Panda mechanical arm RCM point in the Y direction, and the dashed line e z represents the error of the simulation Panda series mechanical arm RCM point in the Z direction. During the task execution, the errors in the three directions are less than or equal to 1.5x10 -5 meters, and the accuracy is higher than the sub-micron level accuracy.

[0091] As shown in Figure 6are shown, wherein q1, q2, q3, q4, q5, q6, q7 respectively represent the angles of the first rotary joint O1, the second rotary joint O2, the third rotary joint O3, the fourth rotary joint O4, the fifth rotary joint O5, the sixth rotary joint O6 and the seventh rotary joint O7 of the simulated Panda endoscopic robot. During the task execution, the angles of the respective joints are constantly changing, and the corresponding mechanical arm appears various different motions.

[0092] As shown in Figure 7 , wherein, respectively represent the angular change rates of the first rotary joint O1, the second rotary joint O2, the third rotary joint O3, the fourth rotary joint O4, the fifth rotary joint O5, the sixth rotary joint O6 and the seventh rotary joint O7 of the simulated Panda endoscopic robot. It can be seen from Figure 7 that during the task execution, the angular change rates of the respective joint angles can be changed within a certain range, thereby illustrating the effectiveness of the present application for handling double-end constraints and avoiding joint limits in the quadratic programming problem.

[0093] As shown in Figure 8 , in another embodiment of the present application, an endoscopic robot visual servoing and optimization control system 100 under RCM constraints is provided, comprising a target detection module 101, a kinematic control scheme construction module 102, an equivalent conversion module 103, an optimal solution module 104 and a driving module 105.

[0094] The target detection module 101 is used to constrain the motion trajectory, i.e. to set the ideal speed of the feature points under the camera plane of the endoscopic robot.

[0095] The kinematic control scheme construction module 102 is used to establish the corresponding constraint optimization control scheme of the endoscopic robot according to the specific endoscopic robot, and design the performance index as The equality constraint is and and the double-end constraint is q - ≤q≤q + and wherein W represents a weighting matrix, q and respectively represent the driving joint angle and the angular change rate, q - and q + respectively represent the upper and lower limits of the driving joint angle, and respectively represent the upper and lower limits of the driving joint angular change rate, J rcm represents the Jacobian matrix at the RCM point p rcm , J system is the Jacobian matrix of the endoscopic robot, and Let RCM point and feature point velocity be represented respectively. Using an equivalent transformation method, the two double-ended constraints on the driving joint angle and its rate of change can be reorganized into a single double-ended constraint. in And β>0, and then by introducing a matrix n>5 and Constrained optimization control schemes are uniformly represented as quadratic programming problems of a general form, where the performance index is: The constraints are

[0096] The equivalent transformation module 103 is used for transformation based on the projection operator.

[0097]

[0098] With the KKT conditions, the constrained optimization problem is equivalently transformed into a nonlinear system of equations g(t,x,μ)=0;

[0099] The optimal solution module 104 is used to define the error monitoring function e(t):=g(t,x,μ). The error monitoring function is derived by introducing the Dini derivative from the obtained nonlinear equation system, inspired by the design idea of ​​nullable neural networks, to obtain a novel RNN solver. Where γ is the convergence parameter, and Φ(·) represents the array of activation functions.

[0100]

[0101]

[0102] It is a diagonal matrix.

[0103] z = Cx + μ

[0104]

[0105] It is relative to z i The upper right derivative operator of Dini is used to solve the constrained optimization problem through an RNN solver, thereby obtaining the angular change rate of the endoscopic robot's driven joints.

[0106] The driving module 105 is used to process the obtained results. Its integral q is sent to the lower-level machine to drive the endoscopic robot to move.

[0107] Furthermore, in the implementation of the motion planning and control system for the endoscopic robot under RCM constraints in the above embodiments, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the visual servo and optimization control system of the endoscopic robot under RCM constraints is divided into different program modules to complete all or part of the functions described above.

[0108] like Figure 9 As shown, in one embodiment, a robot 200 is provided. The robot 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a visual servoing and optimization control program 203 for an endoscope robot under RCM constraints.

[0109] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the robot 200, such as the robot 200's portable hard drive. In other embodiments, the first memory 202 can also be an external storage device of the robot 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the robot 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the robot 200. The first memory 202 can be used not only to store application software and various types of data installed on the robot 200, such as the code of the visual servoing and optimization control program 203 of the endoscope robot under RCM constraints, but also to temporarily store data that has been output or will be output.

[0110] The first processor 201 may, in some embodiments, be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits of the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, connects various components of the entire electronic device through various interfaces and lines, executes programs or modules stored in the first memory 202 (such as a federated learning defense program, etc.), and calls data stored in the first memory 202 to perform various functions and process data of the robot 200.

[0111] Figure 9 Only the robot with components is shown, and those skilled in the art can understand that, Figure 8 The structure shown does not constitute a limitation on the robot 200, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0112] The endoscopic robot vision servoing and optimization control program 203 stored in the first memory 202 in the robot 200 under RCM constraints is a combination of multiple instructions, which, when running in the first processor 201, can achieve:

[0113] Step one, complete the target detection task in the surgical field, and extract the current state r of the target;

[0114] Step two, establish the corresponding constraint optimization control scheme according to the specific endoscopic robot, and design the performance index as The equality constraint is And And the double-end constraint is q - ≤q≤q + And Where W represents a weighted matrix, q and represent the driving joint angle and angular rate, respectively, q - and q + represent the upper and lower limits of the driving joint angle, And represent the upper and lower limits of the driving joint angular rate, J rcm represents the Jacobian matrix at the RCM point p rcm , J system is the endoscopic robot Jacobian matrix, And respectively, are the linear velocity of RCM point and the velocity of feature point in camera plane; two double-ended constraints on driving joint angle and its rate of change are rearranged into one double-ended constraint through equivalent transformation method where and β>0, and by introducing matrix n>5 and The constraint optimization control scheme is uniformly represented as a quadratic programming problem with general form, where the performance index is The constraint conditions are

[0115] Step three, by introducing projection operator where the specific expression of the projection operator is defined as

[0116]

[0117] And based on KKT conditions, the constraint optimization problem is equivalent to the nonlinear equation group g(t,x,μ)=0;

[0118] Step four, inspired by the design idea of zeroing neural network, Dini derivative is introduced to obtain a new RNN solver where γ is the convergence parameter, and Φ(·) represents the activation function array,

[0119]

[0120]

[0121] is a diagonal matrix,

[0122] z=Cx+μ,

[0123]

[0124] is the Dini right upper derivative operator with respect to z i , the optimal solution of the constraint optimization problem is obtained by solving the RNN solver, and the rate of change of the driving joint angle of the endoscopic robot is obtained

[0125] Step five, the results obtained in step four or the driving joint angle q are sent to the lower computer, and the robot is driven to move to automatically adjust the surgical field.

[0126] Furthermore, if the modules / units integrated into the robot 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0127] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An endoscopic robot vision servoing and optimization control method under RCM constraints, characterized in that, The method comprises the following steps: Step one, completing the target detection task in the surgical field, extracting the current state r of the target; Step two, establish the corresponding constraint optimization control scheme according to the specific endoscopic robot, and design the performance index as The equality constraint is And And the double-end constraint is q - ≤q≤q + And Wherein W represents a weighted matrix, q and Respectively represent the driving joint angle and the angular rate, q - And q + Respectively represent the upper and lower limits of the driving joint angle, And Respectively represent the upper and lower limits of the driving joint angular rate, J rcm Indicates the Jacobian matrix at the RCM point p rcm J system Is the endoscopic robot Jacobian matrix, And Respectively represent the linear velocity of the RCM point and the feature point velocity under the camera plane; through the equivalent transformation method, the two double-end constraints of the driving joint angle and the angular rate are arranged into a double-end constraint Wherein And β>0, and then by introducing the matrix N>5 and The constraint optimization control scheme is uniformly represented as a quadratic programming problem with a general form, wherein the performance index is The constraint condition is Step three, by introducing a projection operator where the specific expression of the projection operator is defined as And based on the KKT condition, the constraint optimization problem is equivalent to a nonlinear equation group g(t, x, μ) = 0; Step four, inspired by the design idea of zeroing neural network, Dini derivative is introduced to get a new RNN solver where γ is the convergence parameter, and Φ(·) represents the activation function array, is a diagonal matrix, Z = Cx + μ, is the Dini right upper derivative operator of z i Dini right upper derivative operator of z, the optimal solution of the constrained optimization problem is solved by the RNN solver, and then the angular rate of change of the endoscopic robot driving joint is obtained Step five, the results of step four are sent to the lower computer or the driving joint angle q is sent to the lower computer, the robot is driven to move, and the automatic adjustment of the surgical field is performed.

2. The endoscopic robotic visual servoing and optimization control method under RCM constraints of claim 1, wherein, The endoscope robot comprises a fixed platform, an end effector, an endoscope and n driving joints q1~q n , n>5, and a mechanical arm velocity layer kinematics equation of the endoscope robot is wherein J represents a Jacobian matrix, represents a camera plane feature point velocity and a linear velocity at an RCM point.

3. The endoscopic robotic visual servoing and optimization control method under RCM constraints of claim 2, wherein, The Jacobian matrix is specifically: where J system = J image J camera , J image is the image Jacobian matrix, J task is the Jacobian matrix of the endoscope tip with respect to the base coordinate system, 0 J n and 0 J n+1 are the Jacobian matrices of the nth coordinate system and the n+1th coordinate system with respect to the base coordinate system, respectively, 0 R n+1 is the rotation matrix of the n+1th coordinate system with respect to the base coordinate system, and λ ∈ (0, 1).

4. The endoscopic robotic visual servoing and optimization control method under RCM constraints of claim 2, wherein, In the robot velocity layer kinematics equation, a design optimization performance index And considering the upper and lower limit constraints of the driving joint angle and its rate of change, a constraint optimization control scheme of the endoscope robot is established After equivalent arrangement, it is mathematically represented as a quadratic programming problem with a general form, where the performance index is The constraint conditions are Based on the projection operator and the KKT condition, the quadratic programming problem is equivalent to a nonlinear equation group g(t, x, μ) = 0.

5. The endoscopic robotic visual servoing and optimization control method under RCM constraints of claim 4, wherein, The projection operator is continuous everywhere.

6. The endoscopic robotic visual servoing and optimization control method under RCM constraints of claim 1, wherein, Inspired by the idea of zeroing neural network, a new RNN solver is obtained Among them: is a diagonal matrix, Z = Cx + μ, 7. The endoscopic robotic visual servoing and optimization control method under RCM constraints of claim 6, wherein, The RNN solver can efficiently solve the optimal solution of the quadratic programming problem without training and iterative calculation, and further obtain the angular rate of change of the driving joint of the endoscopic robot The angular rate of change of the driving joint is obtained by Or the driving joint angle q obtained by integration is sent to the lower computer to drive the robot movement and automatically adjust the surgical field.

8. The endoscopic robot vision servoing and optimization control system under RCM constraints, applied to the endoscopic robot vision servoing and optimization control method under RCM constraints in any one of claims 1-7, characterized in that, The method comprises a target detection module, a kinematic control scheme construction module, an equivalent conversion module, an optimization problem solving module, and a driving module. The target detection module is configured to constrain a motion trajectory, i.e., an ideal speed of a feature point under a plane of an endoscope robot camera determination; The kinematics control scheme construction module is configured to establish a corresponding constraint optimization control scheme according to a specific endoscopic robot, and a performance index is designed as An equation constraint is and and a double-end constraint is q - ≤q≤q + and wherein W represents a weighting matrix, q and respectively represent a driving joint angle and an angular rate, q - and q + respectively represent upper and lower limits of the driving joint angle, and respectively represent upper and lower limits of the driving joint angular rate, J rcm represents a Jacobian matrix at an RCM point p rcm , J system is an endoscopic robot Jacobian matrix, and respectively represent a linear velocity of the RCM point and a feature point velocity under a camera plane; two double-end constraints on the driving joint angle and the angular rate thereof are arranged into one double-end constraint through an equivalent transformation method wherein and β>0, a matrix n>5 and is introduced, and the constraint optimization control scheme is uniformly represented as a quadratic programming problem with a general form, wherein the performance index is and the constraint condition is The equivalent conversion module is configured to convert the constraint optimization problem into a nonlinear equation group g(t, x, μ) = 0 based on a projection operator And the KKT condition; The optimal solution module is inspired by the design idea of the zero neural network, introduces Dini derivative, and obtains a new RNN solver where γ is a convergence parameter, and Φ(·) represents an activation function array. is a diagonal matrix, Z = Cx + μ, is the Dini right upper derivative operator of i the Dini right upper derivative operator of The driving module is configured to send the obtained result to a lower computer, drive the robot to move, and automatically adjust the surgical field. or drive joint angle q to the lower computer, drive the robot movement, and perform automatic adjustment of the surgical field.

9. A computer readable storage medium storing a program, characterized in that, When the program is executed by the processor, the endoscopic robot visual servoing and optimization control method under the RCM constraint is realized.

10. An endoscopic robot, characterized by, The endoscopic robot comprises: At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the endoscopic robot visual servoing and optimization control method under the RCM constraint as claimed in any one of claims 1-7.

Images