Robotic autonomous cutting methods and systems for in-vivo flexible dynamic environments

Abstract

This invention provides a robotic autonomous cutting method for flexible dynamic environments within the body, relating to the field of robotic autonomous cutting. It includes: accurately tracking the cutting path and key tissue areas in complex dynamic environments within the body, ensuring cutting precision. A general linear control system is constructed and a state equation is established. Target controllers are designed for each objective requirement in laparoscopic surgery scenarios. The cumulative value of the target evaluation function of each controller is calculated at the current time to determine the target gradient. Based on a preset weight function, the target gradient values ​​are nested and fused sequentially from low to high weight hierarchy and added to the overall system control input. The fused motion speed and attitude change speed are converted into the joint angles of the surgical robot to achieve autonomous cutting operation. This method meets the needs of multi-target control, ensuring safe, precise, and efficient surgical cutting; and eliminates the reliance on surgeon experience and operational skills in traditional cutting methods.

Description

Technical Field

[0001] This invention relates to the field of autonomous robot cutting technology, specifically to an autonomous robot cutting method, system, storage medium, and electronic device for flexible dynamic environments within the body. Background Technology

[0002] Minimally invasive robotic surgery is controlled by the surgeon's hand, with robotic arms executing the surgical incision. Taking the da Vinci Surgical System as an example, it offers a direct, three-dimensional, high-definition view, with surgical field images magnified 10 to 15 times. It employs intuitive control technology, allowing the surgeon's movements to be reflected in the surgical instruments, while the sensor system automatically filters out tremors, maintaining the surgeon's operational stability. It also features seven degrees of freedom wrist-rotating surgical instruments, with bending and rotation capabilities exceeding the limits of the human hand.

[0003] In minimally invasive robotic surgery, the surgical cutting task is performed by the surgeon controlling the movement of the robotic arm via the master hand and controlling the electrocoagulation switch via foot pedals, allowing the surgical instruments at the end of the robotic arm to complete the cutting task. Existing technology proposes a traditional ophthalmic microscope system with a monocular camera to capture the surgical scene, determine the robot's position, and estimate depth information. Kinematic design with a remote center of motion (RCM) is used for multi-axis robots to execute incision paths, and autonomous cutting is achieved through PID trajectory tracking control. Experiments using ex vivo pig eyes show that autonomous, transparent corneal incisions have a more precise three-plane structure than surgical incisions, and are closer to ideal incisions.

[0004] However, due to the dynamic complexity of the internal environment and the significant differences in the internal environment between individuals, fully autonomous surgery that is completely independent of human control is still difficult to achieve. Most existing studies only explore semi-autonomous surgery for specific scenarios. In particular, tissue deformation caused by cutting tasks can affect the accuracy of robots in tracking target points. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a robotic autonomous cutting method, system, storage medium, and electronic device for flexible dynamic environments within the body, solving the technical problem that tissue deformation caused by cutting tasks affects the accuracy of the robot in tracking target points.

[0007] Technical solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A robotic autonomous cutting method for flexible dynamic environments within the body includes:

[0010] S1. Read the laparoscopic image, obtain the cutting path and the endpoint of the cutting target marked on the image frame according to the doctor's selection, and locate it in the three-dimensional image.

[0011] S2. Construct an overall linear control system and establish state equations. Design target controllers for each target achievement requirement in the laparoscopic surgery scenario. The target controllers include a planning path tracking controller, a target guidance controller, a cutting depth limit controller, and a collision avoidance controller.

[0012] S3. Establish corresponding motion control prediction models and target evaluation functions for the planning path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. Estimate the motion state of the system in the future for a prediction time interval based on the current system motion state, and calculate the corresponding cumulative value of the target evaluation function.

[0013] S4. Calculate the target gradient of the cumulative value of the target evaluation function of each controller at the current time; and according to the preset weight function, nest and fuse the target gradient values ​​of each controller in order of weight hierarchy from low to high, and add them to the total control input of the system.

[0014] S5. Convert the fused motion speed and posture change speed into the joint angles of the surgical robot.

[0015] Preferably, S1 specifically includes:

[0016] S11. Read the laparoscopic image and mark the planned cutting path S, the endpoint O of the cutting target, and the critical tissue area A that needs to be avoided on the initial frame image.

[0017] S12. Using the three-dimensional curve tracking method, the cutting path S marked in two dimensions, the endpoint O of the cutting target, and the key tissue region A are located in the three-dimensional image to obtain the planned trajectory Sd, the endpoint Od of the cutting target, and the key tissue region Ad that change dynamically with the image.

[0018] S13. Based on the doctor's selection, set the cutting depth limiting plane Dd on the plane where the planned trajectory is located;

[0019] S14. Measure the first transformation matrix from the instrument's end effector to the robot coordinate system using the optical positioning instrument; measure the transformation matrix from the 3D image of the laparoscope to the robot coordinate system using the camera model and the optical positioning instrument; unify all points to the robot's base coordinate system using the first and second coordinate transformation matrices to obtain the planned trajectory. The endpoint of the cut target is The key organizational areas are The cutting depth is limited by the plane. .

[0020] Preferably, S2 includes:

[0021] S21. Construct a general linear control system and establish state equations;

[0022]

[0023]

[0024] in, Let t be the motion state of the system at time t. Let be the velocity of the system at time t. This represents the total control input of the system at time t. Let t be the total control output of the system at time t, and A, B, C, and D be the calculation parameters of the state equations.

[0025] S22. Target controllers are designed to meet the various target achievement requirements in laparoscopic surgery scenarios. The target controllers include a path planning and tracking controller, a target guidance controller, a cutting depth limitation controller, and a collision avoidance controller.

[0026]

[0027] in, For controller The control input is a function of the system's total control output at time t and the desired control objective. To control the desired value at time t.

[0028] Preferably, the planned path tracking controller specifically refers to:

[0029]

[0030]

[0031] in, For the input of the path tracking controller, , For the proportional and derivative coefficients of the path tracking controller PD, for The motion state of the path planning and tracking controller is constantly being planned. The deviation between the desired position of the object controlled by the path planning controller and the overall control output of the system is calculated.

[0032] The target guidance controller specifically refers to:

[0033]

[0034]

[0035] in, The input to the target guides the controller. , The proportional and derivative coefficients of the target-guided controller PD are used to control the target. To determine the endpoint of the cut target, The speed at which the target-guided controller moves the object toward the target. The time expected to be taken for the target-guided controller to complete the cutting task;

[0036] The cutting depth limit setting controller specifically refers to:

[0037]

[0038]

[0039] in, For the input of the cutting depth limit controller, , The proportional and derivative coefficients of the PD controller are used to limit the cutting depth. The point on the cutting depth limiting plane that is closest to the overall control output. The deviation between the cutting depth limiting plane of the object controlled by the cutting depth limiting controller and the overall control output of the system. for and The angle between vectors;

[0040] The collision avoidance control setting controller specifically refers to:

[0041]

[0042]

[0043] in, The input to the collision avoidance controller, , For the proportional and derivative coefficients of the collision avoidance controller PD, Let t be the location of the center point of the critical organizational region. Let the radius of the sphere collision detection area be the center point of the obstacle. Let be the radius of the collision avoidance zone for the sphere, with the center point of the obstacle as the reference point, and let cd(t) be the distance between the total control output of the system and the center point of the obstacle. It is a very small constant.

[0044] Preferably, S3 includes:

[0045] S31. Establish corresponding motion control prediction models for the planned path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively.

[0046]

[0047]

[0048] in, For the controller at time t within the prediction interval Predicting motion state, For the controller at time t within the prediction interval Predicted motion speed, To predict the control target at time t within the prediction interval Predictive control output;

[0049] S32. Establish target evaluation functions for the planned path tracking controller, target guidance controller, cutting depth limit controller, and collision avoidance controller respectively, and combine them with the corresponding motion control prediction models at the current time. Based on the system's motion state, estimate the motion state over a future predicted time interval and calculate the corresponding cumulative value of the objective evaluation function;

[0050]

[0051] in, For the first Controller The cumulative value of the objective function at time t. For the first Controller prediction range The objective evaluation function value at a single time point within the timeframe. For the predicted time interval;

[0052] Preferably, in S32:

[0053] To evaluate the effectiveness of the path planning and tracking control controller within the predicted time interval, an objective evaluation function is established.

[0054]

[0055]

[0056] in, It is a path tracking controller The objective function value at time t. The planned path tracking controller is in the prediction interval arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the planned path tracking control target at time t within the prediction interval;

[0057] To evaluate the effectiveness of the target-guided control system within the predicted time interval, a target evaluation function is established.

[0058]

[0059]

[0060] in, It is a target guidance controller The objective function value at time t. Is the target-guided controller in the prediction interval arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the target guidance control target at time t within the prediction interval;

[0061] A target evaluation function is established for the cutting depth limitation control objective to evaluate the effectiveness of the cutting depth limitation controller within the prediction interval.

[0062]

[0063]

[0064] in, It is a cutting depth limit controller The objective function value at time t. The cutting depth limit controller is within the prediction range. arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the cutting depth limit control target within the prediction interval at time t;

[0065] A target evaluation function is established for the collision avoidance control objective to evaluate the effectiveness of the collision avoidance controller within the prediction range;

[0066]

[0067]

[0068] in, It is a collision avoidance controller The objective function value at time t. Is the collision avoidance controller within the prediction range? arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the collision avoidance control target at time t within the prediction interval. It is a very small constant.

[0069] Preferably, S4 includes:

[0070] S41. Calculate the objective evaluation function of each controller in the current state using an optimized method. The descent gradient at time step;

[0071]

[0072]

[0073]

[0074]

[0075] in, , , , These are the path planning and tracking controller, the target guidance controller, the cutting depth limit controller, and the collision avoidance controller. The gradient descent of the objective evaluation function at time step;

[0076] S42. Input the parameters of each control target and the importance of the control target, and sort the controllers according to the importance of the target to determine the priority of each controller;

[0077]

[0078] in, It is the weight hierarchy sequence of the target controller;

[0079] S43. Calculate the target gradient value after fusion by nesting the four controllers in order of weight level from low to high.

[0080]

[0081]

[0082] in, It's about gradients. The normalization function, Indicates the layering parameters, This represents the target gradient value after the nested fusion of four controllers;

[0083] S44. The nested and fused target gradient values ​​are summed into the fused controller, thereby realizing the motion fusion of multiple different target motion controllers;

[0084]

[0085] Where K is the proportionality coefficient.

[0086] Preferably, in S5, converting the fused motion speed and attitude change speed into the joint angles of the surgical robot specifically refers to...

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] in, Indicates a fixed RCM point. The X-axis vector represents the attitude matrix. The Y-axis vector represents the attitude matrix. The Z-axis vector represents the attitude matrix. For the robot's posture, the function This indicates that the attitude rotation matrix is ​​converted to Euler angles in Cartesian coordinates. Euler angles in Cartesian coordinates The rate of change of Euler angles, The rotational speed of the robot's joint angle. It is a Jacobian matrix.

[0093] An autonomous robotic cutting system for flexible, dynamic environments within the body, comprising:

[0094] The marking module is used to read laparoscopic images, obtain the cutting path and the endpoint of the cutting target marked on the image frame according to the doctor's selection, and locate them in the three-dimensional image.

[0095] The design module is used to construct the overall linear control system and establish the state equation. For each target achievement requirement in the laparoscopic surgery scenario, a target controller is designed. The target controller includes a planning path tracking controller, a target guidance controller, a cutting depth limit controller, and a collision avoidance controller.

[0096] The prediction module is used to establish corresponding motion control prediction models and target evaluation functions for the planned path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. Based on the system motion state at the current moment, it estimates the motion state of the future for a prediction time interval and calculates the corresponding cumulative value of the target evaluation function.

[0097] The fusion module is used to calculate the target gradient of the cumulative value of the target evaluation function of each controller at the current time; and according to the preset weight function, it nests and fuses the target gradient values ​​of each controller in order of weight hierarchy from low to high, and adds them to the total control input of the system.

[0098] The conversion module is used to convert the fused motion speed and attitude change speed into the joint angles of the surgical robot.

[0099] A storage medium storing a computer program for robotic autonomous cutting in a flexible, dynamic in vivo environment, wherein the computer program causes a computer to execute the robotic autonomous cutting method as described above.

[0100] An electronic device, comprising:

[0101] One or more processors;

[0102] Memory; and

[0103] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the robotic autonomous cutting method as described above.

[0104] (III) Beneficial Effects

[0105] This invention provides a method, system, storage medium, and electronic device for autonomous robotic cutting in flexible, dynamic in vivo environments. Compared with existing technologies, it offers the following advantages:

[0106] In this invention, the robot can accurately track the cutting path and key tissue areas drawn by the surgeon within the complex and dynamic environment of the body, ensuring the precision of the cutting. An overall linear control system is constructed and a state equation is established. Target controllers are designed to meet the various objective requirements of laparoscopic surgery. The cumulative value of the target evaluation function of each controller is calculated at the current moment, and the target gradient values ​​of each controller are nested and fused sequentially from low to high according to a preset weight function, and added to the overall control input of the system. The fused motion speed and posture change speed are combined with the joint angles of the surgical robot to achieve autonomous cutting operation. This autonomous cutting method meets the requirements of multi-objective control, ensuring the safety, precision, and efficiency of surgical cutting, while eliminating the dependence on the surgeon's experience and operational skills in traditional cutting methods. Attached Figure Description

[0107] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0108] Figure 1 This is a flowchart illustrating a robot autonomous cutting method for flexible dynamic environments within the body, as provided in an embodiment of the present invention. Detailed Implementation

[0109] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0110] This application provides a robotic autonomous cutting method, system, storage medium, and electronic device for flexible dynamic environments within the body, solving the technical problem that tissue deformation caused by cutting tasks affects the accuracy of the robot in accurately tracking target points.

[0111] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0112] The following technical defects exist in the existing technology:

[0113] (1) The main controller is still designed for specific scenarios, and its portability and generalization are still insufficient;

[0114] (2) Faced with a complex internal environment, the model has difficulty adapting to multiple constraints, and the robot lacks the ability to make autonomous decisions when faced with multiple conflicting objectives.

[0115] (3) In the dynamic internal environment, especially the tissue deformation caused by the cutting task, the accuracy of the robot in tracking the target point will be affected.

[0116] This invention provides a robotic autonomous cutting method for minimally invasive surgery. Addressing the shortcomings of existing autonomous cutting methods in terms of model generalization, dynamic tracking, and intelligent decision-making capabilities, this invention constructs a robotic autonomous cutting method for minimally invasive surgery. It achieves dynamic 3D updates of target points through key point tracking and 3D reconstruction, and uses a multi-target motion fusion method to realize autonomous cutting motion under multiple constraints within the body.

[0117] Specifically, in this embodiment of the invention, the robot can accurately track the cutting path and key tissue areas drawn by the surgeon in a complex and dynamic in vivo environment, ensuring the precision of the cutting. An overall linear control system is constructed and a state equation is established. Target controllers are designed for each objective requirement in the laparoscopic surgery scenario. The cumulative value of the target evaluation function of each controller is calculated at the current moment, and the target gradient values ​​corresponding to each controller are nested and fused sequentially from low to high according to a preset weight function, and added to the overall control input of the system. The fused motion speed and attitude change speed are converted into the joint angles of the surgical robot, realizing autonomous cutting operation. This autonomous cutting method can meet the needs of multi-objective control, ensuring the safety, precision, and efficiency of surgical cutting, while eliminating the dependence on the surgeon's experience and operational skills in traditional cutting methods.

[0118] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0119] Example:

[0120] like Figure 1 As shown, this embodiment of the invention provides a robot autonomous cutting method for flexible dynamic environments within the body, including:

[0121] S1. Read the laparoscopic image, obtain the cutting path and the endpoint of the cutting target marked on the image frame according to the doctor's selection, and locate it in the three-dimensional image.

[0122] S2. Construct an overall linear control system and establish state equations. Design target controllers for each target achievement requirement in the laparoscopic surgery scenario. The target controllers include a planning path tracking controller, a target guidance controller, a cutting depth limit controller, and a collision avoidance controller.

[0123] S3. Establish corresponding motion control prediction models and target evaluation functions for the planning path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. Estimate the motion state of the system in the future for a prediction time interval based on the current system motion state, and calculate the corresponding cumulative value of the target evaluation function.

[0124] S4. Calculate the target gradient of the cumulative value of the target evaluation function of each controller at the current time; and according to the preset weight function, nest and fuse the target gradient values ​​of each controller in order of weight hierarchy from low to high, and add them to the total control input of the system.

[0125] S5. Convert the fused motion speed and posture change speed into the joint angles of the surgical robot.

[0126] In this embodiment of the invention, the robot can accurately track the cutting path and key tissue areas drawn by the doctor in a complex and dynamic environment inside the body, ensuring the accuracy of the cutting. The autonomous cutting method of the robot can meet the needs of multi-target control, ensuring the safety, accuracy and efficiency of surgical cutting, while eliminating the dependence of traditional cutting methods on the doctor's experience and operating skills.

[0127] The following section will detail each step of the above technical solution:

[0128] In step S1, the laparoscopic image is read, and the cutting path and the endpoint of the cutting target marked on the image frame are obtained according to the doctor's selection, and located in the three-dimensional image. Specifically, this includes:

[0129] S11. Read the laparoscopic image and mark the planned cutting path S, the endpoint O of the cutting target, and the critical tissue area A that needs to be avoided on the initial frame image.

[0130] S12. Using the three-dimensional curve tracking method, the cutting path S marked in two dimensions, the endpoint O of the cutting target, and the key tissue region A are located in the three-dimensional image to obtain the planned trajectory Sd, the endpoint Od of the cutting target, and the key tissue region Ad that change dynamically with the image.

[0131] S13. Based on the doctor's selection, set the cutting depth limiting plane Dd on the plane where the planned trajectory is located;

[0132] S14. Measure the first transformation matrix from the instrument's end effector to the robot coordinate system using the optical positioning instrument; measure the transformation matrix from the 3D image of the laparoscope to the robot coordinate system using the camera model and the optical positioning instrument; unify all points to the robot's base coordinate system using the first and second coordinate transformation matrices to obtain the planned trajectory. The endpoint of the cut target is The key organizational areas are The cutting depth is limited by the plane. .

[0133] In step S2, a general linear control system is constructed and a state equation is established. Target controllers are designed for each objective requirement in the laparoscopic surgery scenario. The target controllers include a path planning and tracking controller, a target guidance controller, a cutting depth limiting controller, and a collision avoidance controller. Specifically, they include:

[0134] S21. Construct a general linear control system and establish state equations;

[0135]

[0136]

[0137] in, Let t be the motion state of the system at time t. Let be the velocity of the system at time t. This represents the total control input of the system at time t. Let t be the total control output of the system at time t, and A, B, C, and D be the calculation parameters of the state equations.

[0138] S22. Target controllers are designed to meet the various target achievement requirements in laparoscopic surgery scenarios. The target controllers include a path planning and tracking controller, a target guidance controller, a cutting depth limitation controller, and a collision avoidance controller.

[0139]

[0140] in, For controller The control input is a function of the system's total control output at time t and the desired control objective. To control the desired value at time t.

[0141] It should be noted that the above controller setting principles must meet the requirements of the control objectives and the system stability requirements.

[0142] In this step, in response to the autonomous cutting target requirements in laparoscopic surgery scenarios, a planning path tracking controller, a target guidance controller, a cutting depth limiting controller, and a collision avoidance controller are designed to achieve autonomous cutting under multiple constraints within the energy body.

[0143] Specifically, to ensure that the robot's autonomous cutting process meets the planned trajectory settings, the planned path tracking controller refers to:

[0144]

[0145]

[0146] in, For the input of the path tracking controller, , For the proportional and derivative coefficients of the path tracking controller PD, for The motion state of the path planning and tracking controller is constantly being planned. The deviation between the desired position of the object controlled by the path planning controller and the overall control output of the system is calculated.

[0147] To ensure the robot's tracking speed towards the target point along the cutting path, the control input is a function of the current position and the target position. Specifically, the target guidance controller refers to:

[0148]

[0149]

[0150] in, The input to the target guides the controller. , The proportional and derivative coefficients of the target-guided controller PD are used to control the target. To determine the endpoint of the cut target, The speed at which the target-guided controller moves the object toward the target. The time expected to be taken for the target-guided controller to complete the cutting task;

[0151] To ensure that the depth of tissue cutting by the robot is controlled within a certain range, the cutting depth limit setting controller specifically refers to:

[0152]

[0153]

[0154] in, For the input of the cutting depth limit controller, , The proportional and derivative coefficients of the PD controller are used to limit the cutting depth. The point on the cutting depth limiting plane that is closest to the overall control output. The deviation between the cutting depth limiting plane of the object controlled by the cutting depth limiting controller and the overall control output of the system. for and The angle between vectors;

[0155] To ensure the shortest distance between the device's end effector and the collision avoidance zone, and to maximize the safety of non-target areas, the control input is a function related to the current position and the obstacle's position. Specifically, the collision avoidance control setting controller refers to:

[0156]

[0157]

[0158] in, The input to the collision avoidance controller, , For the proportional and derivative coefficients of the collision avoidance controller PD, Let t be the location of the center point of the critical organizational region. Let the radius of the sphere collision detection area be the center point of the obstacle. Let be the radius of the collision avoidance zone for the sphere, with the center point of the obstacle as the reference point, and let cd(t) be the distance between the total control output of the system and the center point of the obstacle. It is a very small constant.

[0159] In step S3, corresponding motion control prediction models and target evaluation functions are established for the planned path tracking controller, target guidance controller, cutting depth limit controller, and collision avoidance controller, respectively. Based on the current system motion state, the motion state over a future predicted time interval is estimated, and the corresponding cumulative value of the target evaluation function is calculated. Specifically, this includes:

[0160] S31. Establish corresponding motion control prediction models for the planned path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively.

[0161]

[0162]

[0163] in, For the controller at time t within the prediction interval Predicting motion state, For the controller at time t within the prediction interval Predicted motion speed, To predict the control target at time t within the prediction interval Predictive control output;

[0164] S32. Establish target evaluation functions for the planned path tracking controller, target guidance controller, cutting depth limit controller, and collision avoidance controller respectively, and combine them with the corresponding motion control prediction models at the current time. Based on the system's motion state, estimate the motion state over a future predicted time interval and calculate the corresponding cumulative value of the objective evaluation function;

[0165]

[0166] in, For the first Controller The cumulative value of the objective function at time t. For the first Controller prediction range The objective evaluation function value at a single time point within the timeframe. For the predicted time interval.

[0167] It should be noted that the principles for setting the above objective function are: ① the objective function is monotonically increasing; ② the objective function is first-order differentiable; ③ the objective function value is non-negative.

[0168] Specifically, in S32:

[0169] To evaluate the effectiveness of the path planning and tracking control controller within the predicted time interval, an objective evaluation function is established.

[0170]

[0171]

[0172] in, It is a path tracking controller The objective function value at time t. The planned path tracking controller is in the prediction interval arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the planned path tracking control target at time t within the prediction interval;

[0173] To evaluate the effectiveness of the target-guided control system within the predicted time interval, a target evaluation function is established.

[0174]

[0175]

[0176] in, It is a target guidance controller The objective function value at time t. Is the target-guided controller in the prediction interval arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the target guidance control target at time t within the prediction interval;

[0177] A target evaluation function is established for the cutting depth limitation control objective to evaluate the effectiveness of the cutting depth limitation controller within the prediction interval.

[0178]

[0179]

[0180] in, It is a cutting depth limit controller The objective function value at time t. The cutting depth limit controller is within the prediction range. arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the cutting depth limit control target within the prediction interval at time t;

[0181] A target evaluation function is established for the collision avoidance control objective to evaluate the effectiveness of the collision avoidance controller within the prediction range;

[0182]

[0183]

[0184] in, It is a collision avoidance controller The objective function value at time t. Is the collision avoidance controller within the prediction range? arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the collision avoidance control target at time t within the prediction interval. It is a very small constant.

[0185] In step S4, the cumulative value of the target evaluation function of each controller is calculated at the current time as the target gradient; and according to the preset weight function, the target gradient values ​​of each controller are nested and fused in order from low to high according to the weight hierarchy sequence, and added to the total control input of the system.

[0186] S41. Calculate the objective evaluation function of each controller in the current state using an optimized method. The descent gradient at time step;

[0187]

[0188]

[0189]

[0190]

[0191] in, , , , These are the path planning and tracking controller, the target guidance controller, the cutting depth limit controller, and the collision avoidance controller. The gradient descent of the objective evaluation function at time step;

[0192] S42. Input the parameters of each control target and the importance of the control target, and sort the controllers according to the importance of the target to determine the priority of each controller.

[0193]

[0194] in, It is the weight hierarchy sequence of the target controller, satisfying , The total target control quantity. For the number of levels, For the first A set of target controllers at multiple levels.

[0195] Specifically, in this embodiment of the invention (assuming the importance of the four controllers is: path planning and tracking controller > target guidance controller = cutting depth limit controller > collision avoidance controller):

[0196]

[0197] S43. Calculate the target gradient value after fusion by nesting the four controllers in order of weight level from low to high.

[0198]

[0199]

[0200] in, It's about gradients. The normalization function, where α represents the stratification parameter. From 1 to Nested calculation process method.

[0201] Specifically, in the embodiments of the present invention:

[0202]

[0203]

[0204] in, It's about gradients. The normalization function, Indicates the layering parameters, This represents the target gradient value after the nesting and fusion of four controllers.

[0205] S44. The nested and fused target gradient values ​​are summed into the fused controller, thereby realizing the motion fusion of multiple different target motion controllers.

[0206]

[0207] Where K is the proportionality coefficient.

[0208] In step S5, the fused motion speed and attitude change speed are converted into the joint angles of the surgical robot.

[0209] In this step, the fused motion velocity and posture change velocity, specifically the joint angles of the surgical robot, refer to:

[0210]

[0211]

[0212]

[0213]

[0214]

[0215]

[0216] in, Indicates a fixed RCM point. The X-axis vector represents the attitude matrix. The Y-axis vector represents the attitude matrix. The Z-axis vector represents the attitude matrix. For the robot's posture, the function This indicates that the attitude rotation matrix is ​​converted to Euler angles in Cartesian coordinates. Euler angles in Cartesian coordinates The rate of change of Euler angles, The rotational speed of the robot's joint angle. It is a Jacobian matrix.

[0217] In summary, in the autonomous cutting process performed by a laparoscopic surgical robot, the surgeon first draws the cutting path and safe avoidance area on a two-dimensional image and determines the cutting depth. A three-dimensional curve tracking method is used to dynamically update the area drawn by the surgeon on the three-dimensional image. To address the multiple constraints and control objectives faced by the intraoperative electrocoagulation cutting task, multiple target controllers are designed, such as a path planning and tracking controller, a target guidance controller, a cutting depth limiting controller, and a collision avoidance controller. Then, the surgeon inputs the parameters and importance of each control objective, and a multi-objective motion fusion control method is employed to achieve nested fusion of multi-objective control inputs according to a hierarchical weighting from low to high. Finally, the robot executes the control strategy to complete the autonomous electrocoagulation cutting operation.

[0218] This invention provides a robotic autonomous cutting system for flexible dynamic environments within the body, comprising:

[0219] The marking module is used to read laparoscopic images, obtain the cutting path and the endpoint of the cutting target marked on the image frame according to the doctor's selection, and locate them in the three-dimensional image.

[0220] The design module is used to construct the overall linear control system and establish the state equation. For each target achievement requirement in the laparoscopic surgery scenario, a target controller is designed. The target controller includes a planning path tracking controller, a target guidance controller, a cutting depth limit controller, and a collision avoidance controller.

[0221] The prediction module is used to establish corresponding motion control prediction models and target evaluation functions for the planned path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. Based on the system motion state at the current moment, it estimates the motion state of the future for a prediction time interval and calculates the corresponding cumulative value of the target evaluation function.

[0222] The fusion module is used to calculate the target gradient of the cumulative value of the target evaluation function of each controller at the current time; and according to the preset weight function, it nests and fuses the target gradient values ​​of each controller in order of weight hierarchy from low to high, and adds them to the total control input of the system.

[0223] The conversion module is used to convert the fused motion speed and attitude change speed into the joint angles of the surgical robot.

[0224] This invention provides a storage medium storing a computer program for autonomous robot cutting in a flexible, dynamic in vivo environment, wherein the computer program causes a computer to execute the autonomous robot cutting method described above.

[0225] An electronic device, comprising:

[0226] One or more processors;

[0227] Memory; and

[0228] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the robotic autonomous cutting method as described above.

[0229] It is understood that the robot autonomous cutting system, storage medium and electronic device for flexible dynamic environments in vivo provided in the embodiments of the present invention correspond to the robot autonomous cutting method for flexible dynamic environments in vivo provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the robot autonomous cutting method, and will not be repeated here.

[0230] In summary, compared with existing technologies, it has the following beneficial effects:

[0231] In this embodiment of the invention, the robot can accurately track the cutting path and key tissue areas drawn by the doctor in a complex and dynamic environment inside the body, ensuring the accuracy of the cutting. The autonomous cutting method of the robot can meet the needs of multi-target control, ensuring the safety, accuracy and efficiency of surgical cutting, while eliminating the dependence of traditional cutting methods on the doctor's experience and operating skills.

[0232] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0233] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robotic autonomous cutting method for flexible dynamic environments within the body, characterized in that, include: S1. Read the laparoscopic image, obtain the cutting path and the endpoint of the cutting target marked on the image frame according to the doctor's selection, and locate it in the three-dimensional image. S2. Construct an overall linear control system and establish state equations. Design target controllers for each target achievement requirement in the laparoscopic surgery scenario. The target controllers include a planning path tracking controller, a target guidance controller, a cutting depth limit controller, and a collision avoidance controller. S3. Establish corresponding motion control prediction models and target evaluation functions for the planning path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. Estimate the motion state of the system in the future for a prediction time interval based on the current system motion state, and calculate the corresponding cumulative value of the target evaluation function. S4. Calculate the target gradient of the cumulative value of the target evaluation function of each controller at the current time; and according to the preset weight function, nest and fuse the target gradient values ​​of each controller in order of weight hierarchy from low to high, and add them to the total control input of the system. S5. Convert the fused motion speed and attitude change speed into joint angles of the surgical robot; S3 includes: S31. Establish corresponding motion control prediction models for the planned path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. in, For the controller at time t within the prediction interval Predicting motion state, For the controller at time t within the prediction interval Predicted motion speed, To predict the control target at time t within the prediction interval The predictive control output; A, B, C, and D are the calculation parameters of the state equation; For controller The control input is a function of the total control output of the system at time t and the desired control objective; S32. Establish target evaluation functions for the planned path tracking controller, target guidance controller, cutting depth limit controller, and collision avoidance controller respectively, and combine them with the corresponding motion control prediction models at the current time. Based on the system's motion state, estimate the motion state over a future predicted time interval and calculate the corresponding cumulative value of the objective evaluation function; in, For the first Controller The cumulative value of the objective function at time t. For the first Controller prediction range The objective evaluation function value at a single time point within the timeframe. For the predicted time interval; To control the expected value of the target at time t; S4 includes: S41. Calculate the objective evaluation function of each controller in the current state using an optimized method. The descent gradient at time step; in, , , , These are the path planning and tracking controller, the target guidance controller, the cutting depth limit controller, and the collision avoidance controller. The gradient descent of the objective evaluation function at time step; S42. Input the parameters of each control target and the importance of the control target, and sort the controllers according to the importance of the target to determine the priority of each controller; in, It is the weight hierarchy sequence of the target controller; S43. Calculate the target gradient value after fusion by nesting the four controllers in order of weight level from low to high. in, It's about gradients. The normalization function, Indicates the layering parameters, This represents the target gradient value after the nested fusion of four controllers; S44. The nested and fused target gradient values ​​are summed into the fused controller, thereby realizing the motion fusion of multiple different target motion controllers; Where K is the proportionality coefficient.

2. The robot autonomous cutting method as described in claim 1, characterized in that, S1 specifically includes: S11. Read the laparoscopic image and mark the planned cutting path S, the endpoint O of the cutting target, and the critical tissue area A that needs to be avoided on the initial frame image. S12. Using the three-dimensional curve tracking method, the cutting path S marked in two dimensions, the endpoint O of the cutting target, and the key tissue region A are located in the three-dimensional image to obtain the planned trajectory Sd, the endpoint Od of the cutting target, and the key tissue region Ad that change dynamically with the image. S13. Based on the doctor's selection, set the cutting depth limiting plane Dd on the plane where the planned trajectory is located; S14. Measure the first transformation matrix from the instrument's end effector to the robot coordinate system using the optical positioning instrument; measure the transformation matrix from the 3D image of the laparoscope to the robot coordinate system using the camera model and the optical positioning instrument; unify all points to the robot's base coordinate system using the first and second coordinate transformation matrices to obtain the planned trajectory. The endpoint of the cut target is The key organizational areas are The cutting depth is limited by the plane. .

3. The robot autonomous cutting method as described in claim 2, characterized in that, S2 includes: S21. Construct a general linear control system and establish state equations; in, Let t be the motion state of the system at time t. Let be the velocity of the system at time t. This represents the total control input of the system at time t. Let t be the total control output of the system at time t, and A, B, C, and D be the calculation parameters of the state equations. S22. Target controllers are designed to meet the various target achievement requirements in laparoscopic surgery scenarios. The target controllers include a path planning and tracking controller, a target guidance controller, a cutting depth limitation controller, and a collision avoidance controller. in, For controller The control input is a function of the system's total control output at time t and the desired control objective. To control the desired value at time t.

4. The robot autonomous cutting method as described in claim 3, characterized in that, The planned path tracking controller specifically refers to: in, For the input of the path tracking controller, , For the proportional and derivative coefficients of the path tracking controller PD, for The motion state of the path planning and tracking controller is constantly being planned. The deviation between the desired position of the object controlled by the path planning controller and the overall control output of the system is calculated. The target guidance controller specifically refers to: in, The input to the target guides the controller. , The proportional and derivative coefficients of the target-guided controller PD are used to control the target. To determine the endpoint of the cut target, The speed at which the target-guided controller moves the object toward the target. The time expected to be taken for the target-guided controller to complete the cutting task; The cutting depth limit setting controller specifically refers to: in, For the input of the cutting depth limit controller, , The proportional and derivative coefficients of the PD controller are used to limit the cutting depth. The point on the cutting depth limiting plane that is closest to the overall control output. The deviation between the cutting depth limiting plane of the object controlled by the cutting depth limiting controller and the overall control output of the system. for and The angle between vectors; The collision avoidance control setting controller specifically refers to: in, The input to the collision avoidance controller, , For the proportional and derivative coefficients of the collision avoidance controller PD, Let t be the location of the center point of the critical organizational region. Let the radius of the sphere collision detection area be the center point of the obstacle. Let be the radius of the collision avoidance zone for the sphere, with the center point of the obstacle as the reference point, and let cd(t) be the distance between the total control output of the system and the center point of the obstacle. It is a very small constant.

5. The robot autonomous cutting method as described in claim 4, characterized in that, In S32: To evaluate the effectiveness of the path planning and tracking control controller within the predicted time interval, an objective evaluation function is established. in, It is a path tracking controller The objective function value at time t. The planned path tracking controller is in the prediction interval arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the planned path tracking control target at time t within the prediction interval; To evaluate the effectiveness of the target-guided control system within the predicted time interval, a target evaluation function is established. in, It is a target guidance controller The objective function value at time t. Is the target-guided controller in the prediction interval arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the target guidance control target at time t within the prediction interval; A target evaluation function is established for the cutting depth limitation control objective to evaluate the effectiveness of the cutting depth limitation controller within the prediction interval. in, It is a cutting depth limit controller The objective function value at time t. The cutting depth limit controller is within the prediction range. arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the cutting depth limit control target within the prediction interval at time t; A target evaluation function is established for the collision avoidance control objective to evaluate the effectiveness of the collision avoidance controller within the prediction range; in, It is a collision avoidance controller The objective function value at time t. Is the collision avoidance controller within the prediction range? arrive +T represents the objective evaluation function value at a single time point. It is the predictive control output of the collision avoidance control target at time t within the prediction interval. It is a very small constant.

6. The robot autonomous cutting method according to any one of claims 3 to 5, characterized in that, In S5, converting the fused motion velocity and attitude change velocity into the joint angles of the surgical robot specifically refers to: in, Indicates a fixed RCM point. The X-axis vector represents the attitude matrix. The Y-axis vector represents the attitude matrix. The Z-axis vector represents the attitude matrix. For the robot's posture, the function This indicates that the attitude rotation matrix is ​​converted to Euler angles in Cartesian coordinates. Euler angles in Cartesian coordinates The rate of change of Euler angles, The rotational speed of the robot's joint angle. It is a Jacobian matrix.

7. A robotic autonomous cutting system for flexible dynamic environments within the body, characterized in that, For performing the robot autonomous cutting method as described in any one of claims 1 to 6, comprising: The marking module is used to read laparoscopic images, obtain the cutting path and the endpoint of the cutting target marked on the image frame according to the doctor's selection, and locate them in the three-dimensional image. The design module is used to construct the overall linear control system and establish the state equation. For each target achievement requirement in the laparoscopic surgery scenario, a target controller is designed. The target controller includes a planning path tracking controller, a target guidance controller, a cutting depth limit controller, and a collision avoidance controller. The prediction module is used to establish corresponding motion control prediction models and target evaluation functions for the planned path tracking controller, target guidance controller, cutting depth limit controller and collision avoidance controller respectively. Based on the system motion state at the current moment, it estimates the motion state of the future for a prediction time interval and calculates the corresponding cumulative value of the target evaluation function. The fusion module is used to calculate the target gradient of the cumulative value of the target evaluation function of each controller at the current time; and according to the preset weight function, it nests and fuses the target gradient values ​​of each controller in order of weight hierarchy from low to high, and adds them to the total control input of the system. The conversion module is used to convert the fused motion speed and attitude change speed into the joint angles of the surgical robot.

8. A storage medium, characterized in that, It stores a computer program for autonomous robotic cutting in flexible, dynamic environments within the body, wherein the computer program causes the computer to execute the autonomous robotic cutting method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the robotic autonomous cutting method as described in any one of claims 1 to 6.

Images