Control methods and robot systems

Abstract

This invention provides a control method and a surgical system. The control method includes: acquiring an operation command, the operation command being used solely to direct the movement of an instrument; predicting the expected pose of the instrument based on the operation command; calculating the following pose of a surgical field imaging device based at least on the expected pose; and outputting a control command based on the operation command and the following pose, causing the instrument and the surgical field imaging device to move collaboratively. This configuration enables the surgical field imaging device to automatically follow the movement of the instrument without manual operation by the operator, thus solving the problem in existing technologies where frequent switching between instrument control and endoscope control is required, which can affect surgical progress and even surgical safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to a control method and a robot system. BACKGROUND

[0002] In the existing laparoscopic minimally invasive surgery robot system, an operator remotely manipulates the master operating arm of the surgeon console to complete surgical operations such as separation, cutting, and suturing. During the surgical operation, the operator needs to frequently remotely manipulate the master operating arm of the surgeon console to switch to endoscope control to adjust the field of view.

[0003] After the surgical operation area changes, the operator usually needs to remotely manipulate the master control arm to operate the endoscope arm to adjust the field of view of the surgical area, so that the surgical area is more suitable for surgical operation. This method requires the operator to frequently switch between endoscope adjustment and surgical instrument operation, affecting the progress of the surgery; it is difficult to ensure a good surgical field of view at all times during the surgical operation, affecting the safety of the surgical operation; and it is difficult to adjust the operation field of view to the most suitable operation during the surgical operation according to the operator's habits and surgical procedures.

[0004] In summary, in the prior art, it is necessary to frequently switch between the control of the instrument and the control of the endoscope, thereby affecting the progress of the surgery and even affecting the safety of the surgery. SUMMARY

[0005] The present application aims to provide a control method and a robot system to solve the problem in the prior art that the control of the instrument and the control of the endoscope need to be frequently switched, thereby affecting the progress of the surgery and even affecting the safety of the surgery.

[0006] To solve the above technical problems, the present application provides a control method applied to a robot system, wherein the robot system includes an instrument and a surgical field shooting device, the surgical field shooting device is used at least for shooting the instrument, and the control method includes: obtaining an operation instruction, the operation instruction is used only for commanding the instrument to move; predicting an expected pose of the instrument based on the operation instruction; calculating a following pose of the surgical field shooting device based on at least the expected pose; and outputting a control instruction based on the operation instruction and the following pose, so that the instrument and the surgical field shooting device move cooperatively.

[0007] Optionally, the step of calculating the following pose of the surgical field shooting device based on at least the expected pose includes: calculating the following pose based on at least one of the following information in combination with the expected pose:

[0008] The procedure, patient information, human body model, target region, type of the implementation instrument, type of the surgical field imaging device, real-time image collected by the surgical field imaging device, current pose of the implementation instrument, current pose of the surgical field imaging device, model of tissue in the target region, positioning configuration, operator ID and operation stage.

[0009] Optionally, the control method comprises: obtaining a pose adjustment instruction; controlling movement of the surgical field imaging device based on the pose adjustment instruction; and updating the pose following model based at least on the current pose of the implementation instrument and the pose of the surgical field imaging device after movement.

[0010] Optionally, the pose following model used to solve the following pose is learned based on an actor-critic method, and specifically comprises the following steps: setting an actor role and a critic role; the actor role selects a policy function, generates an action based on the policy function and interacts with an environment; the critic role calculates a value function for the actor, which is used to evaluate performance of the actor and guide the action of the actor in the next stage; the critic uses a critic neural network to calculate a time difference error and update network parameters, and an output of the critic neural network is an evaluation value representing a state value or a state-action value; the actor uses an actor neural network to update network parameters, and an output of the actor neural network is a probability distribution of each action; the actor neural network adopts a policy gradient update; and the critic neural network uses a time sequence error to calculate an error update value function between a current state value and a next time state value.

[0011] Optionally, at least one of the following information is added to the learning process in the learning process: the procedure, patient information, human body model, target region, type of the implementation instrument, type of the surgical field imaging device, real-time image collected by the surgical field imaging device, current pose of the implementation instrument, current pose of the surgical field imaging device, model of tissue in the target region, positioning configuration, operator ID and operation stage.

[0012] Optionally, learning data of the pose following model used to solve the following pose is calculated based on a heuristic algorithm. In the learning process, multi-objective classification is performed, and a plurality of weak learners are integrated to obtain a strong learner based on a self-adaptive enhancement method.

[0013] Optionally, the step of controlling the implement and the surgical field imaging device to move cooperatively based on the operation instruction and the following pose output control instruction comprises: obtaining a following trajectory based on the following pose; inputting the following trajectory into a pose adjustment model to obtain a joint planning speed, wherein the joint planning speed is constrained by a lower limit speed and an upper limit speed; the lower limit speed and the upper limit speed are obtained based on a learning process of a pose following model used to solve the following pose; adjusting the pose adjustment model or the joint planning speed based on an evaluation function to obtain a final planning result; and controlling the surgical field imaging device to move based on the planning result.

[0014] Optionally, the control method comprises: determining whether at least a part of the implement is outside the surgical field or is blocked based on machine vision and / or kinematic analysis; and if the determination result is yes, prompting and / or alarming.

[0015] To solve the above technical problems, the present application further provides a robot system, which comprises a controller, an implement and a surgical field imaging device, wherein the surgical field imaging device is used to at least image the implement, and the controller is used to control the implement and the surgical field imaging device to move based on the above control method.

[0016] Optionally, the robot system further comprises a doctor console and a slave operating arm, and the controller comprises a motion control unit, an image processing unit and an optimal visual field learning unit.

[0017] The doctor console unit comprises at least two master operating arms, which are used to obtain the operation instruction. The slave operating arm is used to hold and drive the implement and the surgical field imaging device to move. The motion control unit is used to calculate the motion mapping and control between the master operating arm and the slave operating arm. The image processing unit is used to process the image information of the surgical field imaging device and transmit it to the doctor console or an external display screen to provide the surgical operation image visual field. The optimal visual field learning unit is used to participate in big data learning to obtain and store a pose following model, and solve the following pose of the surgical field imaging device based on the pose following model.

[0018] Compared with the prior art, the control method and the robot system provided by the application comprise: obtaining an operation instruction, the operation instruction is only used to command the implement instrument to move; predicting an expected pose of the implement instrument based on the operation instruction; calculating a following pose of the surgical field shooting device based on at least the expected pose; and outputting a control instruction based on the operation instruction and the following pose, so that the implement instrument and the surgical field shooting device move cooperatively. In this way, the surgical field shooting device can automatically follow the movement of the implement instrument and move, and the operator does not need to manually operate, thereby solving the problem that the implement instrument control and the endoscope control need to be frequently switched in the prior art, and further solving the problem that the surgery progress is affected and even the surgery safety is affected. BRIEF DESCRIPTION OF DRAWINGS

[0019] Those skilled in the art will understand that the provided drawings are for the purpose of better illustrating the present application and do not constitute any limitation on the scope of the present application. Among them:

[0020] Figure 1 is a flowchart of the control method of an embodiment of the present application.

[0021] Figure 2 is a surgical robot assisted surgery application scenario of an embodiment of the present application.

[0022] Figure 3 is a doctor control console diagram of an embodiment of the present application.

[0023] Figure 4 is a surgical trolley and image trolley of an embodiment of the present application.

[0024] Figure 5 is a schematic diagram of a surgical field shooting device of an embodiment of the present application.

[0025] Figure 6a is an optimal surgical field schematic diagram of a kidney partial resection of an embodiment of the present application.

[0026] Figure 6b is an optimal surgical field schematic diagram of a kidney total resection of an embodiment of the present application.

[0027] Figure 7 is a centered optimal surgical field schematic diagram of an embodiment of the present application.

[0028] Figure 8 is a single instrument optimal surgical field schematic diagram of an embodiment of the present application.

[0029] Figure 9 is an active surgical field adjustment logic flow of an embodiment of the present application.

[0030] Figure 10is an optimal surgical field learning process of an embodiment of the present application.

[0031] Figure 11 is an optimal surgical field learning process of an embodiment of the present application based on the actor-critic method.

[0032] Figure 12 is an optimal surgical field learning process of an embodiment of the present application based on the space optimization and adaptive enhancement method.

[0033] Figure 13 is an endoscope automatic following strategy schematic diagram of an embodiment of the present application.

[0034] Figure 14 is an endoscope arm joint trajectory planning process of an embodiment of the present application.

[0035] Figure 15 is a field of view operation prompt schematic diagram of an embodiment of the present application.

[0036] Figure 16 is a master-slave teleoperation principle schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objects, advantages and features of the present application clearer, the following will further describe the present application in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are very simplified and not drawn according to scale, and are only used to facilitate and clarify the purpose of assisting the description of the embodiments of the present application. In addition, the structures shown in the drawings are often a part of the actual structures. In particular, the emphasis of each drawing needs to be different, and sometimes different scales are used.

[0038] The core idea of the present application is to provide a control method and a robot system to solve the problem that the existing technology needs to frequently switch between the control of the implement and the control of the endoscope, thereby affecting the progress of the operation and even affecting the safety of the operation.

[0039] The following is described with reference to the accompanying drawings.

[0040] The present embodiment provides a control method applied to a robot system, the robot system comprising an implement and a surgical field shooting device, the surgical field shooting device being used at least for shooting the implement. The implement refers to all surgical instruments other than the shooting device, and the surgical field shooting device can generally be understood as an endoscope, but other imaging devices such as micro-CT are not excluded.

[0041] Please refer to Figure 1 , the control method comprises:

[0042] S10, an operation instruction is acquired, which is used to instruct the movement of the implement. That is, in the embodiment, the pose adjustment instruction of the surgical field imaging device is not required to control the movement of the surgical field imaging device. In an embodiment, the operation instruction is acquired through the master operating arm, so that the operator can intuitively operate and understand, which is low in cost. However, in other embodiments, the operation instruction can also be acquired based on a button, a keyboard, a mouse, a steering wheel and the like, or generated by an external algorithm and then acquired through a corresponding port / interface.

[0043] S20, an expected pose of the implement is predicted based on the operation instruction.

[0044] S30, a following pose of the surgical field imaging device is calculated based on at least the expected pose.

[0045] And S40, a control instruction is output based on the operation instruction and the following pose, so that the implement and the surgical field imaging device move cooperatively.

[0046] A convenient example is as follows: when the acquired operation instruction is that all the implements are to be translated rightward by 1 mm, the control method concludes through a series of calculations that the surgical field imaging device should also be translated rightward by 1 mm. Then the implement and the surgical field imaging device are simultaneously translated rightward by 1 mm. For the operator, he only operates the implement, but the surgical field also changes accordingly, and the change is even not perceived by him. He only feels that the surgical field is always in an optimal or optimal state during the operation.

[0047] The pose following model used to calculate the following pose is learned based on big data. In the prior art, the surgical field following algorithm is implemented based on an optimization algorithm. This scheme has two disadvantages. First, the optimization algorithm consumes more computing resources and is not real-time, and sometimes may cause the following action to lag and bring adverse effects. Second, under the same parameter working condition, the result obtained by the optimization algorithm is always the same, but under different surgical procedures, different operators and different patient conditions, the optimal scheme may not be fixed (for example, a certain expert doctor may prefer to place the implement near the edge of the surgical field, while another expert doctor may prefer to place the implement at the center of the surgical field). However, the optimization algorithm cannot take these problems into account, and the final following effect cannot be adjusted for different scenes. By learning through big data, on the one hand, the data training process is independent of the control process, and the requirements for computing resources and time are correspondingly reduced. In the implementation process, the pose following model consumes less computing resources and can quickly obtain results. On the other hand, various external information can be taken into account to generate personalized following schemes.

[0048] The application scenario of the embodiment will be introduced as follows.

[0049] like Figure 2 As shown, in a surgical robot-assisted surgery application scenario, the system mainly includes a doctor's console 100, a surgical operating table 110, a vision imaging cart 120, a surgical instrument table 130, an operating bed 140, and an anesthesia cart 150. The doctor, at the doctor's console 100, operates the main operating arm to control the movement of the slave operating arms in the surgical operating table 110. Surgical instruments are mounted on the slave operating arms, thus enabling the operation and control of the surgical instruments. The doctor's console 100 can be located outside the operating room, away from the sterile surgical area. Simultaneously, the visual image of the surgical space inside the patient can be displayed on the screen of the vision imaging cart 120 and the monitor of the doctor's console 100, allowing both the doctor and surgical assistants to directly view the surgical space. The surgical instrument cart 130 is mainly used to hold surgical instruments and other necessary equipment. The anesthesia cart 150 is mainly used for intraoperative anesthesia and vital sign monitoring.

[0050] In this application scenario, the doctor only needs to operate the instrument through the main control arm, while the surgical field imaging device achieves automatic following through control methods.

[0051] like Figure 3 As shown, this embodiment provides a doctor's control console 100, which includes: an adjustment component 103, main operating arms 102, a carriage component 104, and an imaging component 101. The two main operating arms 102 detect the surgeon's hand movements via control handles at their ends, serving as motion control inputs for the entire system. The carriage component 104 is a basic support for mounting other components; the control carriage has movable casters, allowing it to be moved or fixed as needed. A foot switch is installed on the carriage component 104 to detect on / off control signals emitted by the surgeon. The adjustment component 103 can electrically adjust the positions of the main operating arms 102, the imaging component, the operator's handrail, and other devices, i.e., a human-machine parameter adjustment function. The imaging component 101 provides the surgeon with stereoscopic images detected from the imaging system, providing reliable image information for the surgeon's surgical operations. During surgery, the surgeon, seated in front of the control console, is located outside the sterilization area. The surgeon controls the instruments and the laparoscope (i.e., the surgical field imaging device) by operating the control handles at the ends of the main operating arms 102. The surgeon observes the transmitted intracavitary images through the imaging component, and controls the movement of the robotic arms and instruments on the patient's surgical platform with hand movements to complete various operations, thereby achieving the purpose of performing surgery on the patient. At the same time, the surgeon can control some actions through the foot switch, such as completing related operations inputs such as electrocautery and electrocoagulation through the foot switch.

[0052] Operating table 110 (also known as operating cart 200), such as Figure 4As shown. It mainly includes: an adjustment arm 210, a tool arm 201 (i.e., the operating arm), a tool arm 202 (i.e., the operating arm), a tool arm 203 (i.e., the operating arm), an instrument 204, and an endoscope assembly 205 (i.e., the surgical field imaging device). The instrument 204 and the endoscope assembly 205 are mounted at the end of the tool arm, and the surgical instruments and endoscope can be inserted into the body through puncture holes on the surface of the body. The image carriage 300 (also known as the visual image carriage 120) transmits the image information inside the body to the display screen through the endoscope assembly 205. Figure 4 As shown, 301 is the instrument used in the image, and 302 is the field of view at the end of the endoscope. The image carriage 300 of the surgical robot system mainly includes a display device. The endoscope assembly 205 is used to acquire and process images of the surgical space inside the patient; the display device is used to display the images acquired and processed by the endoscope assembly 205 in real time. In the prior art, the surgeon can manipulate the instrument and the endoscope assembly 205 by operating the control handle at the end of the main operating arm 102, thereby completing the corresponding surgical operation.

[0053] The instrument can extend and retract along the axis of the instrument rod; it can rotate around the axis of the instrument rod; the operating mechanism can perform pitching, yaw, and opening / closing movements to achieve various applications in surgical procedures.

[0054] like Figure 5 As shown, the endoscope assembly 205 includes: a connector 251, a transmission line 252, a snap-fit ​​part 253, an endoscope rod 254, and a lens part 255. Generally, the connector 251 is connected to the imaging host to obtain image information; the transmission line 252 is used to directly transmit the image information obtained by the lens part 255 to the imaging host; the snap-fit ​​part 253 is used to snap the endoscope onto the operating arm; the endoscope rod 254 extends into the target area inside the patient's body through a hole fixed to the patient's skin surface; the lens part 255 can obtain image information within the covered area, and the lens surface can form different angles with the endoscope rod to suit different surgical needs.

[0055] In a preferred embodiment, step S30, which involves calculating the following pose of the field-of-field imaging device based at least on the expected pose, includes: calculating the following pose based on at least one of the following information in conjunction with the expected pose:

[0056] Surgical procedure, patient information, human body model, target area, type of the instrument used, type of surgical field imaging device, real-time images acquired by the surgical field imaging device, current pose of the instrument used, current pose of the surgical field imaging device, model of the tissue in the target area, positioning configuration, surgeon's ID, and surgical operation stage.

[0057] Surgical procedure refers to the type of surgery, such as cholecystectomy, lumbar renal cyst decortication, robot-assisted laparoscopic right adrenal lesion resection, adrenal tumor resection, etc. Different follow-up strategies may be required for different surgical procedures.

[0058] Patient information includes, but is not limited to: gender, height, weight, age, etc. If the patient is obese, the surgical space may be narrower, and adjustments to the surgical field should be reduced.

[0059] Human models refer to CT scans taken before surgery, which help to build more accurate surgical area models.

[0060] The types of instruments used include electric scissors, electric hooks, needle holders, etc. When using electric scissors to cut tissue, try to avoid adjusting the surgical field to prevent accidentally touching critical tissues.

[0061] The type of surgical field imaging equipment refers to single-camera, binocular, etc., which can be used to evaluate the accuracy of the obtained surgical field images.

[0062] The practitioner's identity ID is used to record the practitioner's habitual operations, such as a habit of rapid, wide-range operations and a pursuit of visual stability.

[0063] Please refer to Figure 6a and Figure 6b For different surgical procedures, the optimal surgical field of view will vary depending on the positioning configuration, the target area, and the model of the tissues within that target area. For example, in partial nephrectomy... Figure 6a ) and total nephrectomy ( Figure 6b For example, in the diagram, the kidney is represented by 1000, the artery by 1001, the vein by 1002, other vessels by 1003, the partially resected area by 1004, and the endoscopic components by 205. In partial nephrectomy, the optimal surgical field (represented by a dashed line) should coincide with the resection line. In total nephrectomy, due to the handling of critical components such as blood vessels, the center of the endoscopic field (represented by a dashed line) should ideally coincide with the renal vascular connection, and the lens should be kept as close to the target area as possible during the procedure to ensure optimal visualization. This information will be taken into account during the optimal surgical field training process to achieve the best possible surgical field.

[0064] Figure 7 A schematic diagram of a centered optimal surgical field is shown. (For example...) Figure 7As shown, in a surgical operation area scenario, the target area includes tissues 2001, 2002, 2004, and 2005, with the target surgical tissue being 2002; two instruments, 2003 and 2008, are used to perform surgical operations such as cutting, traction, suturing, and coagulation; an endoscope component inserted into the human body is 205, and the target area it illuminates is 2007, which is the image area displayed to the operator; if the surgical objective at this stage is to partially remove tissue 2002 along the dotted line 2006, then the optimal field of view should be that the irradiation area of ​​the endoscope covers the instrument operation area, and the center of the irradiation area coincides with the center of the instrument operation as much as possible. This ensures that the operator can see the two instruments and the operated tissue as clearly as possible.

[0065] Figure 8 A schematic diagram of an optimal surgical field using a single instrument is shown. (For example...) Figure 8 As shown, in a surgical operation area scenario, the tissues in the target area include 2011, 2012, 2015, 2016, and 2018, where the target surgical tissue is 2012, which is covered by other tissues 2018; two instruments, 2014 and 2017, are used to perform surgical operations such as cutting, traction, suturing, and coagulation, where instrument 2017 is mainly used to traction the obstructing tissue 2018 to expose the target surgical tissue 2012; and instruments are inserted into the human body. The endoscope is 205, and the target area it illuminates is 2013, which is the image area displayed to the operator. If the goal of the surgery at this stage is to completely remove tissue 2012, and the instrument 2017 must pull the obstructing tissue 2018, then the removal of tissue 2012 is mainly completed by the instrument 2014. Therefore, the optimal field of view should be that the irradiation area of ​​the endoscope can cover the operating area of ​​the instrument 2014, and the center of the irradiation area should coincide with the tissue removal site as much as possible. This can ensure the best operating field of view for the operator.

[0066] Understandably, the above only illustrates that the "optimal" standard can change under different circumstances. This embodiment does not select different algorithms for following based on different working conditions. Instead, through training with large amounts of data, an effect similar to "selecting different algorithms based on different working conditions" emerges. Essentially, the different following effects are due to the training data covering different working conditions.

[0067] This embodiment incorporates information such as surgical procedure, patient information, human body model, and target area to differentiate between different working conditions, thereby achieving superior performance under each working condition.

[0068] In this embodiment, it is unavoidable that the automatic following solution may not satisfy the operator under certain circumstances. Therefore, please continue to refer to [the relevant documentation / reference needed]. Figure 1The control method includes: S50, acquiring a pose adjustment command; S60, controlling the movement of the surgical field imaging device based on the pose adjustment command; and S70, updating the pose following model based at least on the current pose of the implementing instrument and the pose of the surgical field imaging device after movement.

[0069] Before executing step S50, the current controlled object can be switched via a control target switching command. In this case, the pose adjustment command is still obtained through the main manipulator. Alternatively, other input devices can be set up specifically for obtaining pose adjustment commands. In this case, the operator needs to switch operating methods, such as rotating the body or moving the arm. Since the duration of triggering steps S50-S70 is relatively short, both of the above methods can achieve step S50. For step S70, the update process can be real-time, or the relevant information can be stored first, and a new round of learning process can be triggered at a specific time later.

[0070] Please refer to Figure 9 In one embodiment, steps S50 and S60 are implemented. During master-slave operation, i.e., when the operator operates the master arm to drive the slave instrument, the endoscope arm will automatically adjust according to the above-mentioned optimal endoscopic field of view method to ensure the best surgical field of view. When the operator still wants to adjust the endoscopic field of view manually, the operator only needs to step on the endoscope foot pedal (i.e., an optional target switching command) to perform endoscopic adjustment operations by operating the master arm. At the same time, the system will add relevant data to the database, including but not limited to: target area image information before and after adjustment, instrument and endoscope arm pose before and after adjustment, surgical stage, surgical actions before and after adjustment, etc., to further improve the learning results of the optimal following position of the endoscope and meet the operator's real-time field of view needs as much as possible.

[0071] Pose-following models are learned based on at least one of machine learning, deep learning, reinforcement learning, intelligent algorithms, and meta-learning.

[0072] Please refer to Figure 10 Based on the operator's surgical data, including but not limited to the surgical procedure, target area, the position of instruments and lenses in the target area, tissue model of the target area, and surgical operation stage, a learning algorithm is used to obtain the optimal following position of the endoscope in order to obtain the optimal surgical field of view for different target areas.

[0073] During the preparation phase, surgical-related data is collected, including but not limited to: surgical procedure, patient information, human model, target area, instrument type, endoscope type, instrument pose, endoscope pose, real-time images of the target area, surgical operation stage, surgical robot operation data, positioning configuration, operator information, etc. The above data is then preprocessed, that is, transformed into a more suitable algorithm form.

[0074] During the training phase, certain algorithms are used to identify and analyze the data or explore the implicit relationships between the data in order to find the optimal field of view.

[0075] In the application phase, the optimized model is used to predict the optimal field of view based on data from real-time surgical procedures, and to obtain the target pose of the endoscopic arm corresponding to the optimal field of view.

[0076] The methods that can be used in the above learning process include, but are not limited to: machine learning, deep learning, reinforcement learning, and intelligent optimization methods.

[0077] Please refer to Figure 11 In one embodiment, the pose following model is learned based on the actor-evaluator method, specifically including the following steps:

[0078] Set up actor roles and evaluator roles.

[0079] Actor Role Selection Strategy Function π θ (a|s), and generate actions based on the policy function and interact with the environment.

[0080] The evaluator's role in calculating the value function V for the actor. π (s) are used to evaluate the actor's performance and guide the actor's actions in the next stage.

[0081] The evaluator uses an evaluator neural network to calculate the time difference error and update the network parameters. The output of the evaluator neural network is an evaluation value representing the state value or state-action value.

[0082] The actor uses an actor neural network to update the network parameters, and the output of the actor neural network is the probability distribution of each action.

[0083] The policy gradient update method used in actor neural networks is: Where J(θ) represents the performance of the target policy. Represents the policy gradient, π(a) t |s t ) indicates that in state s t Choose action a t The probability of A π (s t ,a t ) represents the dominant function relative to the benchmark function.

[0084] Furthermore, the evaluator neural network uses time series error to calculate the error update function between the current state value and the state value at the next time step: δ=r+γV π (s t+1 )-V π (s t), where r is the reward at the current moment, γ is the discount factor, and V π (s t+1 V is the state value at the next moment. π (s t ) is the current state value.

[0085] Furthermore, during the learning process, at least one of the following information is incorporated: surgical procedure, patient information, human model, target area, type of instrument used, type of surgical field imaging device, real-time images acquired by the surgical field imaging device, current pose of the instrument used, current pose of the surgical field imaging device, model of tissue in the target area, positioning configuration, surgeon ID, and surgical procedure stage.

[0086] In another embodiment, the learning data for the pose-following model is calculated based on a heuristic algorithm. Preferably, during the learning process, multi-object classification is performed, and multiple weak learners are integrated to obtain a strong learner based on an adaptive enhancement method.

[0087] Please refer to Figure 12 In an optimal field-of-view learning method based on spatial optimization and adaptive enhancement, the current pose P of the endoscope is used as the basis for the learning. endo Define the endoscope adjustment space area R as the midpoint. e , where P endo =[p x p y p z r x r y r z ] T Region R e For P endo The adjustment space region is a sphere with center r and radius r (r can be set empirically), or it can be set in other ways; within the endoscope adjustment space region R... e Internally, heuristic intelligent algorithms can be used to find the optimal endoscopic pose for the field of view, with the optimization objective being:

[0088]

[0089] Where N represents the number of influence quantities of the optimal field of view, w i G(x) represents the weighting factor of the i-th influence quantity. i ) represents the calculated value of the i-th influence quantity, such as the distance between the center point of the image and the center point of the instrument.

[0090] Based on the endoscope pose with the optimal field of view obtained by the above method, an adaptive boosting method (Ada-boosting) is used to further evaluate the effect of the optimal field of view to ensure the global optimal field of view. The adaptive boosting method integrates multiple weak learners to obtain a strong learner, which can perform better training and prediction for multi-object classification.

[0091] The above learning process may require coordinate transformations. An instrument is mounted on an operating arm; let its coordinate system be {Ot1} and its base coordinate system be {Ob1}. An endoscope is mounted on another operating arm; let its coordinate system be {Oe} and its base coordinate system be {Ob2}. Let the instrument's pose in the base coordinate system be Tob1_ot1, the transformation matrix of arm 1's base coordinate system {Ob1} relative to arm 2's base coordinate system {Ob2} be Tob2_ob1, and the transformation matrix of the endoscope arm's base coordinate system {Ob2} relative to the endoscope's base coordinate system {Oe} be Toe_ob2. Based on the pose transformation relationships, the instrument's pose in the endoscope's field of view can be obtained as follows:

[0092] Toe_ot1=Toe_ob2*Tob2_ob1*Tob1_ot1

[0093] Based on the position and pose of the instruments in the endoscopic field of view, combined with information such as the focal length and magnification of the endoscope, the relative relationship of the instruments in the image obtained by the endoscope can be calculated. Therefore, the above information can be directly used for optimal field of view training and learning.

[0094] Further, please refer to Figure 13 Step S40, the step of coordinating the movement of the instrument and the surgical field imaging device based on the operation command and the following pose output control command, includes: obtaining the following trajectory based on the following pose; inputting the following trajectory into the pose adjustment model to obtain the joint planning velocity, where v jnlim_i ≤v jnt_i ≤v jplim_i v jnt_i Indicates the joint planning velocity, v jnlim_i Represents the lower limit of velocity, v jplim_i The upper limit velocity represents the lower limit velocity; the upper and lower limit velocities are obtained based on the learning process of the pose following model; the pose adjustment model or joint planning velocity is adjusted based on the evaluation function to obtain the final planning result, wherein the evaluation function is calculated according to the following formula:

[0095]

[0096] Among them, v c The system represents the joint planning speed; q represents the comfort level of the surgical field; data from n cycles are used for evaluation; and the movement of the surgical field imaging device is controlled based on the planning results.

[0097] Among them, the following trajectory is according to Figure 14 The calculation is performed as shown. After obtaining the optimal field of view of the target, i.e. the optimal Cartesian pose of the endoscope arm, the target joint position of the endoscope arm is obtained through inverse kinematics calculation; inverse kinematics can be performed using analytical or numerical methods.

[0098] By setting the initial joint position and the target joint position, and setting the maximum joint velocity and the maximum joint acceleration, a time-based joint position trajectory can be calculated using a trajectory planning algorithm. The trajectory planning algorithm includes, but is not limited to, polynomial interpolation planning, trigonometric interpolation planning, and sigmoid interpolation planning.

[0099] This example uses fifth-order polynomial programming, employing fifth-order polynomial interpolation. Here, t represents time, a0, a1, a2, a3, a4, and a5 are coefficients, and q(t) represents the planned value.

[0100] q(t) = a5t 5 +a4t 4 +a3t 3 +a2t 2 +a1t+a0

[0101] The constraints are set as follows:

[0102] q(0)=q s The starting position q s .

[0103] q(t f )=q f End time t f Position q f .

[0104] Velocity at start and end:

[0105]

[0106] Acceleration at the start and end times:

[0107]

[0108] Maximum speed limit

[0109] Maximum acceleration limit

[0110] The coefficient values ​​can be obtained as follows:

[0111] a0 = q s a1 = a2 = 0

[0112] Furthermore, the control method includes: determining, based on machine vision and / or kinematic analysis, whether at least a portion of the instrument is outside the surgical field or obstructed; and, if the determination result is yes, providing a prompt and / or alarm.

[0113] In one embodiment, such as Figure 15 As shown, when a surgical instrument is not in the surgical field of view or is obstructed by tissue, a message can be displayed on the screen, such as "Right-hand instrument is obstructed, please pay attention to operation safety".

[0114] This embodiment also provides a surgical system, which includes a controller, an instrument, and a surgical field imaging device. The surgical field imaging device is used to at least image the instrument, and the controller is used to control the movement of the instrument and the surgical field imaging device based on the control method described above.

[0115] Optionally, the surgical system also includes a surgeon's console and slave manipulators. The controller includes a motion control unit, an image processing unit, and an optimal field of view learning unit. The surgeon's console unit includes at least two master manipulators, which are used to acquire operating commands. The slave manipulators are used to hold and drive the instruments and surgical field imaging equipment. The motion control unit is used to calculate and control the motion mapping between the master and slave manipulators. The image processing unit is used to process the image information from the surgical field imaging equipment and transmit it to the surgeon's console or an external display screen to provide the surgical operation image field of view. The optimal field of view learning unit is used to participate in big data learning, obtain and store the pose following model, and calculate the following pose of the surgical field imaging equipment based on the pose following model.

[0116] The master and slave manipulators are controlled according to the master-slave teleoperation principle, such as... Figure 16 As shown. In one embodiment of the master-slave teleoperation principle, the operator can operate the master operating end (including the master operating arm). Through master-slave mapping, the operator's actions can be remotely controlled to achieve corresponding movement operations on the slave operating arm end (including the slave operating arm). Simultaneously, the slave operating end transmits image information of the operating area to the doctor's console for the operator to view. The operator can perform movement operations on the slave end through master-slave operation. The slave end can carry instruments or endoscopes, etc. That is, the operator can operate the master operating arm to drive the slave end instruments to perform corresponding surgical operations; at the same time, the operator can also operate the master operating arm to drive the slave end end endoscope to achieve corresponding movements to adjust the observed surgical operation field of view. Therefore, during the surgical operation, the operator may need to frequently switch between instrument operation and endoscope adjustment to achieve a better operating field of view and surgical actions.

[0117] In summary, this embodiment provides a control method and a surgical system. The control method includes: acquiring an operation command, which is used solely to direct the movement of the surgical instrument; predicting the expected pose of the surgical instrument based on the operation command; calculating the following pose of the surgical field imaging device based at least on the expected pose; and outputting a control command based on the operation command and the following pose, causing the surgical instrument and the surgical field imaging device to move collaboratively. This configuration enables the surgical field imaging device to automatically follow the movement of the surgical instrument without manual operation by the operator, thus solving the problem in existing technologies where frequent switching between controlling the surgical instrument and controlling the endoscope is required, which can affect surgical progress and even surgical safety.

[0118] The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A robotic system, characterized by, The robot system comprises a controller, an implement instrument, and a surgical field shooting device used at least for shooting the implement instrument, the controller is used for controlling the implement instrument and the surgical field shooting device to move based on a control method, the control method comprises: obtaining an operation instruction, the operation instruction is used only for instructing the implement instrument to move; predicting an expected pose of the implement instrument based on the operation instruction; calculating a following pose of the surgical field shooting device based on at least the expected pose; and outputting a control instruction based on the operation instruction and the following pose, so that the implement instrument and the surgical field shooting device move cooperatively; wherein the step of outputting the control instruction based on the operation instruction and the following pose, so that the implement instrument and the surgical field shooting device move cooperatively, comprises: obtaining a following trajectory based on the following pose; inputting the following trajectory into a pose adjustment model to obtain a joint planning speed; adjusting the pose adjustment model or the joint planning speed based on an evaluation function to obtain a final planning result; and controlling the surgical field shooting device to move based on the planning result.

2. The robotic system of claim 1, wherein, The step of calculating the following pose of the surgical field shooting device based on at least the expected pose comprises: calculating the following pose based on at least one of the following information in combination with the expected pose: a surgical type, patient information, a human body model, a target region, a type of the implement instrument, a type of the surgical field shooting device, a real-time image collected by the surgical field shooting device, a current pose of the implement instrument, a current pose of the surgical field shooting device, a model of tissue in the target region, a positioning configuration, an operator ID, and an operation stage.

3. The robotic system of claim 1, wherein, The control method comprises: obtaining a pose adjustment instruction; controlling the surgical field shooting device to move based on the pose adjustment instruction; and updating a pose following model based on at least a current pose of the implement instrument and a pose of the surgical field shooting device after moving; the pose following model is used for calculating the following pose.

4. The robotic system of claim 3, wherein, The control method further comprises: before obtaining the pose adjustment instruction, switching a current control object by a control target switching instruction; and / or The process of updating the pose following model is real-time, or relevant information is stored in advance and a new round of learning process is triggered at a preset time point.

5. The robotic system of claim 1, wherein, The pose following model used for calculating the following pose is learned based on an actor-critic method, and specifically comprises the following steps: setting an actor role and a critic role; the actor role selects a policy function, generates an action based on the policy function, and interacts with an environment; the critic role calculates a value function for the actor, which is used for evaluating the performance of the actor and guiding the action of the actor in the next stage; the critic uses a critic neural network to calculate a time difference error and update network parameters, and an output of the critic neural network is an evaluation value representing a state value or a state-action value; the actor uses an actor neural network to update network parameters, and an output of the actor neural network is a probability distribution of each action. The actor neural network adopts a policy gradient update; and The critic neural network uses time series errors to calculate error update value functions between current state values and next time state values.

6. The robotic system of claim 5, wherein, During the learning process, at least one of the following information is added to the learning process: Surgical procedure, patient information, human body model, target area, type of implementation instrument, type of surgical field imaging device, real-time image collected by the surgical field imaging device, current pose of the implementation instrument, current pose of the surgical field imaging device, model of tissue in the target area, positioning configuration, operator ID and operation stage.

7. The robotic system of claim 6, wherein, The operator ID is used to record the operator's habitual operation, so as to solve the problem that the optimal following scheme corresponding to different operators is not fixed.

8. The robotic system of claim 1, wherein, The learning data of the pose following model used to solve the following pose is calculated based on a heuristic algorithm; During the learning process, multi-target classification is performed, and a strong learner is obtained by integrating multiple weak learners based on an adaptive enhancement method.

9. The robotic system of claim 1, wherein, In the step of obtaining joint planning speed by the following trajectory input pose adjustment model, the joint planning speed is constrained by a lower limit speed and an upper limit speed; the lower limit speed and the upper limit speed are obtained based on the learning process of the pose following model; and the pose following model is used to solve the following pose.

10. The robotic system of claim 1, wherein, The control method comprises: Based on machine vision and / or kinematic analysis, it is determined whether at least part of the implementation instrument is outside the surgical field or is blocked; and If the determination result is yes, a prompt and / or alarm is given.

11. The robotic system of claim 1, wherein, The robot system further comprises a doctor console and a slave operating arm, and the controller comprises a motion control unit, an image processing unit and an optimal visual field learning unit, wherein The doctor console unit comprises at least two master operating arms, and the master operating arms are used to obtain the operation instruction; The slave operating arm is used to hold and drive the implementation instrument and the surgical field imaging device to move; The motion control unit is used to calculate and control the motion mapping between the master operating arm and the slave operating arm; The image processing unit is used to process the image information of the surgical field imaging device and transmit it to the doctor console or an external display screen to provide a surgical operation image visual field; The optimal visual field learning unit is used to participate in big data learning, obtain and store a pose following model, and solve the following pose of the surgical field imaging device based on the pose following model.

Images