Control method and robot system

Abstract

The invention provides a control method and a surgical system. The control method comprises the steps that an operation instruction is obtained, and the operation instruction is only used for instructing the implementation instrument to move; predicting an expected pose of an implementation instrument based on the operation instruction; solving a following pose of the field shooting equipment at least based on the expected pose; and outputting a control instruction based on the operation instruction and the follow-up pose, so that the implementation instrument and the surgical field shooting equipment cooperatively move. By means of the configuration, the operation field shooting equipment can automatically move along with movement of the implementation instrument, manual operation of an operator is not needed, and therefore the problem that in the prior art, frequent switching between control over the implementation instrument and control over an endoscope is needed, and then the operation progress and even the operation safety are affected is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a control method and a robot system. Background Art

[0002] In the existing laparoscopic minimally invasive surgical robot system, an operator is located away from the sterile area and remotely controls the main operating arm of the doctor's console to complete surgical operations such as separation, cutting, and suturing. During the surgical operation, the operator often needs to remotely control the main operating arm of the doctor's console to switch to endoscope control to achieve operations such as field of view adjustment.

[0003] After the surgical operation area is changed, the operator is usually required to remotely operate the main control arm to operate the endoscope arm to adjust the surgical field of view, so that the surgical area is more suitable for surgical operation. This method requires the operator to frequently switch between endoscope adjustment and surgical instrument operation, affecting the progress of the operation; it is difficult to ensure that there is always a good surgical field of view during the surgical operation, affecting the safety of the surgical operation; during the surgical operation, it is difficult to adjust to the most suitable operating field of view for the operation according to the operator's habits and surgical procedures.

[0004] In summary, in the prior art, it is necessary to frequently switch between the control of the instrument and the control of the endoscope, thereby affecting the progress of the operation and even affecting the safety of the operation. Summary of the invention

[0005] The object of the present invention is to provide a control method and a robot system to solve the problem in the prior art that it is necessary to frequently switch between the control of the implementation instrument and the control of the endoscope, thereby affecting the progress of the operation and even affecting the safety of the operation.

[0006] In order to solve the above technical problems, the present invention provides a control method applied to a robot system, wherein the robot system includes an implementation instrument and a surgical field shooting device, wherein the surgical field shooting device is at least used to shoot the implementation instrument, and the control method includes: obtaining an operation instruction, wherein the operation instruction is only used to command the movement of the implementation instrument; predicting the expected posture of the implementation instrument based on the operation instruction; solving the following posture of the surgical field shooting device at least based on the expected posture; and outputting a control instruction based on the operation instruction and the following posture so that the implementation instrument and the surgical field shooting device move in coordination.

[0007] Optionally, the step of calculating the following posture of the surgical field shooting device at least based on the expected posture includes: calculating the following posture based on at least one of the following information in combination with the expected posture:

[0008] The surgical procedure, patient information, human body model, target area, the type of the implementation instrument, the type of the surgical field shooting device, the real-time image collected by the surgical field shooting device, the current position of the implementation instrument, the current position of the surgical field shooting device, the model of the tissue in the target area, the positioning configuration, the operator's identity ID and the operation stage.

[0009] Optionally, the control method includes: obtaining a posture adjustment instruction; controlling the movement of the surgical field capturing device based on the posture adjustment instruction; and updating the posture following model based at least on the current posture of the implementation instrument and the posture of the surgical field capturing device after movement.

[0010] Optionally, a posture following model for solving the following posture is learned based on the actor-evaluator method, and specifically includes the following steps: setting an actor role and an evaluator role; the actor role selects a policy function, generates actions based on the policy function and interacts with the environment; the evaluator role calculates a value function for the actor, which is used to evaluate the performance of the actor and guide the actor's actions in the next stage; the evaluator uses an evaluator neural network to calculate a time difference error and update network parameters, and the output of the evaluator neural network is an evaluation value representing a state value or a state-action value; the actor uses an actor neural network to update network parameters, and the output of the actor neural network is a probability distribution of each action; the actor neural network uses a policy gradient update; and the evaluator neural network uses a time series error to calculate the error between the current state value and the state value at the next moment to update the value function.

[0011] Optionally, during the learning process, at least one of the following information is added to the learning process: surgical procedure, patient information, human body model, target area, type of the implementation instrument, type of the surgical field shooting device, real-time images captured by the surgical field shooting device, current position of the implementation instrument, current position of the surgical field shooting device, model of tissue in the target area, positioning configuration, operator identity ID and operation stage.

[0012] Optionally, the learning data of the posture following model for solving the following posture is calculated based on a heuristic algorithm. In the learning process, multi-target classification is performed, and multiple weak learners are integrated to obtain a strong learner based on an adaptive enhancement method.

[0013] Optionally, based on the operation instructions and the following posture output control instructions, the step of making the implementation instrument and the surgical field shooting device move in coordination includes: obtaining a following trajectory based on the following posture; inputting the following trajectory into a posture adjustment model to obtain a joint planning speed, wherein the joint planning speed is constrained by a lower limit speed and an upper limit speed; the lower limit speed and the upper limit speed are obtained based on a learning process of a posture following model; the posture following model is used to solve the following posture; adjusting the posture adjustment model or the joint planning speed based on an evaluation function to obtain a final planning result; and controlling the movement of the surgical field shooting device based on the planning result.

[0014] Optionally, the control method includes: judging whether at least a portion of the implementation instrument is outside the surgical field or is blocked based on machine vision and / or kinematic analysis; and, if the judgment result is yes, giving a prompt and / or an alarm.

[0015] In order to solve the above technical problems, the present invention also provides a robot system, which includes a controller, an implementation instrument and a surgical field shooting device. The surgical field shooting device is at least used to shoot the implementation instrument, and the controller is used to control the movement of the implementation instrument and the surgical field shooting device based on the above control method.

[0016] Optionally, the robot system further includes a doctor's console and a slave operating arm, and the controller includes a motion control unit, an image processing unit and an optimal field of view learning unit.

[0017] Wherein, the doctor's console unit includes at least two main operating arms, and the main operating arms are used to obtain the operating instructions. The slave operating arms are used to clamp and drive the implementation instrument and the surgical field shooting device to move. The motion control unit is used to calculate the motion mapping and control between the main operating arms and the slave operating arms. The image processing unit is used to process the image information of the surgical field shooting device and transmit it to the doctor's console or an external display screen to provide a surgical operation image field of view. The optimal field of view learning unit is used to participate in big data learning, obtain and store a posture following model, and solve the following posture of the surgical field shooting device based on the posture following model.

[0018] Compared with the prior art, the present invention provides a control method and a robot system, wherein the control method includes: obtaining an operation instruction, the operation instruction is only used to command the movement of the implementation instrument; predicting the expected posture of the implementation instrument based on the operation instruction; solving the follow-up posture of the arithmetic field shooting device based on at least the expected posture; and outputting a control instruction based on the operation instruction and the follow-up posture, so that the implementation instrument and the surgical field shooting device move in coordination. With such a configuration, the surgical field shooting device can automatically move with the movement of the implementation instrument without the operator having to operate manually, thereby solving the problem in the prior art of frequently switching between the control of the implementation instrument and the control of the endoscope, thereby affecting the progress of the operation and even affecting the safety of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0020] Figure 1 It is a flowchart of a control method according to an embodiment of the present invention.

[0021] Figure 2 This is an application scenario of surgical robot-assisted surgery according to an embodiment of the present invention.

[0022] Figure 3 Schematic diagram of a doctor's console according to an embodiment of the present invention.

[0023] Figure 4 The present invention relates to an operating table trolley and an imaging table trolley according to an embodiment of the present invention.

[0024] Figure 5 It is a schematic diagram of a surgical field photographing device according to an embodiment of the present invention.

[0025] Figure 6a It is a schematic diagram of the optimal surgical field for partial nephrectomy according to an embodiment of the present invention.

[0026] Figure 6b Schematic diagram of the optimal surgical field for radical nephrectomy according to an embodiment of the present invention.

[0027] Figure 7 It is a schematic diagram of the optimal centered surgical field according to an embodiment of the present invention.

[0028] Figure 8 It is a schematic diagram of the optimal surgical field of a single instrument according to an embodiment of the present invention.

[0029] Fig. 9 This is the logic flow of active surgical field adjustment according to an embodiment of the present invention.

[0030] Fig.10This is the optimal surgical field learning process of an embodiment of the present invention.

[0031] Fig.11 This is an optimal surgical field learning process based on the actor-evaluator method according to an embodiment of the present invention.

[0032] Fig.12 This is an optimal surgical field learning process based on spatial optimization and adaptive enhancement methods according to an embodiment of the present invention.

[0033] Fig.13 It is a schematic diagram of an automatic following strategy for an endoscope according to an embodiment of the present invention.

[0034] Fig.14 This is an endoscope arm joint trajectory planning process according to an embodiment of the present invention.

[0035] Fig.15 It is a schematic diagram of a visual field operation prompt according to an embodiment of the present invention.

[0036] Fig.16 Schematic diagram of the master-slave remote operation principle of an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the drawings is often a part of the actual structure. In particular, the emphasis of each drawing is different, and sometimes different scales are used.

[0038] The core idea of ​​the present invention is to provide a control method and a robot system to solve the problem in the prior art that it is necessary to frequently switch between the control of the implementation instrument and the control of the endoscope, thereby affecting the progress of the operation and even affecting the safety of the operation.

[0039] The following description is given with reference to the accompanying drawings.

[0040] This embodiment provides a control method, which is applied to a robot system, wherein the robot system includes an implementation instrument and an operation field photographing device, wherein the operation field photographing device is at least used to photograph the implementation instrument. The implementation instrument refers to all surgical instruments other than the photographing device, and the operation field photographing device can generally be understood as an endoscope, but imaging devices based on other principles, such as micro CT, etc., are not excluded.

[0041] Please refer to Figure 1 , control methods include:

[0042] S10, obtain the operation instruction, the operation instruction is only used to command the movement of the instrument. That is, in this embodiment, the movement of the surgical field shooting device can be controlled without obtaining the corresponding posture adjustment instruction of the surgical field shooting device. In one embodiment, the operation instruction is obtained by the main operating arm, so that the operator can perform intuitive operation with low understanding cost. However, in other embodiments, the operation instruction can also be obtained based on devices such as buttons, keyboards, mice, steering wheels, etc., or the operation instruction can be generated by an external algorithm and then obtained through the corresponding port / interface.

[0043] S20, predicting the expected posture of the implementation device based on the operation instruction.

[0044] S30, solving a following posture of the arithmetic field shooting device based at least on the expected posture.

[0045] And, S40, based on the operation instruction and the follow-up posture output control instruction, the implementation instrument and the surgical field shooting device move in coordination.

[0046] An example that is easy to understand is as follows: When the operation instruction obtained is: all the implementation instruments are translated to the right by 1 mm. Then the control method concludes through a series of calculations that the surgical field photography device should also be translated to the right by 1 mm. Then the implementation instruments and the surgical field photography device are translated to the right by 1 mm at the same time. For the operator, he only operates the implementation instruments, but the surgical field also changes accordingly, and this change is not even perceived by him. He only feels that the surgical field is always in a better or optimal state during the operation.

[0047] Among them, the posture following model used to solve the following posture is obtained based on big data learning. In the prior art, the surgical field following algorithm is implemented based on the optimization algorithm. This solution has two disadvantages. First, the optimization algorithm consumes more computing resources and is not real-time, which may sometimes cause the following action to lag and bring counterproductive effects. Second, under the same parameter conditions, the results obtained by the optimization algorithm are always the same, but under different surgical procedures, different operators and different patient conditions, the optimal solution may not be fixed (for example, an expert doctor may prefer to place the implementation instrument near the edge of the surgical field, while another expert doctor may prefer to place the implementation instrument in the center of the surgical field). However, the optimization algorithm cannot take these problems into account, and the final following effect cannot be adjusted for different scenarios. Through big data learning, on the one hand, the data training process is independent of the control process, and the requirements for computing resources and time are correspondingly reduced. During the implementation process, the posture following model consumes less computing resources and can quickly obtain results; on the other hand, a variety of external information can be taken into account to generate a personalized following solution.

[0048] The application scenarios of this embodiment are introduced as follows.

[0049] like Figure 2 As shown, in a surgical robot-assisted surgery application scenario, it mainly includes a doctor's console 100, a surgical operating table 110, a visual image trolley 120, a surgical instrument table 130, an operating bed 140, and an anesthesia trolley 150. The doctor can realize the motion control of the slave operating arm in the surgical operating table 110 by operating the main operating arm at the doctor's console 100, and the surgical instrument is installed on the slave operating arm, so as to realize the operation control of the surgical instrument; the doctor's console 100 can be located outside the operating room, away from the surgical sterile area; at the same time, the visual image of the surgical space in the patient's body can be displayed on the display screen of the visual image trolley 120 and the display of the doctor's console 100 by the visual image trolley 120, so that the doctor and the auxiliary surgical personnel can directly view the situation of the surgical space; the surgical instrument trolley 130 is mainly used to place surgical instruments and other equipment required for surgery; the anesthesia trolley 150 is mainly used to realize intraoperative anesthesia and vital sign monitoring.

[0050] In this application scenario, the doctor only needs to operate the instrument through the main operating arm, while the surgical field shooting equipment achieves automatic following effect through control methods.

[0051] like Figure 3 As shown, this embodiment provides a doctor console 100, which includes: an adjustment component 103, a main operating arm 102, a trolley component 104, and an image component 101. The two main operating arms 102 detect the operator's hand motion information through the control handle at the end thereof as the motion control input of the entire system; the trolley component 104 is a basic bracket for installing other components, and the control trolley has movable casters and can be moved or fixed as needed; a foot switch is installed on the trolley component 104 to detect the switch control signal sent by the operator; the adjustment component 103 can electrically adjust the position of the main operating arm 102, the image component, the operator's armrest and other devices, that is, the human-machine parameter adjustment function. The image component 101 can provide the operator with a stereoscopic image detected from the image system, and provide the operator with reliable image information for performing surgical operations. During the operation, the operator sitting in front of the doctor's console is located outside the disinfection area, and the operator controls the implementation instrument and laparoscope (that is, the surgical field shooting device) by operating the control handle at the end of the main operating arm 102. The surgeon observes the intracavitary images transmitted back by the imaging component and uses both hands to control the movement of the robotic arm and instruments of the patient's surgical platform to complete various operations, thereby achieving the purpose of performing surgery on the patient. At the same time, the surgeon can control some actions through the foot switch, such as completing electroresection, electrocoagulation and other related operation inputs through the foot switch.

[0052] The surgical table 110 (also referred to as the surgical trolley 200) is Figure 4As shown. It mainly includes: an adjustment arm 210, a tool arm 201 (i.e., a slave operating arm), a tool arm 202 (i.e., a slave operating arm), a tool arm 203 (i.e., a slave operating arm), an implementation instrument 204, and an endoscope assembly 205 (i.e., an operation field photographing device). The implementation instrument 204 and the endoscope assembly 205 are installed at the end of the tool arm, and the surgical instrument and the endoscope can be inserted into the body through the poking holes on the surface of the human body. The image information inside the body is transmitted to the display screen by the image trolley 300 (also called the visual image trolley 120) through the endoscope assembly 205. As shown Figure 4 As shown, 301 is the implementation instrument in the image, and 302 is the end field of view of the endoscope. The image trolley 300 of the surgical robot system mainly includes: a display device. The endoscope assembly 205 is used to obtain and process the image of the surgical space in the patient's body; the display device is used to display the image acquired and processed by the endoscope assembly 205 in real time. In the prior art, the operator can operate the implementation instrument and the endoscope assembly 205 by operating the control handle at the end of the main operating arm 102, thereby completing the corresponding surgical operation.

[0053] The implementing instrument can be telescopically moved along the axis of the instrument rod; it can be rotated around the direction of the instrument rod; the operating mechanism can perform pitch movement, yaw movement and opening and closing movement, so as to realize various applications in surgical operations.

[0054] like Figure 5 As shown, the endoscope assembly 205 includes: a connector 251, a transmission line 252, a clamping part 253, an endoscope rod 254, and a lens part 255. Generally, the connector 251 is connected to the image host to obtain image information; the transmission line 252 is used to directly transmit the image information obtained by the lens part 255 to the image host; the clamping part 253 can be used to clamp the endoscope on the slave operating arm; the endoscope rod 254 extends into the target area in the patient's body through a hole fixed on the surface of the patient's skin; the lens part 255 can obtain image information in the covered area, and the lens surface can form different angles with the endoscope rod to adapt to different surgical needs.

[0055] In a preferred embodiment, step S30, the step of solving the following posture of the arithmetic field shooting device based at least on the expected posture comprises: solving the following posture based on at least one of the following information in combination with the expected posture:

[0056] The surgical procedure, patient information, human body model, target area, the type of the implementation instrument, the type of the surgical field shooting device, the real-time image captured by the surgical field shooting device, the current position of the implementation instrument, the current position of the surgical field shooting device, the model of the tissue in the target area, the positioning configuration, the operator's identity ID and the surgical operation stage.

[0057] The surgical procedure refers to the type of surgery, such as cholecystectomy, translumbar renal cyst removal, robot-assisted laparoscopic right adrenal lesion resection, adrenal tumor resection, etc. Different follow-up strategies may be required in different surgical procedures.

[0058] Patient information includes but is not limited to: gender, height, weight, age, etc. If the patient is obese, the surgical space may be narrow and the surgical field adjustment should be reduced.

[0059] The human body model refers to the CT model of the patient taken before surgery, which helps to build a more accurate model of the surgical area.

[0060] The types of instruments used include electric scissors, electric hooks, needle holders, etc. If electric scissors are used to cut tissue, try to avoid adjusting the surgical field to avoid accidentally touching critical tissues.

[0061] The types of surgical field photography equipment include single camera, binocular camera, etc., which can be used to evaluate the accuracy of the surgical field images obtained.

[0062] The operator's identity ID is used to record the operator's habitual operations, such as the habit of fast and large-scale operations and the pursuit of visual field stability.

[0063] Please refer to Figure 6a and Figure 6b For different surgical procedures, the optimal surgical field of view will be different according to the positioning configuration, the target area, the model of the tissue in the target area, and other information. Figure 6a ) and radical nephrectomy ( Figure 6b ) as an example, in the figure, the kidney is 1000, the artery is 1001, the vein is 1002, the other pipelines are 1003, the partial resection area is 1004, and the endoscope component is 205. In partial nephrectomy, the optimal surgical field (indicated by the dotted line) should be that the center of the endoscope field of view coincides with the resection line; in total nephrectomy, due to the involvement of key components such as blood vessels, the center of the endoscope field of view (indicated by the dotted line) should be as close as possible to the kidney vascular connection, and the lens and the target area should be as close as possible during the operation to ensure the best visual effect. In the process of optimal surgical field training and learning, this information will be taken into account to achieve the optimal surgical field.

[0064] Figure 7 A schematic diagram of a centered optimal surgical field is shown. Figure 7As shown, in a surgical operation area scenario, the tissues in the target area include 2001, 2002, 2004, and 2005, wherein the target surgical tissue is 2002; two implementation instruments are 2003 and 2008, respectively, for performing some surgical operations, such as cutting, pulling, suturing, coagulation, etc.; the endoscope component inserted into the human body is 205, and the target area irradiated by it is 2007, that is, the image area displayed to the operator; if the surgical goal at this stage is to partially remove the tissue 2002 along the dotted line 2006, then the optimal field of view at this time should be that the irradiation area of ​​the endoscope can cover the instrument operation area, and make the center of the irradiation area coincide with the instrument operation center as much as possible, so as to ensure that the operator can see the two operating instruments and the operated tissue as clearly as possible.

[0065] Figure 8 A schematic diagram of the optimal surgical field for a single instrument is shown. Figure 8 As shown, in a surgical operation area scenario, the tissues in the target area include 2011, 2012, 2015, 2016, and 2018, wherein the target surgical tissue 2012 is covered by other tissues 2018; two implementation instruments are 2014 and 2017, which are used to perform some surgical operations, such as cutting, pulling, suturing, coagulation, etc., wherein the instrument 2017 is mainly used to pull the obstruction tissue 2018 to expose the target surgical tissue 2012; The endoscope is 205, and the target area it illuminates is 2013, that is, the image area displayed to the operator; if the surgical goal at this stage is to completely remove the tissue 2012, and the instrument 217 must pull the obstructing tissue 218, then at this time the removal of the tissue 2012 is mainly completed by the instrument 214, so the optimal field of view should be that the irradiation area of ​​the endoscope can cover the operating area of ​​the instrument 214, and make the center of the irradiation area coincide with the tissue removal site as much as possible, so as to ensure the operator's best operating field of view.

[0066] It is understandable that the above content only illustrates that the "optimal" standard will change under different circumstances, and this embodiment does not select different algorithms for following according to different working conditions, but through big data training, an effect similar to "selecting different algorithms according to different working conditions" emerges. In essence, different following effects are caused by the fact that the training data covers different working conditions.

[0067] This embodiment introduces information such as surgical procedures, patient information, human body models, target areas, etc. to distinguish different working conditions, so that better performance can be achieved under each working condition.

[0068] In this embodiment, it is inevitable that the automatic following solution may not satisfy the operator in certain situations. Therefore, please continue to refer to Figure 1The control method includes: S50, obtaining a posture adjustment instruction; S60, controlling the movement of the surgical field shooting device based on the posture adjustment instruction; and S70, updating the posture following model based on at least the current posture of the implementation instrument and the posture of the surgical field shooting device after the movement.

[0069] Before executing step S50, the current control object can be switched through a control target switching instruction. At this time, the posture adjustment instruction is still obtained through the main operating arm; other input devices can also be set up to obtain posture adjustment instructions. At this time, the operator needs to switch the operation mode, for example, turning the body, moving the arm, etc. Since the duration of triggering steps S50 to S70 accounts for a small proportion, both of the above methods can implement step S50. For step S70, the update process can be real-time, or the relevant information can be stored first, and a new round of learning process is triggered at a specific time point later.

[0070] Please refer to Fig. 9 In one embodiment, steps S50 and S60 are implemented. During the master-slave operation, that is, when the operator operates the master operating arm to drive the slave end instrument to move, according to the above-mentioned endoscopic field of view optimization method, the endoscope arm will automatically adjust to ensure the best surgical operation field of view; when the operator still wants to adjust the endoscope field of view by himself, the operator only needs to step on the endoscope foot pedal (i.e., an optional target switching instruction) to operate the master operating arm to perform the endoscope adjustment operation; at the same time, the system will add relevant data to the database, including but not limited to: image information of the target area before and after adjustment, the position of the instrument and endoscope arm before and after adjustment, the surgical stage, the surgical action before and after adjustment, etc., to further improve the learning result of the optimal follow-up position of the endoscope and meet the operator's real-time field of view requirements as much as possible.

[0071] The pose following model is learned based on at least one of machine learning, deep learning, reinforcement learning, intelligent algorithm and meta-learning.

[0072] Please refer to Fig.10 ,According to the operator’s surgical operation data, including but not limited to the surgical procedure, target area, the position of the instrument and lens in the target area, the tissue model of the target area, the surgical operation stage and other data, a learning algorithm is used to obtain the optimal following position of the endoscope to obtain the optimal surgical field of view for different target areas.

[0073] During the preparation stage, surgery-related data are collected, including but not limited to: surgical procedure, patient information, human body model, target area, instrument type, endoscope type, instrument posture, endoscope posture, real-time image of the target area, surgical operation stage, surgical robot operation data, positioning configuration, operator information, etc., and the above data are preprocessed, that is, converted into a more suitable algorithm form.

[0074] During the training phase, certain algorithms are used to identify and analyze data or explore implicit relationships between data in order to find the optimal field of view.

[0075] During the application phase, the optimized model is used to predict the optimal field of view for real-time data during surgical operations, and to obtain the target position of the endoscope arm corresponding to the optimal field of view.

[0076] In the above learning process, the methods that can be used include but are not limited to: machine learning, deep learning, reinforcement learning, intelligent optimization methods, etc.

[0077] Please refer to Fig.11 In one embodiment, the posture following model is learned based on the actor-evaluator method, which specifically includes the following steps:

[0078] Set up actor and evaluator roles.

[0079] Actor role selection strategy function π θ (a|s), and generates actions based on the policy function and interacts with the environment.

[0080] The evaluator role calculates the value function V for the actor π (s), used to evaluate the actor’s performance and guide the actor’s actions in the next stage.

[0081] The evaluator uses an evaluator neural network to calculate the temporal difference error and update the network parameters. The output of the evaluator neural network is an evaluation value representing a state value or a state-action value.

[0082] The actor uses the actor neural network to update the network parameters, and the output of the actor neural network is the probability distribution of each action.

[0083] The policy gradient update method for the actor neural network uses: Where J(θ) represents the performance of the target strategy, represents the policy gradient, π(a t |s t ) means in state s t Next select action a t The probability of A π (s t ,a t ) represents the advantage function relative to the benchmark function.

[0084] And, the evaluator neural network uses the time series error to calculate the error update value function between the current state value and the next state value: δ = r + γV π (s t+1 )-V π (s t), where r is the reward at the current moment, γ is the discount factor, and V π (s t+1 ) is the state value at the next moment, V π (s t ) is the current state value.

[0085] Furthermore, during the learning process, at least one of the following information is added to the learning process: surgical procedure, patient information, human body model, target area, type of implementation instrument, type of surgical field shooting device, real-time image captured by the surgical field shooting device, current position of the implementation instrument, current position of the surgical field shooting device, model of tissue in the target area, positioning configuration, operator identity ID and surgical operation stage.

[0086] In another embodiment, the learning data of the posture following model is calculated based on a heuristic algorithm. Preferably, during the learning process, multi-target classification is performed, and multiple weak learners are integrated to obtain a strong learner based on an adaptive enhancement method.

[0087] Please refer to Fig.12 In an optimal field of view learning method based on spatial optimization and adaptive enhancement method, the current pose P of the endoscope is endo As the midpoint, set the endoscope adjustment space area R e , where P endo =[p x p y p z r x r y r z ] T Region R e P endo The sphere is a sphere with a center and a radius of r (r can be set according to experience), or the adjustment space area can be set in other ways; in the endoscope adjustment space area R e In the endoscope, a heuristic intelligent algorithm can be used to find the endoscope posture with the optimal field of view, and its optimization goal is:

[0088]

[0089] Where N represents the number of influences with the best field of view, w i represents the weight factor of the i-th influence, g(x i ) represents the calculated value of the i-th influence quantity, such as the distance between the center point of the image and the center point of the instrument.

[0090] According to the endoscope posture with the optimal field of view obtained by the above method, the adaptive enhancement method (Ada-boosting) is used to further evaluate the optimal field of view effect to ensure the global optimal field of view. The adaptive enhancement method integrates multiple weak learners to obtain a strong learner, which can perform better training prediction for multi-target classification.

[0091] In the above learning process, coordinate transformation may be required. An instrument is installed on the slave operating arm, and its instrument coordinate system is set to {Ot1}, and the arm base coordinate system is {Ob1}. An endoscope is installed on another slave operating arm, and its endoscope coordinate system is set to {Oe}, and the arm base coordinate system is {Ob2}. Let the posture of the instrument in the base coordinate system be Tob1_ot1, let the transformation matrix of the arm 1 base coordinate system {Ob1} relative to the arm 2 base coordinate system {Ob2} be Tob2_ob1, let the transformation matrix of the endoscope arm base coordinate system {Ob2} relative to the endoscope base coordinate system {Oe} be Toe_ob2, according to the posture transformation relationship, the posture of the instrument in the endoscope field of view can be obtained as:

[0092] Toe_ot1=Toe_ob2*Tob2_ob1*Tob1_ot1

[0093] According to the posture of the above instruments in the endoscope field of view, combined with information such as the focal length and magnification of the endoscope, the relative relationship of the instruments in the image obtained by the endoscope can be calculated, so the above information can be directly used for optimal field of view training and learning.

[0094] For further information, please refer to Fig.13 , step S40, based on the operation instruction and the following posture output control instruction, the step of implementing the coordinated movement of the instrument and the surgical field shooting device includes: obtaining a following trajectory based on the following posture; following the trajectory input posture adjustment model to obtain the joint planning speed, wherein v jnlim_i ≤v jnt_i ≤v jplim_i , v jnt_i represents the joint planning velocity, v jnlim_i represents the lower speed limit, v jplim_i Represents the upper speed limit; the lower speed limit and the upper speed limit are obtained based on the learning process of the posture following model; the posture adjustment model or the joint planning speed is adjusted based on the evaluation function to obtain the final planning result, where the evaluation function is calculated according to the following formula:

[0095]

[0096] Among them, v c represents the joint planning speed; q represents the comfort of the surgical field; n cycles of data are used for evaluation; and the movement of the surgical field shooting device is controlled based on the planning results.

[0097] Among them, follow the trajectory according to Fig.14 After obtaining the optimal field of view of the target, that is, the optimal Cartesian position of the endoscope arm, the target joint position of the endoscope arm is obtained by inverse kinematics calculation; inverse kinematics can be calculated by analytical method or numerical method.

[0098] The initial joint position and the target joint position are set, and the maximum joint velocity and the maximum joint acceleration are set. The trajectory planning algorithm can be used to calculate the time-based joint position trajectory; the trajectory planning algorithm includes but is not limited to polynomial interpolation planning, trigonometric function interpolation planning, S-type interpolation planning, etc.

[0099] Here, we take the fifth-order polynomial programming as an example and use the fifth-order polynomial interpolation, t is the time, a 0 、a 1 、a 2 、a 3 、a 4 、a 5 is the coefficient, and q(t) represents the planning value:

[0100] q(t)=a 5 t 5 +a 4 t 4 +a 3 t 3 +a 2 t 2 +a 1 t+a 0

[0101] Set the constraints as:

[0102] q(0)=q s , the position q at the start time s .

[0103] q(t f )=q f , end time t f Location q f .

[0104] Speed ​​at start and end time:

[0105]

[0106] Acceleration at the start and end time:

[0107]

[0108] Maximum speed limit

[0109] Maximum acceleration limit

[0110] The coefficient values ​​can be obtained:

[0111] a 0 =q s , a 1 =a 2 =0,

[0112] Furthermore, the control method includes: judging whether at least a part of the implementation instrument is outside the surgical field or is blocked based on machine vision and / or kinematic analysis; and, if the judgment result is yes, prompting and / or alarming.

[0113] In one embodiment, if Fig.15 As shown, when a surgical instrument is not within the surgical field of view or is blocked by tissue, a message prompt may be displayed on the display screen, such as "the right-hand instrument is blocked, please pay attention to safe operation."

[0114] This embodiment also provides a surgical system, which includes a controller, an implementation instrument and an operation field shooting device. The operation field shooting device is at least used to shoot the implementation instrument. The controller is used to control the movement of the implementation instrument and the operation field shooting device based on the above-mentioned control method.

[0115] Optionally, the surgical system also includes a doctor's console and a slave operating arm, and the controller includes a motion control unit, an image processing unit and an optimal field of view learning unit, wherein the doctor's console unit includes at least two master operating arms, the master operating arms are used to obtain operating instructions; the slave operating arms are used to clamp and drive the implementation instruments and surgical field shooting equipment to move; the motion control unit is used to calculate the motion mapping and control between the master operating arms and the slave operating arms; the image processing unit is used to process the image information of the surgical field shooting equipment and transmit it to the doctor's console or an external display screen to provide a surgical operation image field of view; the optimal field of view learning unit is used to participate in big data learning, obtain and store a posture following model, and solve the following posture of the surgical field shooting equipment based on the posture following model.

[0116] The master operating arm and the slave operating arm are controlled according to the master-slave remote operation principle, such as Fig.16As shown. In an embodiment of the master-slave remote operation principle, the operator can operate the master operation end (including the master operation arm), and the operator's operation action can be realized by remote operation to realize the corresponding movement operation of the slave operation arm end (including the slave operation arm) through master-slave mapping; at the same time, the slave operation end transmits the image information of the operation area to the doctor's console for the operator to watch. The operator can realize the movement operation of the slave end through master-slave operation, and the slave end can carry instruments or endoscopes, etc. That is, the operator can realize the corresponding surgical operation by operating the master operation arm to drive the slave end instrument; at the same time, the operator can also realize the corresponding movement by operating the master operation arm to drive the slave end endoscope to adjust the surgical operation field of view. Therefore, during the surgical operation, the operator may need to frequently switch between instrument operation and endoscope adjustment to achieve a better operation field of view and surgical action.

[0117] In summary, this embodiment provides a control method and a surgical system. Among them, the control method includes: obtaining an operation instruction, the operation instruction is only used to command the movement of the implementation instrument; predicting the expected posture of the implementation instrument based on the operation instruction; solving the follow-up posture of the arithmetic field shooting device at least based on the expected posture; and outputting a control instruction based on the operation instruction and the follow-up posture, so that the implementation instrument and the surgical field shooting device move in coordination. With such a configuration, the surgical field shooting device can automatically move with the movement of the implementation instrument without the operator manually operating, thereby solving the problem in the prior art that it is necessary to frequently switch between the control of the implementation instrument and the control of the endoscope, thereby affecting the progress of the operation and even affecting the safety of the operation.

[0118] The above description is only a description of the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A control method, characterized in that: Applied to a robot system, the robot system includes an implementation instrument and an operation field photographing device, the operation field photographing device is at least used to photograph the implementation instrument, and the control method includes: Obtaining an operation instruction, wherein the operation instruction is only used to instruct the implementation device to move; Predicting an expected posture of the implementation device based on the operation instruction; At least based on the expected posture, solve the following posture of the surgical field shooting device; and, Based on the operation instruction and the following posture output control instruction, the implementation instrument and the surgical field shooting device move in coordination.

2. The control method according to claim 1, characterized in that: The step of solving the following posture of the surgical field shooting device at least based on the expected posture comprises: solving the following posture based on at least one of the following information in combination with the expected posture: The surgical procedure, patient information, human body model, target area, the type of the implementation instrument, the type of the surgical field shooting device, the real-time image collected by the surgical field shooting device, the current position of the implementation instrument, the current position of the surgical field shooting device, the model of the tissue in the target area, the positioning configuration, the operator's identity ID and the operation stage.

3. The control method according to claim 1, characterized in that: The control method comprises: Get posture adjustment instructions; Controlling the movement of the surgical field photographing device based on the posture adjustment instruction; and The posture following model is updated based on at least the current posture of the implementation instrument and the posture of the surgical field capturing device after movement.

4. The control method according to claim 1, characterized in that: The posture following model for solving the following posture is learned based on the actor-evaluator method, and specifically includes the following steps: Set up actor roles and evaluator roles; The actor role selects a policy function, generates actions based on the policy function and interacts with the environment; The evaluator role calculates a value function for the actor to evaluate the actor's performance and guide the actor's actions in the next stage; The evaluator uses an evaluator neural network to calculate a temporal difference error and update network parameters, and the output of the evaluator neural network is an evaluation value representing a state value or a state-action value; The actor uses an actor neural network to update network parameters, and the output of the actor neural network is the probability distribution of each action; The actor neural network is updated using policy gradients; and, The evaluator neural network uses the time series error to calculate the error update value function between the current state value and the state value at the next moment.

5. The control method according to claim 4, characterized in that: During the learning process, at least one of the following information is added to the learning process: The surgical procedure, patient information, human body model, target area, the type of the implementation instrument, the type of the surgical field shooting device, the real-time image collected by the surgical field shooting device, the current position of the implementation instrument, the current position of the surgical field shooting device, the model of the tissue in the target area, the positioning configuration, the operator's identity ID and the operation stage.

6. The control method according to claim 1, characterized in that: The learning data of the posture following model for solving the following posture is calculated based on a heuristic algorithm; During the learning process, multi-target classification is performed, and multiple weak learners are integrated to obtain a strong learner based on the adaptive boosting method.

7. The control method according to claim 1, characterized in that: Based on the operation instruction and the follow-up posture output control instruction, the step of making the implementation instrument and the surgical field shooting device move in coordination includes: Obtaining a following trajectory based on the following posture; The following trajectory input posture adjustment model obtains the joint planning speed, wherein the joint planning speed is constrained by a lower limit speed and an upper limit speed; the lower limit speed and the upper limit speed are obtained based on the learning process of the posture following model; the posture following model is used to solve the following posture; adjusting the posture adjustment model or the joint planning speed based on the evaluation function to obtain a final planning result; and, The movement of the surgical field photographing device is controlled based on the planning result.

8. The control method according to claim 1, characterized in that: The control method comprises: Based on machine vision and / or kinematic analysis, determining whether at least a portion of the implementation instrument is outside the surgical field or is blocked; and, If the judgment result is yes, a prompt and / or alarm is issued.

9. A robot system, characterized in that: The robot system includes a controller, an implementation instrument and an operation field shooting device, wherein the operation field shooting device is at least used to shoot the implementation instrument, and the controller is used to control the movement of the implementation instrument and the operation field shooting device based on the control method described in any one of claims 1 to 8.

10. The robot system according to claim 11, characterized in that: The robot system further includes a doctor's console and a slave operating arm, and the controller includes a motion control unit, an image processing unit and an optimal field of view learning unit, wherein: The doctor console unit includes at least two main operating arms, and the main operating arms are used to obtain the operating instructions; The slave operating arm is used to clamp and drive the implementation instrument and the surgical field photographing device to move; The motion control unit is used to calculate the motion mapping and control between the master operating arm and the slave operating arm; The image processing unit is used to process the image information of the surgical field shooting device and transmit it to the doctor's console or an external display screen to provide an image field of view for the surgical operation; The optimal field of view learning unit is used to participate in big data learning, obtain and store a posture following model, and solve the following posture of the surgical field shooting device based on the posture following model.

Images