Surgical robot system and control method thereof

Abstract

A surgical robot system and a control method thereof are provided. The surgical robot system includes: a plurality of robot arms; an instrument provided with an end effector that is selectively attached to at least one of the plurality of robot arms and that is inserted into a surgical site at the tip for operation; a camera mounted on at least one of the plurality of robot arms and inserted into a surgical site to capture an image of the surgical site and generate image information; and a control unit that controls the operation of the plurality of robot arms and the end effector, and adjusts the intensity of the driving force transmitted to the end effector by recognizing the state of the end effector in the image information.

Description

Technical Field

[0001] The embodiments disclosed in this specification relate to a surgical robot system and a control method thereof. Specifically, the embodiments relate to a robot system and a control method thereof that can be inserted into a surgical site of a human body, and while confirming the surgical site through an image, perform surgery using surgical tools.

[0002] This research was conducted based on the research results of the "Inter-Ministerial and Full-Cycle Medical Device Research and Development Project (R&D) (Ministry of Science and Technology Information, Ministry of Health and Welfare, Ministry of Industry)" of the multi-ministry and (foundation) inter-ministerial full-cycle medical device research and development consortium (KMDF_PR_20200901_0090-01). Background Art

[0003] Minimal Invasive Surgery generally refers to surgeries that minimize the size of the patient's wound.

[0004] Specifically, different from laparotomy, which makes a large incision in a part of the human body (such as the abdomen), minimal invasive surgery forms at least one incision (or access port) sized from 0.5 cm to 1.5 cm on the human body, and through this incision, an endoscopic camera and various surgical tools are inserted, and the surgery is performed while viewing the image.

[0005] Compared with laparotomy, this minimal invasive surgery has the advantages of less postoperative pain, early recovery of intestinal motility, ability to eat early, short hospital stay, quick recovery to normal state, small incision range, and excellent cosmetic effect. Due to these advantages, minimal invasive surgery has been widely applied to cholecystectomy, prostate cancer surgery, hernia repair, etc., and its application fields are constantly expanding.

[0006] Generally, the surgical robots used in minimal invasive surgery include the surgical robot disclosed in Korean Patent Publication No. 10-2014-0102465, including a master robot and a slave robot. The master robot generates a control signal and transmits it to the slave robot. The slave robot receives the control signal sent by the master robot and performs the required surgical operations on the patient.

[0007] Here, the slave robot is equipped with at least one or more robotic arms. At the end of each robotic arm, there are instruments for performing surgery and a camera for providing an image of the surgical site. At the end of the instrument, there is an end effector for grasping surgical tools.

[0008] In the minimal invasive surgery performed using such a surgical robot, the end effector and the camera of the instrument enter the patient's body, the surgical site is confirmed through the image of the camera, and necessary treatment is performed.

[0009] When the end effector grasps a surgical tool such as a needle at the surgical site for treatment, in order to ensure the firm fixation of the needle, a relatively large force is usually applied to grasp the needle.

[0010] Here, during the process of the end effector of the surgical robot grasping the needle, there may be a problem of grasping human tissue together with the needle or grasping human tissue without grasping the needle.

[0011] In addition, the grip force of the end effector of the traditional surgical robot is set to a fixed value in order to only grasp the needle. Especially when grasping a relatively thin needle, the grip force is controlled to be a relatively large force. Therefore, if the traditional end effector grasps human tissue other than the needle, it may cause serious damage to the human tissue.

[0012] Therefore, a new technology is needed to overcome the limitations of the above prior art.

[0013] On the other hand, the above background technology is technical information owned by the inventor for deriving the present invention or obtained during the derivation process, and is not necessarily publicly known technology disclosed to the public before the application of the present invention. Summary of the Invention

[0014] The purpose of the embodiments disclosed in this specification is to propose a surgical robot system and its control method. The system can be inserted into the surgical site of the human body, confirm the image of the surgical site through a camera, and at the same time selectively assemble surgical tools on the end effector to perform surgery.

[0015] In particular, the purpose of the embodiments disclosed in this specification is to propose a surgical robot system and its control method that can judge whether surgical tools such as needles are assembled on the end effector through the image information of the camera.

[0016] In addition, the purpose of the embodiments disclosed in this specification is to propose a surgical robot system and its control method. When the end effector consists of a gripper, the grasping state of the surgical tool can be confirmed through the image information of the camera, and the grasping force of the gripper can be controlled with different forces according to the grasping state of the surgical tool.

[0017] In addition, the purpose of the embodiments disclosed in this specification is to propose a surgical robot system and its control method. When the end effector rotates and adjusts the grasping direction during the process of grasping the surgical tool, the grasping force of the end effector can be adjusted according to the grasping direction to prevent the reduction of the grasping force.

[0018] In addition, the purpose of the embodiments disclosed in this specification is to provide a surgical robot system and its control method, which can detect the grasping state of the end effector by the change in the current acting on the drive motor.

[0019] To solve the above problems, an embodiment of a surgical robot system may include multiple robotic arms; an instrument selectively mounted on at least one of the multiple robotic arms and having an end effector at the front end for inserting into a surgical site and performing operations; a camera mounted on at least one of the multiple robotic arms, inserted into the surgical site, and capturing the surgical site to generate image information; and a control unit for controlling the actions of the multiple robotic arms and the end effector, which can identify the state of the end effector in the image information and adjust the intensity of the driving force transmitted to the end effector.

[0020] In addition, to solve the above problems, an embodiment of a control method for a surgical robot system may include multiple robotic arms; an instrument selectively mounted on at least one of the multiple robotic arms and having an end effector at the front end for inserting into a surgical site and performing operations; a camera mounted on at least one of the multiple robotic arms, inserted into the surgical site, and capturing the surgical site to generate image information; and a control unit for controlling the actions of the multiple robotic arms and the end effector. The steps performed by the control unit include: identifying the state of the end effector in the image information; and adjusting the intensity of the driving force transmitted to the end effector according to the state of the end effector.

[0021] According to any of the above problem-solving means, a surgical robot system and its control method can be provided, which can be inserted into a surgical site of the human body, confirm the image of the surgical site through a camera, and selectively grasp surgical tools to perform surgery.

[0022] In particular, according to any of the above problem-solving means, the control unit determines whether a surgical tool is assembled on the end effector through the image information of the camera, and controls the grasping force of the end effector with different forces according to the judged assembly situation of the surgical tool, so as to prevent damage to human tissues by the end effector and be able to grasp surgical tools such as needles more safely and firmly. A surgical robot system and its control method.

[0023] In addition, according to any of the above problem-solving means, a surgical robot system and its control method are provided, in which the grasping force is adjusted according to the grasping direction of the end effector, so that even if the end effector rotates while grasping a surgical tool, the surgical tool can be grasped more stably.

[0024] The effects obtained in the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of the overall configuration of a surgical robot system according to an embodiment.

[0026] Figure 2 is a schematic diagram of the instrument configuration of a surgical robot system according to an embodiment.

[0027] Figure 3 is an example diagram of the image information of the surgical site captured by a camera.

[0028] Figure 4 is a flowchart of the control method of a surgical robot system according to an embodiment.

[0029] Figure 5 is Figure 4 a flowchart of an embodiment of step S410 of the control method in

[0030] Figure 6 is Figure 4 another flowchart of an embodiment of step S410 of the control method in

[0031] Figure 7 is Figure 4 yet another flowchart of an embodiment of step S410 of the control method in

[0032] Figure 8 is Figure 4 a flowchart of an embodiment of step S420 of the control method in DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Various embodiments will be described in detail below with reference to the drawings. The embodiments described below can be modified in various different forms. To more clearly illustrate the features of the embodiments, detailed descriptions of matters widely known to those of ordinary skill in the art will be omitted below. In addition, parts of the drawings unrelated to the description of the embodiments will be omitted, and similar parts will be marked with similar reference numerals throughout the specification.

[0034] Throughout the specification, when a certain component is "connected" to another component, it refers not only to "direct connection" but also to "connection with other components intervening therebetween". In addition, when a certain component "includes" a certain component, unless otherwise stated to the contrary, this means that not only other components are not excluded, but other components may also be included.

[0035] Various embodiments will be described in detail below with reference to the drawings.

[0036] Figure 1 Schematic diagram of the overall configuration of a surgical robot system according to an embodiment.

[0037] A surgical robot system according to an embodiment is a system designed to perform surgery through a robotic arm under the control of a doctor. Among them, the instruments and cameras installed at the tip of the robotic arm are inserted into the body, enabling the doctor to confirm the surgical site through imaging while appropriately controlling the robotic arm and the instruments.

[0038] Reference Figure 1 , a surgical robot system 1 according to an embodiment includes a slave robot 10 that performs surgery on a patient P lying on an operating table 2, a master console 20 that enables an operator O to remotely control the slave robot 10, and a vision cart 30 that enables an assistant A to confirm the progress of the surgery through a display unit 35.

[0039] The slave robot 10 may include one or more robotic arms 11. Generally, the robotic arm 11 has functions similar to those of a human arm and / or wrist, and specific tools can be attached at corresponding positions of the wrist. In this specification, the robotic arm 11 may be defined as including all components such as the upper arm, forearm, wrist, elbow, etc. The robotic arm 11 of such a slave robot 10 can achieve multi-degree-of-freedom movement.

[0040] In addition, an instrument 12 for performing surgery and a camera 13 for photographing the surgical site to provide image information can be installed at the front end of the robotic arm 11. The front end of the instrument 12 is provided with an end effector 123 that can selectively install surgical tools such as needles. In addition, the instrument 12 can be configured with various different types of end effectors 123 according to its use. For example, one of the instruments 12, a needle clamp, can be configured as the end effector 123, including a gripper for strongly grasping the needle. Here, a gripper is an end effector having a structure for grasping surgical tools such as needles, and any structure that can selectively grasp or place surgical tools according to the operation of the operator O is acceptable.

[0041] On the other hand, the robotic arm 11 can achieve the movement of the instrument 12 or rotation within a specific range through multi-degree-of-freedom movement, and at the same time, it can provide driving force for the end effector 123 of the instrument 12.

[0042] However, the configuration of the robotic arm 11 is not limited to this, and the above examples should not be construed as limiting the scope of the rights of the present invention. Here, the specific description of the actual control process of the operator O rotating, moving, etc. the robotic arm 11 by manipulating the lever will be omitted.

[0043] On the other hand, the slave robot 10 can be used to configure one or more units to perform surgery on the patient P. In order to present the surgical site in the form of an image through the display unit 35, the camera 13 can also be independently configured as a form of the slave robot 10.

[0044] In addition, the embodiments of the present invention can be widely applied to surgeries performed using various surgical endoscopes (e.g., thoracoscopes, arthroscopes, nasal endoscopes, etc.), not limited to laparoscopes.

[0045] In addition, the master console 20, the slave robot 10, and the vision cart 30 do not necessarily need to be physically separated. At least a part of them can be integrated together to form an integrated structure. However, the case where the master console 20, the slave robot 10, and the vision cart 30 are physically separated will be described as an example below.

[0046] The master console 20 may include an operation lever (not shown) and a display component (not shown). In addition, the master console 20 can be additionally configured with an external display device 25 for displaying the state of the operator O.

[0047] Specifically, the master console 20 can be configured with an operation lever (not shown) that can be operated by the operator O. The operation lever can be implemented by one or more handles. The operation signals of the handles by the operator O can be transmitted to the slave robot 10 through a wired or wireless communication network to control the robotic arm 11. That is, through the operation of the handles by the operator O, surgical actions such as the displacement of the robotic arm 11, the rotation of the instrument 12, and cutting can be performed.

[0048] For example, the operator O can use an operation lever in the form of a handle to operate the robotic arm 11 or the instrument 12 of the slave robot 10, etc. Such an operation lever can have various mechanical structures according to its operation mode. It can be a main handle for operating the robotic arm 11 or the instrument 12 of the slave robot 10, and an input tool attached to the master console 20 for operating the functions of the entire system, such as a joystick, a keyboard, a trackball, a touch screen, etc., to operate the robotic arm 11 or other surgical devices of the slave robot 10 in various structures. Here, the operation lever is not limited to the shape of a handle, as long as it can control the movement of the robotic arm 11 through a network such as a wired or wireless communication network.

[0049] The display component of the master console 20 can display the image captured by the camera 13. In addition, the display component can display a predetermined virtual operation panel together with the image captured by the camera 13, or display it independently. In addition, the display component of the master console 20 can provide a stereoscopic image of the image captured by the camera 13, with different left-eye and right-eye images.

[0050] The display component can be configured in various forms so that the operator O can view the images. For example, the display device can be mounted in front of the operator O's eyes. Another example is that it can consist of one or more monitors, and each monitor separately displays the information required during the operation. The number of display components can be diversely determined according to the type or category of the information to be displayed, etc.

[0051] The vision cart 30 is installed in a place separate from the slave robot 10 or the master console 20, and the progress of the operation can be externally confirmed through the display unit 35.

[0052] The images displayed on the display unit 35 can be the same as the images displayed on the display component of the operator O. The assistant A can assist the operator O in the surgical work while confirming the images on the display unit 35. For example, the assistant A can replace the instrument 12 or the surgical instrument on the instrument cart 3 according to the progress state of the operation.

[0053] On the other hand, the control unit 40 is connected to the slave robot 10, the master console 20, and the vision cart 30 and can send and receive signals to and from each other. The control unit 40 can be installed in any one of the slave robot 10, the master console 20, and the vision cart 30, or set independently.

[0054] The control unit 40, as a component that controls the overall operation of the surgical robot system 1 including the master console 20 and the slave robot 10, can include one or more processors, such as a CPU, etc. The control unit 40 controls other components in the surgical robot system and performs the calculations for the actions required by the slave robot 10 during the operation.

[0055] Such a control unit 40 can include a main control unit provided on the master console 20 and an auxiliary control unit provided on the slave robot 10.

[0056] The control unit 40 processes the information input on the master console 20, generates commands transmitted to the slave robot 10, and receives the signals transmitted from the slave robot 10 to confirm the action state of the slave robot 10, or outputs the image information transmitted from the camera 13 equipped on the slave robot 10 to the display unit 35 of the vision cart 30 and the display component of the master console 20. For this purpose, the control unit 40 can further include additional components such as a memory or a GPU. The more specific actions of the control unit 40 will be described in detail later.

[0057] On the other hand, Figure 2 is a schematic diagram of the structure of an instrument of a surgical robot system according to an embodiment. Refer to Figure 2, the instrument 12 is formed to extend along its length direction, with its front end inserted into the surgical site of the human body. The front end is provided with an end effector 123 described later, and surgical tools such as selectively assembled needles are installed through the end effector 123 to perform surgical components.

[0058] The instrument 12 can be installed on at least one of the multiple robotic arms 11 provided on the slave robot 10, inserted into the surgical site, and the driving force is transmitted to the front end through a driving motor (not shown) controlled by the control unit 40, thereby driving the end effector 123.

[0059] Specifically, the instrument 12 can be as Figure 2 shown, including a housing 121, a shaft 122, an end effector 123, and a power transmission member 124.

[0060] The housing 121 can be detachably coupled to the robotic arm (11) of the slave robot 10. Specifically, the front end of the robotic arm 11 is equipped with a connector (not shown) for coupling with the housing 121, and the housing 121 can be inserted into the connector of the robotic arm 11 for coupling. Here, the connector and the housing 121 can each be equipped with one or more driving wheels of corresponding shapes so as to engage with each other and rotate together. Therefore, on one side of the slave robot 10, for example, the driving force of a driving motor (not shown) provided on the robotic arm 11 is transmitted to the driving wheel of the connector by winding or unwinding a cable, and then the driving wheel on the housing 121 is rotated to provide power for the end effector 123 of the instrument 12.

[0061] On the other hand, the shaft 122 is a member formed with a certain length, inserted into the surgical site of the human body, extending along the length direction of the housing 121, and under the action of the robotic arm 11, the front end in its length direction can be inserted into the surgical site. Here, the front end of the shaft 122 refers to the distal end of the shaft 122 from the robotic arm 11 among the two ends of the shaft 122.

[0062] The front end of the shaft 122 is equipped with an end effector 123, and the other end of the end effector 123, that is, the rear end, is equipped with the housing 121 and can be coupled to the robotic arm 11 together with the housing 121.

[0063] Moreover, the shaft 122 can transmit the driving force of the above-mentioned driving motor to the end effector 123 installed at the front end. For this purpose, the shaft 122 can be formed into a hollow tubular structure, and a power transmission member 124 can be disposed inside.

[0064] The above-mentioned power transmission member 124 is a member that transmits the rotational force of the driving wheel on the housing 121 to the end effector 123 at the front end of the shaft 122. For example, it can include a cable (not shown) that is unwound or wound by the rotation of the driving wheel on the housing 121.

[0065] The end effector 123 is installed at the front end of the shaft 122 constituting the instrument 12 as described above, inserted into the surgical site together with the shaft 122, and operated by a driving force. Here, the end effector 123 may include grasping means according to an embodiment, for example, including the above-described gripper.

[0066] In addition, the surgical tool can be, for example, a needle, a knife, etc., all tools for performing operations such as suturing and cutting at the surgical site, and the end effector 123 can be configured in a structure so as to selectively assemble surgical tools in various ways.

[0067] Such an end effector 123 can be as Figure 2 shown, configured as a gripper including a pair of jaws 123a, and operated by a driving force transmitted to the power transmission member 124, so that the pair of jaws 123a can be opened or closed, thereby grasping and assembling surgical tools.

[0068] On the other hand, referring again to Figure 1 , the camera 13 is installed on one of the robotic arms 11 of the slave robot 10, inserted into the surgical site of the human body, and provides image information of the surgical site. The camera 13 can be constituted by, for example, an endoscope assembly equipped with an optical device such as a CCD camera at the front end of a tube extending in the longitudinal direction. The camera 13 can be moved by the robotic arm 11 operated according to a command transmitted from the main console (20), and its front end is inserted into the body to capture an image inside the body to generate image information. The generated image information can be transmitted again to the vision cart (30) and / or the main console 20.

[0069] On the other hand, the control unit 40 can control the end effector 123 of the instrument 12 including the robotic arm 11. The control unit 40 can provide driving forces for the end effector 123, for example, a pair of jaws 123a, so that the pair of jaws 123a can approach and engage with each other or move away and open, or so that the pair of jaws 123a can rotate in a certain direction together. For example, the control unit 40 can control the rotation of the drive motor that provides driving forces for the pair of jaws 123a, so that the cable is released or wound, thereby controlling the gripper.

[0070] In particular, the control unit 40 can detect the state of the end effector 123 in order to control the grasping force of the end effector 123 with different intensities. Here, the "state" of the end effector 123 includes, for example, whether the end effector 123 has grasped a surgical tool such as a needle, or whether it has not grasped a surgical tool, whether the pair of jaws 123a are opened or engaged, whether they move in the same direction, and in which direction the gripper is facing, etc.

[0071] According to one embodiment, the control unit 40 may detect the state of the end effector 123 based on the image information received from the camera 13. The control unit 40 may analyze the image information to determine whether the end effector 123 in the image has grasped the surgical tool.

[0072] To this end, the control unit 40 may use a pre-trained machine learning model. For example, the machine learning model may be pre-trained using a plurality of prepared learning images by obtaining images of the state where the end effector 123 has grasped the surgical tool and the state where it has not grasped the surgical tool in a sufficient number. At this time, each learning image may identify the end effector 123 from the actually acquired surgical image information, be cropped into an image of a certain size based on the end effector 123, and record information related to the grasping state of the surgical tool.

[0073] Therefore, the machine learning model can classify whether the end effector 123 has grasped the surgical tool by sufficiently learning whether the end effector 123 has grasped the surgical tool in each learning image.

[0074] Therefore, the control unit 40 can use such a trained machine learning model to detect whether the end effector 123 has grasped the surgical tool in at least some frames of the image information of the actual surgery. Here, the control unit 40 preferentially identifies the end effector 123 in the image information, crops a certain range of the image based on the end effector 123 to generate detection data, and then inputs the detection data into the machine learning model to obtain a result, thereby detecting whether the end effector 123 has grasped the surgical tool.

[0075] On the other hand, in other embodiments, the control unit 40 may separately identify the end effector 123 and the surgical tool in the image information, and determine whether the end effector 123 has grasped the surgical tool based on the positions of the identified end effector 123 or surgical tool.

[0076] To this end, the control unit 40 may perform object detection or object recognition in the image signal. And, in order to perform such object detection or recognition, the control unit 40 may use a pre-trained machine learning model.

[0077] Here, the machine learning model may be a learning model that has pre-learned a learning data set of a sufficient number of learning images including various end effectors 123 and various surgical tools.

[0078] Therefore, the control unit 40 can input at least some frames of the image information into the machine learning model, perform object detection or object recognition, thereby detecting the end effector 123 or the surgical tool, and judging the distance between the detected end effector 123 and the surgical tool to judge the assembly condition of the surgical tool.

[0079] That is to say, as Figure 3 shown, when the surgical tool is the needle N, the control unit 40 can respectively detect the needle N and the end effector 123 in the image information provided by the camera 13, and judge the distance between the detected end effector 12) and the needle N. If the judged distance is lower than the set value, it can be judged that the end effector 123 has grasped the needle N.

[0080] According to the embodiment, when the control unit 40 cannot recognize the surgical tool in the image information, or cannot recognize the surgical tool within a certain range of the end effector 12), it can recognize the state that the end effector 123 has not grasped the surgical tool.

[0081] On the other hand, according to the embodiment, the machine learning model can learn to judge whether the end effector 123 has grasped the surgical tool and the type of the surgical tool by learning the state images of the end effector 123 grasping two or more different types of surgical tools.

[0082] The control unit 40 can control the magnitude of the current applied to the drive motor 14 according to whether the end effector 123 has grasped the surgical tool judged, so as to control the grasping force of the end effector 123 with different intensities.

[0083] That is to say, the control unit 40 can detect the grasping condition of the surgical tool through the image information to appropriately adjust the grasping force of the end effector 123.

[0084] Specifically, when the end effector 123 has not grasped the surgical tool, the control unit 40 can set the current value applied to the drive motor 14 to a control signal lower than the default value, so as to relatively reduce the grasping force of the end effector 123.

[0085] Therefore, when the control unit 40 judges that the end effector 123 is not assembled with the surgical tool, by setting the grasping force of the end effector 123 to be lower than the default value, it can prevent the end effector 123 from damaging the human tissue when grasping the human tissue.

[0086] In addition, when the end effector 123 has grasped the surgical tool, the control unit 40 can set the current value applied to the drive motor 14 to a control signal of the default value, so as to relatively increase the grasping force of the end effector 123.

[0087] Therefore, when the control unit 40 determines that the surgical tool is assembled to the end effector 123, the grasping force of the end effector 123 can be increased to the default value, so as to firmly grasp the surgical tool.

[0088] On the other hand, the control unit 40 can refer to the imaging information and the control value provided to the drive motor of the end effector 123 or the current value measured from the drive motor to determine whether the end effector 123 has grasped the surgical tool. For example, according to the control value applied to the drive motor, if the control unit 40 receives a command to close a pair of jaws 123a, it can analyze the imaging information at that moment to determine whether the end effector 123 has grasped the surgical tool. Or, when the current value measured from the drive motor increases significantly at least temporarily, the control unit 40 can consider that the end effector 123 has grasped an object and analyze the imaging information at that moment to determine whether the end effector 123 has grasped the surgical tool.

[0089] On the other hand, when a control signal is applied through the main console 20 to perform, for example, a suturing operation in a state where the end effector 123 has grasped the surgical tool, the current value applied to the drive motor 14 can be set higher than the value in the grasping state, so as to strongly control the grasping force of the end effector 123.

[0090] On the other hand, a pair of jaws 123a constituting the end effector 123 are allowed to rotate under the control of the main console 20 in the state of grasping the surgical tool, and the grasping direction can be adjusted. The control unit 40 can adjust the amount of current applied to the drive motor 14 according to the grasping direction of the end effector 123 adjusted through the main console 20, so as to adjust the grasping force of the end effector 123 in different ways.

[0091] Here, when a pair of jaws 123a rotate in the state of grasping the surgical tool, one of the jaws will move in the opening direction, while the other jaw will move in the closing direction. Therefore, the grasping force may be reduced during the rotation process.

[0092] When the grasping direction of a pair of jaws 123a may cause the grasping force to decrease, the control unit 40 will adjust the control value applied to the drive motor 14, so as to enhance the grasping force of the pair of jaws 123a. Specifically, the control unit 40 can adjust the grasping force between the pair of jaws 123a to be stronger than the default value.

[0093] At this time, the control unit 40 will only detect the rotation direction of the pair of jaws 123a and change the grasping force when it recognizes that the end effector 123 has grasped the surgical tool.

[0094] For example, when the control unit 40 confirms the state in which the end effector 123 has grasped the surgical tool, if a pair of jaws 123a rotate in the same direction simultaneously, the grasping force can be instantaneously increased.

[0095] Here, the control unit 40 can detect the rotation direction of the pair of jaws 123a by referring to the image information, or by referring to the control value applied to the drive motor or the current value measured from the drive motor. For example, when the pair of jaws 123a receives drive commands in the same direction and with the same value, the control unit 40 can adjust the control value of at least one of the jaws 123a to increase the grasping force between the pair of jaws 123a. Alternatively, when the pair of jaws 123a has grasped the surgical tool, if the current value measured from the drive motor temporarily decreases slightly, the control unit 40 can recognize that the grasping force between them has decreased and increase the grasping force.

[0096] On the other hand, the control unit 40 can detect the grasping direction of the end effector 123 based on the image information from the camera 13 and adjust the grasping force of the end effector 123 according to the detected grasping direction.

[0097] As described above, the control unit 40 can determine whether the end effector 123 has grasped the surgical tool based not only on the image information but also on the current change measured from the drive motor. To this end, a measuring device for measuring the drive motor current can be configured to transmit it to the control unit 40.

[0098] Hereinafter, reference will be made to Figures 4 to 8 explain the control method of the surgical robot system 1 including the above components.

[0099] Figure 4 is a flowchart of the control method of the surgical robot system according to an embodiment, Figures 5 to 7 is Figure 4 a flowchart of multiple embodiments of step S410 in the control method of Figure 8 is Figure 4 a flowchart of one embodiment of step S420 in the control method of

[0100] The control method of the surgical robot system according to an embodiment is as Figure 4 shown, including a step (S410) of identifying the state of the end effector 123 and a step (S420) of adjusting the driving force intensity transmitted to the end effector 123.

[0101] The step (S410) of identifying the state of the end effector 123 is that the control unit 40 identifies the state of the end effector 123 through the image information of the camera 13, that is, identifies whether the end effector 123 has grasped a surgical tool such as a needle or whether it has not grasped the surgical tool.

[0102] More specifically, referring to Figure 5 , the step (S410) of identifying the state of the end effector 123 includes the control unit 40 training a machine learning model using the learning images of the end effector 123 (S510), and then inputting the image information into the trained machine learning model to detect the state of the end effector 123 (S520).

[0103] In the step (S510) of training the machine learning model, the control unit 40 can use a sufficient number of learning images collected for different states of the end effector 123 (e.g., the state of grasping the surgical tool and the state of not grasping the surgical tool) to train the machine learning model.

[0104] In the step (S520) of detecting the state of the end effector 123, the control unit 40 can input the image information into the trained machine learning model to detect whether the end effector 123 in the image information has grasped the surgical tool.

[0105] On the other hand, referring to Figure 6 , when the end effector 123 is composed of a pair of jaws 123a, when the control unit 40 executes the step (S610) of training the machine learning model, it can use multiple learning images of the states of grasping the surgical tool and not grasping the surgical tool to train the machine learning model. Here, the control unit 40 can identify the jaws from the pre-acquired actual surgical image information, crop the image to a certain size according to the jaws, and record the information related to the grasping state of the surgical tool to generate learning images.

[0106] In addition, when the control unit (40) executes the step (S620) of detecting the state of the end effector 123, it can input the image information into the trained machine learning model to detect whether the jaws have grasped the surgical tool in the image information. Here, the control unit 40 preferentially identifies the end effector 123 in the image information, crops a certain range of the image based on the end effector 123 to generate detection data, and then inputs the detection data into the machine learning model to obtain the result, so as to detect whether the end effector 123 has grasped the surgical tool.

[0107] As another embodiment, referring to Figure 7 , when the control unit 40 executes the step (S710) of training the machine learning model, it can use multiple learning images of the jaws and the surgical tool to train the machine learning model. Here, the control unit 40 can pre-learn a learning data set containing a sufficient number of learning images, and these learning images contain the end effector 123 and various surgical tools.

[0108] In addition, when the control unit 40 executes the step of detecting the state of the end effector 123, it may input the image information into a trained machine learning model to detect whether the gripper recognizes the surgical tool in the image information (S720). Here, the control unit 40 may input at least some frames of the image information into the machine learning model to perform object detection or object recognition, thereby detecting the gripper or the surgical tool.

[0109] Then, the control unit 40 may determine whether the gripper has grasped the surgical tool based on the distance between the recognized gripper and the surgical tool (S730). Here, the control unit 40 may determine the distance between the gripper and the surgical tool based on the positions of the gripper and the surgical tool detected in the image information, and if the determined distance is lower than the set value, it may be determined that the gripper has grasped the surgical tool.

[0110] It should be noted that although in step S410 it is described that the control unit 40 distinguishes the state of the end effector 123 as the state of grasping the surgical tool and the state of not grasping the surgical tool, if necessary, the machine learning model may also be trained to distinguish whether a pair of grippers 123a of the gripper are engaged with each other or separated, or to respectively recognize the rotation state of the pair of grippers 123a, etc.

[0111] Refer again to Figure 4 the step (S420) of adjusting the driving force intensity transmitted to the end effector 123 may adjust the intensity of the driving force according to the state of the end effector 123 recognized through step S410.

[0112] At this time, the control unit 40 may control the magnitude of the current applied to the drive motor according to whether the end effector 123 is equipped with a surgical tool as determined in step S410, thereby adjusting the grasping force of the end effector 123 in different ways. That is, the control unit 40 sets the current value to the default value in the state of being equipped with a surgical tool, thereby relatively enhancing the grasping force of the end effector 123, and sets the current value to be lower than the default value in the state of not being equipped with a surgical tool, thereby relatively weakening the grasping force of the end effector 123.

[0113] On the other hand, referring to Figure 8 when the end effector 123 is composed of a pair of grippers 123a, when the control unit 40 executes the step of adjusting the driving force intensity, it may detect the state of whether the gripper has grasped the surgical tool and detect the respective rotation directions of the pair of grippers 123a (S810).

[0114] That is to say, the control unit 40 can detect whether a pair of jaws 123a rotate in a direction that may cause a decrease in the grasping force. Herein, the control unit 40 can identify a pair of jaws 123 in the image information, detect their respective rotation directions, or detect the rotation direction of the jaws based on the control value applied to the drive motor.

[0115] Then, the control unit 40 can adjust the grasping force of the jaws according to the detected rotation direction of the jaws (S820). Herein, when the grasping direction of a pair of jaws 123a rotates in a direction that may cause a decrease in the grasping force, the control unit 40 can adjust the control value applied to the drive motor 14 to be higher than the default value, so as to enhance the grasping force of a pair of jaws 123a.

[0116] As described above, according to the surgical robot system 1 and its control method of one embodiment, the control unit 40 determines whether the end effector 123 is equipped with a surgical tool through the image information of the camera, and controls the grasping force of the end effector 123 with different intensities according to the determined assembly situation, so as to prevent the end effector 123 from damaging human tissues and be able to grasp surgical tools such as needles more safely and stably.

[0117] In this specification, the term "~ unit" refers to a software or a hardware component such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), indicating that the "~ unit" performs certain functions. However, the "~ unit" is not limited to software or hardware. The "~ unit" can be configured to be stored in an addressable storage medium or configured to run on one or more processors. Therefore, as an example, the "~ unit" can include software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0118] The functions provided in the components and the "~ unit" can be combined by fewer components and "~ unit" or separated from other components and "~ unit".

[0119] In addition, the components and the "~ unit" can be implemented as a device driving one or more CPUs or a secure multimedia card.

[0120] By Figures 4 to 8The control method of the described embodiments can be implemented in the form of a computer-readable medium that stores instructions and data executable by a computer. Here, the instructions and data can be stored in the form of program code, which, when executed by a processor, generates specific program modules to perform specific operations. In addition, the computer-readable medium can be any available medium accessible by a computer, including all volatile and non-volatile media, removable and non-removable media. Moreover, the computer-readable medium can be a computer recording medium, which can be implemented by any method or technology for storing computer-readable instructions, data structures, program modules, or other data, including all volatile and non-volatile, removable and non-removable media. For example, the computer recording medium can be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD, and Blu-ray Disc, or the memory in a server accessed through a network.

[0121] In addition, through Figures 4 to 8 The control method of the described embodiments can also be implemented as a computer program (or computer program product) containing computer-executable instructions. The computer program includes programmable machine instructions processed by a processor and can be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language, etc. In addition, the computer program can be recorded on a type of computer-readable recording medium (e.g., memory, hard disk, magnetic / optical medium, or SSD, etc.).

[0122] Therefore, through Figures 4 to 8 The control method of the described embodiments can be implemented by executing the above computer program on a computing device. The computing device can include a processor, a memory, a storage device, a memory connected by a high-speed interface and a high-speed expansion port, and a low-speed bus and a low-speed interface connected to the storage device, at least including a part of them. These components are interconnected through various buses and can be installed on a common motherboard or in other appropriate ways.

[0123] Here, the processor can process instructions within the computing device. For example, instructions for providing graphic information of a GUI (Graphical User Interface) on a display connected to the high-speed interface, and these instructions are stored in the memory or the storage device. As another embodiment, multiple processors and (or) multiple buses can be appropriately used together with multiple memories and memory forms. In addition, the processor can be implemented as a chipset containing multiple independent analog and (or) digital processors.

[0124] In addition, the memory stores information within the computing device. For example, the memory can consist of volatile memory units or a set of them. As another example, the memory can consist of non-volatile memory units or a set of them. Moreover, the memory can also be other forms of computer-readable media, such as a magnetic disk or an optical disc.

[0125] A storage device can provide a large-capacity storage space for a computing device. The storage device can be a computer-readable medium or a component including such a medium. For example, it may include devices or other components within a storage area network (SAN), floppy disks, hard disks, optical disc devices, tape devices, flash memories, other similar semiconductor storage devices, or an array of devices.

[0126] The above embodiments are only examples. Embodiments with technical ideas or essential features understandable by those of ordinary skill in the art can be easily changed into other specific forms without change. Therefore, the above embodiments should be understood as being exemplary in all aspects and not restrictive. For example, each component described in a single form can be implemented dispersedly, and similarly, components described as being dispersed can be implemented in a combined form.

[0127] The scope of protection desired in this specification is determined by the following patent claims rather than the above detailed description, and should be interpreted to include the meaning and scope of the patent claims and all changes or variations derived from their equivalent concepts.

Claims

1. A surgical robot system, characterized in that, Comprising: A plurality of robotic arms; an instrument selectively mounted on at least one of the plurality of robotic arms and having an end effector at the front end for inserting into a surgical site for operation; A camera mounted on at least one of the plurality of robotic arms and inserted into a surgical site to capture the surgical site and generate image information; and a control unit for controlling the actions of the plurality of robotic arms and the end effector, and adjusting the intensity of the driving force transmitted to the end effector by recognizing the state of the end effector in the image information.

2. The surgical robot system according to claim 1, wherein The end effector includes a gripper for selectively grasping a surgical tool. The control unit determines whether the gripper is in a state of grasping the surgical tool by recognizing at least one of the gripper and the surgical tool in the image information, and adjusts the grasping force of the gripper according to whether the gripper is in a state of grasping the surgical tool.

3. The surgical robot system according to claim 2, wherein, When the control unit recognizes the surgical tool in the image information, it determines that the gripper is in a state of grasping the surgical tool, or when the control unit recognizes the surgical tool in the image information and the distance between the gripper and the surgical tool is within a certain range, it determines that the gripper is in a state of grasping the surgical tool.

4. The surgical robot system according to claim 1, characterized in that, The control unit uses a machine learning model that has pre-learned a plurality of learning images representing different two or more states of the end effector to detect the state of the end effector from the image information.

5. The surgical robot system according to claim 2, characterized in that, The control unit uses a machine learning model that has pre-learned a plurality of learning images of the gripper and the surgical tool to recognize at least one of the gripper and the surgical tool from the image information, and determines whether the gripper is in a state of grasping the surgical tool according to the recognition result.

6. The surgical robot system according to claim 2, wherein, The gripper includes a pair of jaws that are opened or closed by adjusting the driving force controlled by the control unit. When the gripper is in a state of grasping the surgical tool, the control unit detects the respective rotation directions of the pair of jaws and adjusts the grasping force of the gripper according to the detected rotation directions of the pair of jaws.

7. The surgical robot system according to claim 6, characterized in that, The control unit detects the rotation direction of the pair of jaws by recognizing the pair of jaws in the image information, and detects the rotation direction of the pair of jaws according to the control value of the drive motor that generates the driving force transmitted to the gripper.

8. A control method for a surgical robot system, the surgical robot system comprising a plurality of robotic arms; an instrument selectively mounted on at least one of the plurality of robotic arms and having an end effector at the front end for inserting into a surgical site for operation; a camera mounted on at least one of the plurality of robotic arms and inserted into a surgical site to capture the surgical site and generate image information; and a control unit for controlling the actions of the plurality of robotic arms and the end effector, the control unit performing the following steps: recognizing the state of the end effector in the image information; and adjusting the intensity of the driving force transmitted to the end effector according to the state of the end effector.

9. The control method of the surgical robot system according to claim 8, characterized in that, The steps of identifying the state of the end effector include: training a machine learning model using a plurality of learning images representing two or more different states of the end effector; and inputting the image information into the trained machine learning model to detect the state of the end effector from the image information.

10. The control method of the surgical robot system according to claim 9, characterized in that, The end effector includes a gripper for selectively grasping a surgical tool. The training step includes training the machine learning model using a plurality of learning images of the gripper in a state of grasping the surgical tool and a state of not grasping the surgical tool. The detecting step includes inputting the image information into the trained machine learning model to detect whether the gripper in the image information is in a state of grasping the surgical tool.

11. The method of the surgical robot system according to claim 9, characterized in that, The end effector includes a gripper for selectively grasping a surgical tool. The training step includes training the machine learning model using a plurality of learning images of the gripper and the surgical tool. The step of detecting the state of the end effector includes inputting the image information into the trained machine learning model to identify the gripper and the surgical tool in the image information; and detecting whether the gripper is in a state of grasping the surgical tool based on the distance between the identified gripper and the surgical tool.

12. The method of the surgical robot system according to any one of claims 10 and 11, characterized in that, The gripper includes a pair of jaws that are opened or closed by adjusting the driving force adjusted by the control unit. The step of adjusting the intensity of the driving force transmitted to the end effector includes adjusting the grasping force of the gripper according to whether the gripper is in a state of grasping the surgical tool; when the gripper is in a state of grasping the surgical tool, detecting the respective rotation directions of the pair of jaws, and adjusting the grasping force of the gripper according to the detected rotation directions of the pair of jaws.

13. The control method of the surgical robot system according to claim 12, characterized in that, The step of adjusting the grasping force of the gripper includes: identifying the pair of jaws in the image information to detect the rotation directions of the pair of jaws, or detecting the rotation directions of the pair of jaws according to the driving motor control values generated by the driving force transmitted to the gripper.

14. A computer-readable recording medium having recorded thereon a program for executing the method of claim 8.

Images