Lag compensation control device for flexible pipe and method thereof

Abstract

The present invention relates to a hysteresis compensation control device for a flexible tube and a method thereof, and more particularly, to a hysteresis compensation control device for a flexible tube for compensating for hysteresis of a surgical tool disposed in a channel of an outer sleeve and a method thereof. To this end, a hysteresis compensation control device for a flexible tube is disclosed, characterized by comprising: an input unit receiving image information and wire tension information of a flexible surgical instrument required for mode switching; a mode switching determination unit determining a current state of the surgical instrument based on the image information of the surgical instrument and generating a control mode change signal according to the current state of the surgical instrument; a mode switching unit switching a control mode from flexible tube control based on the image information to flexible tube control based on the tension information according to the control mode change signal of the mode switching determination unit; and a compensation control unit compensating for the flexible tube having a hysteresis characteristic based on a tension error value calculated through a learning model.

Description

TECHNICAL FIELD

[0001] The present invention relates to a hysteresis compensation control device of a flexible tube and a method thereof, and more particularly, to a hysteresis compensation control device of a flexible tube for compensating control of hysteresis of a surgical tool configured in a channel of an outer sleeve and a method thereof. BACKGROUND

[0002] As shown in Figure 1 , a flexible outer sleeve 10 includes a plurality of flexible surgical instruments 21, 22. The plurality of flexible surgical instruments 21, 22 are configured by being inserted into each channel 11, 12 provided in the main body of the outer sleeve. As an example, the first surgical instrument 21 is pulled by the first and second pull wires (not shown in the drawing) so that the position and direction can be controlled.

[0003] As shown in Figure 2 , the flexible surgical instrument has a hysteresis interval. That is, even if the same pulling force is applied, a hysteresis interval occurs depending on the shape of the flexible outer sleeve or the surgical instrument. There is a disadvantage of the outer sleeve or the surgical instrument that cannot precisely control the hysteresis generation interval.

[0004] PRIOR ART DOCUMENT

[0005] PATENT DOCUMENT

[0006] US 2019 / 0290109 SUMMARY

[0007] TECHNICAL PROBLEM

[0008] Therefore, the present invention is proposed to solve the above-mentioned problems, and the object thereof is to provide an invention that can precisely control a flexible surgical instrument even in a hysteresis interval.

[0009] However, the object of the present invention is not limited to the above-mentioned object, and other objects not mentioned can be clearly understood by those skilled in the art through the following description.

[0010] MEANS FOR SOLVING THE PROBLEM

[0011] The above-mentioned object of the present invention is achieved by providing a hysteresis compensation control device of a flexible outer sleeve, characterized by comprising: an input unit that receives image information and tension information of a flexible surgical instrument required for mode switching; a mode switching determination unit that determines a current state of the surgical instrument based on the image information of the surgical instrument and generates a control mode change signal according to the current state of the surgical instrument; a mode switching unit that switches the control mode from flexible outer sleeve control based on the image information to flexible outer sleeve control based on the tension information according to the control mode change signal of the mode switching determination unit; and a compensation control unit that performs compensation control of the flexible outer sleeve having a hysteresis characteristic based on a tension error value calculated through a learning model.

[0012] And, the image information for judging the current state of the surgical instrument is image information of a tip region of the surgical instrument inserted into a channel of the main body of the overtube and pulled by the pull wire.

[0013] And, the input unit includes: a surgical instrument image capturing unit that captures the surgical instrument inserted into the channel of the main body of the overtube; a tension measuring unit that measures the pulling force of the pull wire of the surgical instrument, and transmits image information of a tip region of the surgical instrument captured by the surgical instrument image capturing unit to the mode switching judging unit, thereby judging the current state of the surgical instrument.

[0014] And, the mode switching judging unit judges whether it is possible to control in the flexible overtube control mode based on the image information based on the image information of the tip region of the surgical instrument.

[0015] And, it further includes: an image information learning unit that learns based on the image information of the surgical instrument according to a preset learning model, and calculates an image error value through learning; and a tension information learning unit that learns based on the tension information of the surgical instrument according to a preset learning model, and calculates a tension error value through learning.

[0016] In addition, the object of the present application is achieved by providing a hysteresis compensation device for a flexible overtube, characterized in that it comprises: a mode switching judging unit that judges the current state of the surgical instrument based on the image information of the flexible surgical instrument, thereby generating a first control mode change signal, or judges whether it conforms to a preset tension control modeling based on the tension information of the flexible surgical instrument, thereby generating a second control mode change signal; an image mode switching unit that switches the control mode from the flexible overtube control based on the tension information to the flexible overtube control based on the image information; a tension mode switching unit that switches the control mode from the flexible overtube control based on the image information to the flexible overtube control based on the tension information; and a compensation control unit that performs compensation control on the flexible overtube having a hysteresis characteristic using a compensation error value generated based on the tension information or the image information.

[0017] And, when the tip region of the surgical instrument included in the image information for judging the current state of the surgical instrument is not visible, or is blocked by other surgical instruments, the first control mode change signal is generated.

[0018] And, the tension mode switching unit switches the control mode according to the first control mode change signal.

[0019] And, when the load of the surgical instrument changes, or the measured value of any one of a plurality of tension measurement values of the surgical instrument is zero and does not conform to the tension control modeling, the second control mode change signal is generated.

[0020] Also, the image mode switching unit switches the control mode in accordance with the second control mode change signal.

[0021] Further, the object of the present application can be achieved by providing a hysteresis compensation control method for a flexible overtube, characterized by comprising the steps of: performing compensation control of a flexible surgical instrument having a hysteresis characteristic based on image information of the flexible surgical instrument; judging whether the surgical instrument is visible based on the image information for judging a current state of the surgical instrument; switching the control mode from compensation control based on the image information to compensation control based on tension information if it is judged that the surgical instrument is not visible; and performing compensation control of the flexible surgical instrument having the hysteresis characteristic based on tension information of the flexible surgical instrument.

[0022] Further, after the control mode is changed from compensation control based on the image information to compensation control based on the tension information, the method further comprises the steps of:

[0023] judging whether a predefined tension control modeling is satisfied based on the tension information of the flexible surgical instrument; switching the control mode from compensation control based on the tension information to compensation control based on the image information if it is judged that the predefined tension control modeling is not satisfied; and performing compensation control of the flexible surgical instrument having the hysteresis characteristic based on the image information of the flexible surgical instrument.

[0024] Effects of the Invention

[0025] According to the above-described present application, there is an effect that the flexible surgical instrument can be precisely controlled even in a hysteresis range. BRIEF DESCRIPTION OF DRAWINGS

[0026] The following drawings for the present application are examples of preferred embodiments of the present application, which are used to explain the technical idea of the present application together with the detailed description of the invention, and cannot be understood as the present application is limited to only the matters described in these drawings.

[0027] Figure 1 FIG. 1 is a diagram showing a flexible overtube according to an embodiment of the present application;

[0028] Figure 2 FIG. 3 is a diagram showing a hysteresis characteristic or curve of a flexible surgical instrument according to an embodiment of the present application;

[0029] Figure 3 FIG. 4 is a diagram showing a schematic structure of a hysteresis compensation control device for a flexible overtube according to a first embodiment of the present application;

[0030] Figure 4 FIG. 5 is a diagram for sequentially explaining a changeover of the control mode from hysteresis compensation control based on image information to hysteresis compensation control based on tension information according to the first embodiment of the present application;

[0031] Figure 5 FIG. 2 is a diagram for sequentially explaining the first embodiment of the present application in which the control mode is changed and switched from the hysteresis compensation control based on the image information to the hysteresis compensation control based on the tension information, and then the control mode is changed and switched from the hysteresis compensation control based on the tension information to the hysteresis compensation control based on the image information again;

[0032] Figure 6 FIG. 3 is a diagram for showing a brief structure of the hysteresis compensation control device of the flexible overtube of the second embodiment of the present application;

[0033] Figure 7 FIG. 4 is a diagram for sequentially explaining the second embodiment of the present application in which the control mode is changed and switched from the hysteresis compensation control based on the tension information to the hysteresis compensation control based on the image information, and then the control mode is changed and switched from the hysteresis compensation control based on the image information to the hysteresis compensation control based on the tension information again;

[0034] Figure 8 FIG. 5 is a diagram for showing a brief structure of the learning of the hysteresis compensation model of the third embodiment of the present application;

[0035] Figure 9 FIG. 6 is a diagram for showing a brief structure of the hysteresis compensation control of the third embodiment of the present application.

[0036] (Explanation of Reference Numerals)

[0037] 10: Flexible overtube 10a: Overtube main body

[0038] 10b: First pull wire 10c: Second pull wire

[0039] 11: First channel 12: Second channel

[0040] 13: Third channel 21: First surgical instrument

[0041] 21a: Tip actuator or surgical instrument tip 22: Second surgical instrument

[0042] 22a: Tip actuator or surgical instrument tip 23: Camera unit

[0043] 110: Input unit 111: Surgical instrument image capturing unit

[0044] 112: Tension measuring unit 113: Tension change amount calculating unit

[0045] 114: Tension input value comparison and determination unit 120: Mode switching determination unit

[0046] 121: Image analysis and determination unit 122: Tension modeling analysis and determination unit

[0047] 130: mode switching unit 131: image mode switching unit

[0048] 132: tension mode switching unit 140: learning unit

[0049] 141: image information learning unit 142: tension information learning unit

[0050] 150: compensation control unit

[0051] 151: lag compensation control unit based on image information

[0052] 152: lag compensation control unit based on tension information

[0053] 210: input unit 211: surgical instrument image capturing unit

[0054] 212: tension measuring unit

[0055] 213: tension change amount calculating unit 214: tension input value comparison and determination unit

[0056] 220: mode switching determination unit 221: image analysis and determination unit

[0057] 222: tension modeling analysis and determination unit 230: mode switching unit

[0058] 231: image mode switching unit 232: tension mode switching unit

[0059] 240: learning unit 241: image information learning unit

[0060] 242: tension information learning unit 250: compensation control unit

[0061] 251: lag compensation control unit based on image information

[0062] 252: lag compensation control unit based on tension information

[0063] 260: control unit 261: wire control unit

[0064] 262: surgical instrument control unit 310: flexible surgical instrument pose calculating unit

[0065] 311: flexible surgical instrument pose detecting unit

[0066] 312: flexible surgical instrument image acquiring unit

[0067] 313: flexible surgical instrument pose measuring unit

[0068] 320: learning unit 321: model condition setting unit

[0069] 322: Tension information and image information acquisition unit; 323: Flexible surgical instrument pose export unit.

[0070] 324: Lag Compensation Model Generation Unit; 325: Compensation Model Learning Unit

[0071] 330: Model search condition input unit; 340: Lag compensation model search unit

[0072] 350: Determining unit for delayed compensation model; 360: Compensation control unit.

[0073] 370: Hysteresis Change Detection Unit; 371: Outer Tube Movement Detection Unit

[0074] 372: Patient Movement Detection Unit Detailed Implementation

[0075] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below do not unreasonably limit the scope of the invention as described in the claims, and it cannot be said that all structures described in this embodiment are necessary solutions for the present invention. In addition, descriptions of matters obvious to those skilled in the art may be omitted, and the omitted descriptions of components (methods) and functions can be fully referenced without departing from the technical spirit of the present invention.

[0076] The hysteresis compensation control device for a flexible tube according to an embodiment of the present invention is a device for compensating and controlling hysteresis occurring in a flexible tube. Here, the flexible tube is a wire-driven or traction tube, and as an example, it may be a flexible outer tube or a flexible surgical instrument or flexible robotic surgical instrument inserted into a channel within the flexible outer tube. In the following description, as an example of a flexible tube, the flexible outer tube 10 or the flexible surgical instruments 21 and 22 will be used as examples.

[0077] The flexible outer cannula 10 has a flexible tube body and flexible surgical instruments inserted into the channel of the flexible tube. The flexible outer cannula 10 is inserted into the human body, and the position and orientation of the flexible tube and surgical instruments are controlled by traction lines to perform surgery on the human body.

[0078] like Figure 1 As shown, the flexible outer sheath 10 (or tube) includes: an outer sheath body 10a, having a flexible body; first, second, and third channels 11, 12, and 13, formed inside the outer sheath 10a with preset diameters; surgical instruments inserted into each channel; and a camera unit. Figure 1As shown, a first surgical instrument 21 is inserted into the first channel 11, a second surgical instrument 22 is inserted into the second channel 12, and a camera unit 23 is inserted into the third channel 13. However, the number of channels and the number of surgical instruments can be changed as needed. The camera unit 23 and the surgical instruments are protruded from the front end of the sheath body 10a and are inserted into the respective channels. Therefore, the camera unit 23 can capture an image of the front, and each surgical instrument can perform a surgery of grasping or cutting a tissue according to the operation control of the pull wire (or drive wire). An operation unit (or control unit, not shown) that can operate the pull wire (not shown) is disposed at the rear end of the sheath body 10a. The pull wire is connected from the operation unit to the front end of the sheath body or the surgical instrument. Therefore, the user can control the position and direction of the pull wire by operating the operation unit. At this time, as an example, in order to control the position and direction of the flexible sheath body 10a, the first pull wire 10b and the second pull wire 10c are adjusted by the operation unit, so that the position and direction can be changed according to the pulling force. Also, the position and direction control of each surgical instrument can be described according to the same principle as the position and direction control of the flexible sheath body 10a.

[0079] In addition, as shown in FIG. 1, the flexible sheath body 10a or the flexible surgical instruments 21 and 22 have a linear slope interval by the pulling force of the pull wire, so that the position and direction of the surgical instrument can be controlled. However, when the surgical instrument is controlled using the pull wire, even if the same pulling force as shown in FIG. 1 is provided, there is an interval in which the position and direction cannot be controlled according to the pulling force. This interval can be referred to as a hysteresis region, in which the position and direction control of the surgical instrument can be different even if the pulling force is the same. As an example, the reason for the hysteresis can be that the shape or twist of the flexible surgical tool or the flexible sheath is changed when it is inserted into the body, a frictional force is generated between the wire and the sheath, or the reason can be the elongation of the wire caused by the use of the pull wire. Also, even if the input control command is pulled by the slack of the wire, a so-called dead zone in which there is no change in output occurs, so that it can also occur. Also, it can be caused by the un-uniform tube characteristics of the sheath or the wire in the flexible sheath. Figure 2 Figure 2 Therefore, in the present application, the flexible surgical instrument having the hysteresis characteristics is compensated and controlled, so that the surgical instrument can be precisely controlled.

[0080] Therefore, in the present application, the flexible surgical instrument having the hysteresis characteristics is compensated and controlled, so that the surgical instrument can be precisely controlled.

[0081] (Structure and function of the hysteresis compensation control device of the flexible sheath: first embodiment)

[0082] As shown in FIG. 1, the flexible sheath body 10a or the flexible surgical instruments 21 and 22 have a linear slope interval by the pulling force of the pull wire, so that the position and direction of the surgical instrument can be controlled. However, when the surgical instrument is controlled using the pull wire, even if the same pulling force as shown in FIG. 1 is provided, there is an interval in which the position and direction cannot be controlled according to the pulling force. This interval can be referred to as a hysteresis region, in which the position and direction control of the surgical instrument can be different even if the pulling force is the same. As an example, the reason for the hysteresis can be that the shape or twist of the flexible surgical tool or the flexible sheath is changed when it is inserted into the body, a frictional force is generated between the wire and the sheath, or the reason can be the elongation of the wire caused by the use of the pull wire. Also, even if the input control command is pulled by the slack of the wire, a so-called dead zone in which there is no change in output occurs, so that it can also occur. Also, it can be caused by the un-uniform tube characteristics of the sheath or the wire in the flexible sheath. Figure 3 ​As shown, the hysteresis compensation control device for a flexible tube according to the first embodiment of the present invention includes an input unit 110, a mode switching judgment unit 120, a mode switching unit 130, a learning unit 140, and a compensation control unit 150.

[0083] The input unit 110 receives image information and tension information of the flexible surgical instrument required for mode switching in hysteresis control. To this end, the input unit 110 includes a surgical instrument image acquisition unit 111, a tension measurement unit 112, a tension change calculation unit 113, and a tension input value comparison and judgment unit 114.

[0084] The surgical instrument image capturing unit 111 is inserted into the third channel 13 to generate an image of the end effector of the first surgical instrument 21 or the tip 21a region of the surgical instrument, which is the hysteresis-compensated controlled object inserted into the third channel 13. At this time, when the controlled object is the second surgical instrument 22, the second surgical instrument can also be captured, and the first and second surgical instruments 21 and 22 can be captured together as needed. Furthermore, the surgical instrument image capturing unit 111 can be fixed, or it can be fixed by inserting it into the channel with a movable controllable structure as needed. Preferably, the shooting conditions of the camera actually inserted into the body for surgery are completely consistent with the shooting position, shooting angle, or shooting area of ​​the surgical instrument used for learning in the learning unit 120.

[0085] The tension measuring unit 112 measures the traction force of the traction line of the surgical instrument. To measure the traction force, for example, a tension measuring sensor can be provided to measure the tension of the traction line, or the load on the motor mounted on the operating unit for driving the traction line can be measured.

[0086] like Figure 5 As shown, after switching the control mode from image-based hysteresis compensation control to tension-based hysteresis compensation control, and then switching the mode back to image-based surgical instrument hysteresis compensation control, the system also includes a tension change calculation unit 113 and a tension input value comparison and judgment unit 114.

[0087] The tension change calculation unit 113 calculates the change in tension value when a load is suddenly applied to the first surgical instrument 21, which is in a state of no load, or conversely, when a load is suddenly lost while the first surgical instrument 21 is under load. This is done to control a surgical instrument via a traction line, as described above. Figure 1 As shown, two traction lines need to be operated separately, thus requiring tension or traction force measurement sensors for the first and second traction lines.

[0088] However, only one traction line is used for the sake of explanation, and in reality, two traction lines are required for 1 degree of freedom, and four traction lines are required for 2 degrees of freedom. Also, one drive motor is required for each traction line, or one drive motor drives two traction lines.

[0089] The tension change amount calculation unit 113 calculates the change amount of the tension value measured by each measurement sensor. By comparing the change amount of the tension value, it is possible to know the case of switching from no load to load or from load to no load, in which case it is difficult to perform the hysteresis compensation control based on the tension information, and it is necessary to switch the control mode. That is, in this case, it is necessary to change from the control based on the tension information to the control based on the image information.

[0090] The tension input value comparison and determination unit 114 looks for the case where either of the tension or traction force measurement values (or tension values) of the above-described first and second lines is "0". When either of the tension values of the first and second lines is "0", it is difficult to perform the hysteresis compensation control based on the tension information, and it is necessary to switch the control mode. That is, in this case, it is necessary to change from the control based on the tension information to the control based on the image information. Hereinafter, the first surgical instrument 21 will be assumed to be the hysteresis compensation control object, and the hysteresis compensation control of the second surgical instrument 22 will be explained using the same principle.

[0091] The mode switching determination unit 120 of one embodiment of the present application determines the current state of the flexible first surgical instrument 21 based on the image information of the first surgical instrument 21, and thereby generates a first control mode change signal (a signal generated in order to switch the mode from the hysteresis compensation control based on the image information to the hysteresis compensation control based on the tension information).

[0092] The mode switching determination unit 120 determines whether the preset tension control modeling is satisfied based on the tension information of the flexible first surgical instrument 21, and thereby generates a second control mode change signal (a signal generated in order to switch the mode from the hysteresis compensation control based on the tension information to the hysteresis compensation control based on the image information).

[0093] The image information used to determine the current state of the first surgical instrument 21 is image information of the tip 21a region of the first surgical instrument 21.

[0094] The mode switching determination unit 120 includes an image analysis and determination unit 121 and a tension modeling analysis and determination unit 122.

[0095] The image analysis and determination unit 121 receives a captured image of the tip region of the first surgical instrument 21 from the surgical instrument image capturing unit 111. The current state of the first surgical instrument 21 can be determined from the received captured image. That is, by analyzing the captured image, when the tip region of the first surgical instrument 21 included in the captured image is not visible or the first surgical instrument 21 is blocked by the second surgical instrument 22, since compensation control based on image information cannot be performed, the image analysis and determination unit 121 generates a first control mode. With generation of the first control mode change signal, the control mode is changed from lag compensation based on image information to lag compensation control based on tension information.

[0096] The tension modeling analysis and determination unit 122 determines whether the tension control modeling is satisfied based on the input value of the tension change amount calculation unit 113 or the tension input value comparison determination unit 114, and according to the determination, if it is determined that the tension control modeling is not satisfied, lag compensation control based on tension information cannot be continued, and thus a second control mode for changing the control mode is generated. That is, the control mode is changed from lag compensation control based on tension information to lag compensation control based on image information. At this time, as an example, the case where the tension control modeling is not satisfied is a case where the load of the first surgical instrument 21 changes (a case where the load is applied in a load-free state and the load disappears in a load-applied state), or the measured value of any one of the respective tension measurement values of the first and second traction lines that pull the first surgical instrument 21 is "0". In the above example, this is not a prediction model assumed at the time of deep learning learning of the tension information learning unit 142 described later, and in this case, lag compensation control based on tension information cannot be performed. Therefore, the control mode needs to be changed.

[0097] The mode changing unit 130 according to an embodiment of the present application includes an image mode changing unit 131 and a tension mode changing unit 132.

[0098] The image mode changing unit 131 changes the control mode from compensation control of the first surgical instrument 21 based on image information to compensation control of the first surgical instrument 21 based on tension information according to the first control mode change signal of the image analysis and determination unit 121.

[0099] The tension mode changing unit 132 changes the control mode from compensation control of the first surgical instrument 21 based on image information to compensation control of the first surgical instrument 21 based on tension information according to the second control mode change signal of the tension modeling analysis and determination unit 122.

[0100] The learning unit 140 according to an embodiment of the present application performs deep learning or machine learning based on the image information or the tension information, and calculates the image error value or the tension error value, respectively, through the learning. To this end, the learning unit 140 includes an image information learning unit 141 and a tension information learning unit 142.

[0101] The image information learning unit 141 performs machine learning or deep learning based on a photographed photo of the tip 21a region of the first surgical instrument 21. Therefore, it is preferable that the photographing conditions of the camera 23 for obtaining the image required for the learning and the photographing conditions of the camera 23 when actually inserted into the body are identical. The image information learning unit 141 calculates the image compensation control value or the image compensation error value corresponding to the image of the first surgical instrument 21 inserted into the body and currently photographed through the learning of the pre-defined learning model. The image compensation error value is transmitted to the compensation control unit 150. Of course, the image compensation error value generated at this time can be information related to the driving of the traction wire driving motor actually used to drive the first surgical instrument.

[0102] The tension information learning unit 142 performs machine learning or deep learning based on the tension information of the traction wire of the first surgical instrument 21. The tension information learning unit 141 calculates the tension compensation control value or the tension compensation error value corresponding to the tension measurement value of the first surgical instrument 21 inserted into the body through the learning of the pre-defined learning model. The tension compensation error value is transmitted to the compensation control unit 150.

[0103] The compensation control unit 150 performs compensation control of the first surgical instrument 21 having a hysteresis curve according to the image compensation error value or the tension compensation error value. To this end, the compensation control unit 150 includes a hysteresis compensation control unit 152 based on image information and a hysteresis compensation control unit 152 based on tension information.

[0104] The hysteresis compensation control unit 151 based on image information performs hysteresis compensation control of the first surgical instrument 21 using the image compensation error value. That is, the hysteresis compensation control is performed using the image compensation error value calculated through the learning based on the image information of the first surgical instrument currently photographed.

[0105] The hysteresis compensation control unit 152 based on tension information performs hysteresis compensation control of the first surgical instrument 21 using the tension compensation error value. That is, the hysteresis compensation control is performed using the tension compensation error value calculated through the learning based on the tension information of the first surgical instrument currently measured.

[0106] (Hysteresis compensation control method of flexible overtube)

[0107] The hysteresis compensation control method of the flexible overtube according to an embodiment of the present application includes as Figure 4The first method of changing the control mode from the compensation control based on the image information to the compensation control based on the tension information, and the second method of changing the control mode from the compensation control based on the tension information to the compensation control based on the image information, as shown in Figure 5 The second method of changing the control mode from the compensation control based on the image information to the compensation control based on the tension information, and changing the control mode from the compensation control based on the tension information to the compensation control based on the image information again, as shown in

[0108] In addition, as shown in Figure 4 From the compensation control based on the image information to the compensation control based on the image information, as shown in

[0109] The compensation control unit 150 first performs the compensation control of the first surgical instrument 21 having a lag curve based on the image information of the first surgical instrument 21 (S11). More specifically, the lag compensation control unit 151 based on the image information controls the tension of the pull wire of the first surgical instrument 21 based on the image information of the first surgical instrument 21.

[0110] At this time, the image analysis and determination unit 121 determines whether the first surgical instrument 21 is visible based on the image information for determining the current state of the first surgical instrument 21. Since the compensation control based on the image information cannot be continued when the first surgical instrument 21 is blocked or disappears in the field of view of the camera unit 23, the current state of the first surgical instrument 21 is determined (S12).

[0111] According to the determination result, when it is determined that the surgical instrument is not visible, the mode switching unit 130 switches the control mode from the compensation control based on the image information to the compensation control based on the tension information (S13).

[0112] According to the switching of the control mode, the lag compensation control unit 152 based on the tension information performs the compensation control of the first surgical instrument 21 having a lag curve based on the tension information of the first surgical instrument 21.

[0113] In addition, as shown in Figure 5 From the compensation control based on the image information to the compensation control based on the image information, as shown in

[0114] First, as shown in Figure 4 and Figure 5 The steps S11 to S14 are the same. The lag compensation control unit 152 based on the tension information performs the compensation control of the first surgical instrument 21 based on the tension information of the first surgical instrument 21.

[0115] After changing the control mode from image-based compensation control to tension-based compensation control, the tension modeling analysis and judgment unit 122 determines whether the control model conforms to the predefined control modeling based on the tension information of the first surgical instrument 21 (S15). This is because if the tension control model does not conform, tension-based compensation control cannot continue.

[0116] Based on the judgment result, when the judgment does not conform to the predefined tension control modeling, the image mode switching unit 131 switches the control mode from tension information-based compensation control to image information-based compensation control (S16).

[0117] According to the switching of control mode, the image information-based hysteresis compensation control unit 151 performs compensation control on the first surgical instrument 21 with hysteresis curve based on the image information of the first surgical instrument 21.

[0118] (Structure and function of the hysteresis compensation control device for the flexible outer jacket: Second embodiment)

[0119] The following is combined with Figure 6 and Figure 7 The hysteresis compensation control device for the flexible outer jacket according to the second embodiment of the present invention will be described in detail. However, the same descriptions can be omitted as needed, referring to the descriptions of the input unit 110, mode switching judgment unit 120, mode switching unit 130, learning unit 140 and compensation control unit 250 described above.

[0120] like Figure 6 As shown, input unit 210 receives image information of the flexible surgical instrument and line tension information required for mode switching. Input unit 210 includes a surgical instrument image capturing unit 211, a tension measurement unit 212, a tension change calculation unit 213, and a tension input value comparison and determination unit 214, which are replaced by the description of input unit 110 above. However, tension input value comparison and determination unit 214 receives the tension measurement values ​​of the first line and the second line, and searches for the case where any of the tension input values ​​is "0". For this purpose, it is preferable to adjust the traction force so that even under no-load conditions, the initial tension measurement value based on the traction force of the line is "0" or higher.

[0121] The mode switching determination unit 220 includes an image analysis determination unit 221 and a tension modeling analysis and determination unit 222. The mode switching determination unit 220 determines the current state of the surgical instrument based on the image information and the tension information of the flexible surgical instrument, and generates a control mode change signal for switching the control mode in accordance with the current state of the surgical instrument. The control mode change signal is generated for each switching condition. The mode switching determination unit 220 generates a first switching condition control mode change signal, a second switching condition control mode change signal, a third switching forced condition control mode change signal, and a signal for guiding the state change of the surgical instrument, respectively.

[0122] The tension modeling analysis and determination unit 222 generates a first switching condition control mode change signal for switching the control mode from the flexible tube control based on the tension information to the flexible tube control based on the image information in the first switching condition in which the flexible tube control based on the tension information cannot be performed when the control based on the tension information is executed.

[0123] Further, when the flexible tube control based on the image information is executed in accordance with the first switching condition control mode change signal, the tension modeling analysis and determination unit 222 generates a second switching condition control mode change signal for switching the control mode from the flexible tube control based on the image information to the flexible tube control based on the tension information when the second switching condition is satisfied, that is, when it is determined that the flexible tube control based on the image information cannot be performed based on the signal of the image analysis and determination unit 221 and the first switching condition is released.

[0124] When the flexible tube control based on the image information is executed in accordance with the first switching condition control mode change signal, the image analysis and determination unit 221 generates a third switching forced condition control mode change signal for switching the control mode from the flexible tube control based on the image information to the flexible tube control based on the tension information when the third switching forced condition is satisfied, that is, when it is determined that the flexible tube control based on the image information cannot be performed and the first switching condition is not released. As an example of the third switching forced condition, there are cases where a part of the region of at least one of the surgical instruments is not visible on the read image, or two or more surgical instruments overlap each other and an overlapping region appears on the read image and it is difficult to read the image, or it is difficult to read the image due to the contact of two or more surgical tools with each other, and the like. Further, as another example, there are cases where the camera lens or the surgical instrument is stained with blood, the field of view of the camera or the surgical instrument is blocked by the organ tissue or smoke generated by the resection in the human body, the overall image profile of the surgical instrument is changed due to the organ or the environment around the organ and it is difficult to identify the surgical instrument, or any of the surgical instruments blocks the field of view of the camera, and the image of the surgical instrument is difficult to identify due to the reflection of the light emitting diode lamp of the camera on the surgical instrument, and the like.

[0125] When the flexible tube control based on the image information is performed in accordance with the control mode change signal of the first switching condition, the fourth switching condition is met, that is, it is determined that the flexible tube control based on the image information cannot be performed and the first switching condition is not removed, and the mode switching determination unit 220 generates a signal for guiding the state change of the surgical instrument to control the wire or the surgical instrument, thereby implementing the flexible tube control based on the tension information or the image information. The generated signal is transmitted to the control unit 260 described later.

[0126] The control unit 260 includes a wire control unit 261 and a surgical instrument control unit 262. The wire control unit 261 receives the signal for guiding the state change of the surgical instrument from the mode switching determination unit 220 to control the wire, thereby controlling the wire to implement the flexible tube control based on the tension information. For example, if the wire is not broken, it can be controlled by repeatedly controlling to pull or release the traction wire, thereby performing control to measure the tension measurement value.

[0127] In addition, the surgical instrument control unit 262 receives the signal for guiding the state change of the surgical instrument from the mode switching determination unit 220 to control the surgical instrument imaging unit 211 or the surgical instrument, thereby implementing the flexible tube control based on the image information that can be read.

[0128] When the tension measurement value is normally input or the image can be read under the control of the control unit 260, the tension information activation signal and the image information activation signal are respectively generated and transmitted to the mode switching determination unit 220.

[0129] When the mode switching determination unit 220 receives the tension information activation signal from the control unit 260, the tension state change control mode change signal that changes the control mode from the flexible tube control based on the image information to the flexible tube control based on the tension information is generated. When the mode switching determination unit 220 receives the image information activation signal from the control unit 260, the control is continued in the flexible tube control mode based on the image information. Therefore, there is no need to change the control mode in particular.

[0130] If, even under the control of the control unit 260, no state change occurs and control based on the tension information and the image information cannot be performed, feedback is sent to the user.

[0131] The mode switching unit 230 receives the first switching condition control mode change signal, the second switching condition control mode change signal, the third switching forced condition control mode change signal, and the tension state change control mode change signal from the mode switching judgment unit 220. Based on each control mode change signal from the mode switching judgment unit 220, it switches the control mode from flexible tube control based on tension information to flexible tube control based on image information, or vice versa.

[0132] The description of the learning unit 240 and the compensation control unit 250 is superseded by the foregoing description.

[0133] like Figure 7 As shown, firstly, flexible tube control based on tension information is performed (S21). When performing flexible tube control based on tension information, if it is determined that the preset tension modeling is not met (first switching condition (S22)), the control mode is changed to flexible tube control based on image information and control is performed (S23, S24).

[0134] On the other hand, during the transition from flexible tube control based on tension information to flexible tube control based on image information, when the first switching condition is released (second switching condition), or when the first switching condition is not released and the image of the surgical instrument cannot be read, thus making it impossible to continue maintaining flexible tube control based on image information, the following controls are performed respectively.

[0135] First, when the first switching condition is released (second switching condition) and the measured tension value is greater than the set threshold (at this time, for example, the threshold is "0") (second switching condition), the control mode is changed from flexible tube control based on image information to flexible tube control based on tension information for control (S31, S32, S33, S34).

[0136] On the other hand, if the first switching condition is not released and the image of the surgical instrument cannot be read (S51), control is performed in the following three ways.

[0137] First, as the first method, it is determined whether the control mode change mandatory condition (third switching mandatory condition) is met. If the third switching mandatory condition is met, the control mode is changed from flexible tube control based on image information to flexible tube control based on tension information and then controlled (S41, S54a).

[0138] As a second method, the line is repeatedly pulled or released to control it and thus release the first switching condition (S52a). When the tension measurement value of the line is normally input as the line is controlled (fourth switching condition), the control mode is changed back from flexible tube control based on image information to flexible tube control based on tension information (S53a, S54a). If the fourth switching condition is not met due to the failure to input a normal tension measurement value, the method is switched to the third method described later or feedback is ultimately given to the user.

[0139] As a third method, control is performed by controlling the position of the surgical instrument or the surgical instrument image capturing unit 211 to read the image of the surgical instrument (S52b). If it is determined that the image can be read according to the controlled position (fourth switching condition), the flexible tube control based on the image information is maintained unchanged (S53b, S54b). If the fourth switching condition is not met because the image cannot be read, the method is switched to the second method described above or feedback is finally given to the user.

[0140] (Structure and function of the hysteresis compensation control device for the flexible outer jacket: Third embodiment)

[0141] Generally, hysteresis varies depending on the degree of friction between the thread and the sheath, and the shape of the sheath varies depending on the shape of the outer sheath. Therefore, the friction between the thread and the sheath may change, resulting in changes in hysteresis characteristics. Depending on the shape of the outer sheath, even with the same thread driving force, the pose (bending angle) of the flexible surgical instrument will change. Therefore, the hysteresis compensation control device for the flexible outer sheath according to the third embodiment of the present invention can search for a corresponding hysteresis characteristic model and derive compensation values ​​even if the shape of the outer sheath changes and the pose of the flexible surgical instrument changes. The following will refer to... Figure 8 and 9 A hysteresis compensation control device for a flexible outer jacket according to a third embodiment of the present invention is described in detail.

[0142] In this invention, a hysteresis compensation model is first learned and corrected through hysteresis model learning. The learned and corrected hysteresis compensation model is searched and determined according to the model search conditions. Then, the compensation value is derived according to the determined compensation model, and the flexible surgical instruments 21 and 22 are compensated and controlled.

[0143] First, refer to Figure 8 This illustrates the learning lag model. For example... Figure 8 As shown, the flexible surgical instrument pose calculation unit 310 calculates the pose values ​​of the flexible surgical instruments 21 and 22, which are inserted into the outer sheath channel and driven by the traction of a wire. The pose values ​​of the flexible surgical instruments can be calculated using the following two methods.

[0144] The first method is a method in which the flexible surgical instrument posture detection unit 311 detects and calculates the posture value of the flexible surgical instrument by a sensor. Various sensors can be configured for this purpose. As an example of the sensor, a fiber Bragg grating sensor (FBG sensor), a magnetic sensor, or a camera can be used. The posture of the flexible surgical tool can be a bending angle of the flexible surgical instrument or a measured value according to advancement, retraction, or rotation of the flexible surgical instrument. The tension information can be a tension value or a wire stretch amount measured by an encoder.

[0145] The second method is a method in which the flexible surgical instrument image acquisition unit 312 acquires an image of the flexible surgical instrument. The acquired image of each flexible surgical instrument is stored as a data set. The flexible surgical instrument posture measurement unit 313 matches the input image data set of the flexible surgical instrument and the posture value of the flexible surgical instrument and stores it as a data set. Accordingly, the flexible surgical instrument posture measurement unit 313 can estimate the actual posture value of the matched flexible surgical instrument with reference to the image data set transmitted from the flexible surgical instrument image acquisition unit 312. The image information of the flexible surgical instrument is an image data set observed by the camera 111 inserted into the channel of the outer cannula.

[0146] The model condition setting unit 321 sets to control the shape of the outer cannula to a preset shape or to control the movement of the flexible surgical instrument to a preset movement in order to set the hysteresis compensation model condition. As an example, the outer cannula can be deformed into various shapes, and the movement of the flexible surgical tool can also vary according to the surgical action. Accordingly, various environmental variables (the shape of the outer cannula and the movement of the flexible surgery) that can occur are set, and a hysteresis compensation model set according to the environmental variables is generated.

[0147] The tension information and image information acquisition unit 322 acquires corresponding tension information and image information according to the various hysteresis compensation model conditions set by the model condition setting unit 321. In addition, the tension information and image information acquisition unit 322 stores the acquired tension information and image information as an input data set (input information) for input to the hysteresis compensation model generation unit. The tension information can be measured by the tension measurement unit 112 described above, and the image information can be acquired by the flexible surgical instrument image acquisition unit 312 (or the surgical instrument image capturing unit 111) described above.

[0148] The flexible surgical instrument posture derivation unit 323 derives the posture value of the flexible surgical instrument when the input information is acquired from the tension information and image information acquisition unit 322. As described above, the posture value of the flexible surgical instrument can be derived from the sensor or the image data set. In addition, the flexible surgical instrument posture derivation unit 323 stores the derived posture value of the flexible surgical instrument as an output data set (output information).

[0149] The hysteresis compensation model generation unit 324 receives the tension information and the image information of the wire and the pose value of the flexible surgical instrument, respectively, wherein the tension information and the image information are the input data set obtained according to the shape of the outer sheath and the motion control of the flexible surgical instrument set by the model condition setting unit 321, and the pose value of the flexible surgical instrument is the output data set obtained when the input information is obtained. The hysteresis compensation model generation unit 324 receives the input data set and the output data set, and generates a corresponding hysteresis compensation model according to various environmental variables (according to the shape change of the outer sheath and the motion change of the flexible surgical instrument) set by the model condition setting unit 321.

[0150] The compensation model learning unit 325 learns and calibrates the hysteresis compensation model by learning the hysteresis compensation model generated by the hysteresis model learning unit 324, which includes the tension information, the image information, and the hysteresis compensation model.

[0151] As described above, the hysteresis compensation model is learned and calibrated by the hysteresis model learning, and then the learned and calibrated hysteresis compensation model is searched and determined according to the model search condition. The compensation value is derived by the determined compensation model to perform compensation control on the flexible surgical instrument 21, 22. In the following, the description is made with reference to Figure 9 The description is made.

[0152] The model search condition input unit 330 obtains the tension information and the image information caused by the specific motion of the flexible surgical instrument after the outer sheath reaches the target position with the lesion. The tension information can be measured by the tension measuring unit 112 as described above, and the image information can be obtained by the flexible surgical instrument image obtaining unit 312 (or the surgical instrument image shooting unit 111). The obtained tension information and image information are transmitted to the hysteresis compensation model search unit 340.

[0153] The hysteresis compensation model search unit 340 searches the learned and calibrated hysteresis compensation model of the compensation model learning unit 325 based on the tension information and the image information input by the model search condition input unit 330.

[0154] The hysteresis compensation model determination unit 350 determines the learned and calibrated hysteresis compensation model searched by the hysteresis compensation model search unit 340, and calculates the compensation value as the tension error value by the determined hysteresis compensation model.

[0155] The compensation control unit 360 performs compensation control on the flexible tube with hysteresis characteristics based on the tension error value of the hysteresis compensation model determined by the hysteresis compensation model determination unit 350.

[0156] The lag change detection unit 370 detects movement of the overtube or detects movement of the patient. Movement of the overtube or the patient is a lag change condition. When a lag change occurs, it can be necessary to re-search for a determined lag compensation model. Therefore, when the lag change detection unit 370 detects a lag change condition, it transmits a detection signal to the model search condition input unit 330. The model search condition input unit 330, which receives the lag change detection signal, re-acquires tension information and image information to re-search for a lag compensation model.

[0157] For example, movement detection of the overtube is as follows: when an image changes in a state in which no control for controlling the overtube or the flexible surgical instrument is input, or when a main operation signal for controlling the overtube is input.

[0158] In addition, when the patient moves, the shape of the overtube inserted into the inside of the body can be deformed. Therefore, movement of the patient is detected to predict a lag change. For example, movement of the patient can be grasped by checking respiration of the patient or image changes in an image taken of the patient.

[0159] In describing the present application, a description of matters obvious to those skilled in the art and the present technology can be omitted, and a description of omitted components (methods) and functions can be sufficiently referred to within the scope of the technical idea of the present application without departing from the technical idea of the present application. Also, the above-described components of the present application are only for convenience of explanation of the present application, and additional unexplained components can be added within the scope of the technical idea of the present application.

[0160] The above-described structures and functions of each part are described separately from each other only for convenience of description, and can be combined into other constituent elements as needed, or can be further subdivided and implemented.

[0161] In the above, an embodiment of the present application has been described, but the present application is not limited thereto, and various modifications and applications can be made. That is, it should be easily understood by those skilled in the art that various modifications can be made within the scope of the gist of the present application. Also, it should be noted that if detailed descriptions of well-known functions associated with the present application and combinations of each constituent are unnecessarily mixed with the gist of the present application, the detailed descriptions thereof are omitted.

Claims

1. A hysteresis compensation control device for a flexible pipe, characterized by, comprises: an input unit that receives image information and wire tension information of a flexible surgical instrument required for mode switching; a mode switching determination unit that determines a current state of the surgical instrument based on the image information and the tension information of the flexible surgical instrument, and generates a control mode change signal for switching a control mode based on the current state of the surgical instrument; a mode switching unit that switches the control mode from flexible tube control based on the tension information to flexible tube control based on the image information, or from flexible tube control based on the image information to flexible tube control based on the tension information, according to the control mode change signal of the mode switching determination unit; a compensation control unit that performs compensation control of the flexible tube having a hysteresis characteristic based on a tension error value calculated by a learning model.

2. The hysteresis compensation control device of the flexible tube according to claim 1, wherein the input unit includes: a surgical instrument image capturing unit that captures a surgical instrument inserted into a channel of a main body of an insertion tube; a tension measuring unit that measures a pulling force of a pulling wire of the surgical instrument; a tension change amount calculation unit that calculates a tension change amount of a wire tension value of the wire based on the pulling force measured by the tension measuring unit; and a tension input value comparison determination unit that compares and determines an input wire tension value based on the pulling force measured by the tension measuring unit.

3. The hysteresis compensation control device of the flexible tube according to claim 1, wherein the mode switching determination unit performs the following operations: in a first switching condition determination in which it is determined that the flexible tube control based on the tension information cannot be performed, a first switching condition control mode change signal that switches the control mode from the flexible tube control based on the tension information to the flexible tube control based on the image information is generated; or, in a second switching condition determination in which it is determined that the flexible tube control based on the image information cannot be performed and the first switching condition is released while the flexible tube control based on the image information is performed according to the first switching condition control mode change signal, a second switching condition control mode change signal that switches the control mode from the flexible tube control based on the image information to the flexible tube control based on the tension information is generated; in a third switching forced condition determination in which it is determined that the flexible tube control based on the image information cannot be performed and the first switching condition is also not released while the flexible tube control based on the image information is performed according to the first switching condition control mode change signal, a third switching forced condition control mode change signal that forcibly switches the control mode from the flexible tube control based on the image information to the flexible tube control based on the tension information is generated; or, in a fourth switching condition determination in which it is determined that the flexible tube control based on the image information cannot be performed and the first switching condition is also not released while the flexible tube control based on the image information is performed according to the first switching condition control mode change signal, a signal that guides a state change of the surgical instrument is generated to control the wire or the surgical instrument so as to achieve the flexible tube control based on the tension information or the image information.

4. The hysteresis compensation control device of the flexible tube according to claim 3, wherein ​ ​ ​ ​ ​ ​ ​ ​ ​ Further comprising a control unit that controls the wire by receiving a signal that guides a change in a state of the surgical instrument, thereby controlling to achieve the flexible tube control based on the tension information, or that controls either of the surgical instrument imaging unit and the surgical instrument, thereby controlling to achieve the flexible tube control based on the image information.

5. The hysteresis compensation control device of the flexible pipe according to claim 1, characterized by, Further comprising: an image information learning unit that learns based on image information of the surgical instrument according to a preset learning model, and calculates an image error value by learning; a tension information learning unit that learns based on tension information of the surgical instrument according to a preset learning model, and calculates a tension error value by learning.

6. The hysteresis compensation control device of the flexible tube according to claim 1, wherein the image information for judging the current state of the surgical instrument is image information of a tip region of the surgical instrument inserted into a channel of a main body of the tube and pulled by the pull wire.

7. The hysteresis compensation control device of the flexible tube according to claim 6, wherein the input unit includes: a surgical instrument image imaging unit that images the surgical instrument inserted into the channel; a tension measuring unit that measures a pulling force of the pull wire of the surgical instrument, the image information of the tip region of the surgical instrument imaged by the surgical instrument image imaging unit is transmitted to the mode switching judging unit, thereby judging the current state of the surgical instrument.

8. The hysteresis compensation control device of the flexible tube according to claim 7, wherein the mode switching judging unit judges whether or not it is possible to control in the flexible tube control mode based on the image information based on the image information of the tip region of the surgical instrument.

Images