Surgical system, processor and control method

Abstract

To provide a surgical system and the like that automatically control steps related to local injection.SOLUTION: A surgical system 1 includes an endoscope 40 including an imager 42 which captures an endoscopic image, a medical manipulator 500 with an injection needle 516 at a distal end section thereof, a drive device 20 which controls the medical manipulator 500 to control a position of the injection needle 516, and a processor 100. The processor 100 is configured to: control to inject a target amount of a local injection fluid into a syringe 512 connected to the injection needle 516; acquire, from the imager 42, the endoscopic image in which a treatment target is captured; and perform a prescribed determination (first determination (step S510) to determine success / failure of local injection using the endoscopic image. The processor 100 also controls the drive device 20 so as to change the position of the injection needle 516 (step S540) on the basis of a determination result of the prescribed determination (NO in step S531).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a surgical system, a processor and a control method. [Background technology]

[0002] There is known a surgical system that controls a medical instrument such as an endoscope by a robot arm, etc. Patent Document 1 discloses an autonomous surgical system that uses machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-074242 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 does not disclose a method for automatically controlling the process related to local injection. [Means for solving the problem]

[0005] One aspect of the present disclosure is an endoscope including an imager for capturing an endoscopic image; A medical manipulator having an injection needle at its tip; a drive unit that controls the medical manipulator to control the position of the injection needle; A processor; Including, The processor, Controlling the syringe connected to the injection needle to inject a target amount of local injection liquid; acquiring the endoscopic image showing the treatment target from the imager; The present invention relates to a surgical system that uses the endoscopic image to make a predetermined judgment as to whether local injection has been successful or not, and controls the drive device to change the position of the injection needle based on the judgment result of the predetermined judgment.

[0006] Another aspect of the present disclosure is A processor for controlling an endoscope including an imager for capturing an endoscopic image and a driving device for controlling a medical manipulator having an injection needle at a tip thereof to thereby control a position of the injection needle, Controlling the syringe connected to the injection needle to inject a target amount of local injection liquid; acquiring the endoscopic image showing the treatment target from the imager; The processor performs a predetermined determination using the endoscopic image to determine whether local injection has been successful or not, and controls the drive device to change the position of the injection needle based on the result of the predetermined determination.

[0007] Another aspect of the present disclosure is 1. A control method for controlling an endoscope including an imager that captures an endoscopic image and a driving device that controls a position of an injection needle by controlling a medical manipulator having an injection needle at a tip end, comprising: Controlling the syringe connected to the injection needle to inject a target amount of local injection liquid; acquiring the endoscopic image showing the treatment target from the imager; The present invention relates to a control method for performing a predetermined judgment using the endoscopic image to judge whether local injection has been successful or not, and for controlling the drive device to change the position of the injection needle based on the judgment result of the predetermined judgment. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a surgical system. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a drive device. [Diagram 3] FIG. 13 is a diagram for explaining an example including three treatment tool channels. [Figure 4] FIG. 13 is another diagram illustrating an example including three treatment tool channels. [Diagram 5] FIG. 13 is another diagram illustrating an example including three treatment tool channels. [Figure 6] FIG. 4 is a diagram for explaining a configuration example of a second treatment tool driving device. [Figure 7] FIG. 4 is a diagram for explaining a configuration example of a first treatment tool driving device. [Figure 8] FIG. 1 is a diagram for explaining a specific configuration example of a surgical system. [Figure 9] FIG. 2 is a diagram for explaining an example of the configuration of a console. [Figure 10] FIG. 2 is a diagram illustrating a drive mechanism controlled by a foot pedal and a handle. [Figure 11] FIG. 4 is a diagram for explaining an example of the configuration of a handle. [Figure 12] FIG. 11 is a diagram for explaining an example of settings for a control signal transmitted from a console. [Figure 13] FIG. 4 is a diagram for explaining an example of a screen displayed on a display of the console. [Figure 14] FIG. 1 is a diagram for explaining an example of an ESD treatment flow. [Figure 15] FIG. 13 is a diagram for explaining another example of an ESD treatment flow. [Figure 16] FIG. 13 is a diagram illustrating another example of the configuration of the drive device. [Figure 17] FIG. 13 is a diagram illustrating another example of the configuration of the drive device. [Figure 18] 11 is a flowchart for explaining a process example of local order processing. [Figure 19] 11 is a flowchart illustrating an example of a first determination process. [Figure 20] 5A to 5C are diagrams for explaining the effects of the present embodiment. [Figure 21] FIG. 11 is another diagram illustrating the effect of the present embodiment. [Figure 22] FIG. 11 is another diagram illustrating the effect of the present embodiment. [Diagram 23] 10 is a flowchart illustrating another example of the first determination process. [Figure 24] FIG. 4 is another diagram for explaining an example of the configuration of the control device. [Diagram 25] FIG. 2 is a diagram for explaining an example of the configuration of a learning device. [Figure 26] 13 is a flowchart for explaining another example of local injection processing. [Figure 27] 10 is a flowchart illustrating an example of a second determination process. [Figure 28]10 is a flowchart illustrating another example of the second determination process. [Figure 29] FIG. 13 is a diagram illustrating an example of data input and output to the first trained model. [Diagram 30] FIG. 4 is a diagram for explaining an example of first training data. [Diagram 31] 11 is a flowchart illustrating an example of a first determination process. [Diagram 32] A diagram explaining an example of data input and output to the second trained model. [Diagram 33] FIG. 11 is a diagram for explaining an example of second training data. [Diagram 34] 11 is a flowchart illustrating an example of a second determination process. [Diagram 35] A diagram explaining an example of data input and output to the third trained model. [Diagram 36] FIG. 11 is a diagram for explaining an example of third training data. [Figure 37] 10 is a flowchart illustrating another example of the second determination process. [Figure 38] 1A and 1B are diagrams illustrating an example of acquiring an endoscopic image using a touch panel. [Figure 39] 13 is a flowchart illustrating an example of a third determination process. [Diagram 40] A diagram explaining an example of data input and output to the fourth trained model. [Diagram 41] FIG. 13 is a diagram for explaining an example of fourth training data. [Diagram 42] 13A and 13B are diagrams for explaining another method for determining position information and a target amount of locally injected liquid. [Diagram 43] FIG. 1 is a diagram for explaining a method for acquiring a predetermined data set. [Diagram 44] FIG. 5 is a diagram for explaining an example of the fifth training data. [Diagram 45] FIG. 13 is a diagram for explaining an example of sixth training data. [Figure 46] FIG. 13 is a diagram for explaining another example of an ESD treatment flow. [Figure 47] FIG. 13 is a diagram for explaining an example of an ESD procedure screen. [Figure 48] 10 is a flowchart illustrating another example of the third determination process. [Figure 49] FIG. 13 is a diagram illustrating another example screen of ESD treatment. [Figure 50] 10 is a flowchart illustrating another example of the first determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following disclosure, many different embodiments and examples are provided for implementing different features of the presented subject matter. Of course, these are merely examples and are not intended to be limiting. Furthermore, the disclosure may repeat reference numbers and / or letters in various examples. Such repetition is for the sake of brevity and clarity and does not, per se, require a relationship between the various embodiments and / or configurations described. Furthermore, when a first element is described as being "connected" or "coupled" to a second element, such description includes embodiments in which the first element and the second element are directly connected or coupled to each other, as well as embodiments in which the first element and the second element are indirectly connected or coupled to each other with one or more other intervening elements therebetween.

[0010] An example of the configuration of a surgical system 1 of this embodiment will be described with reference to Fig. 1. The surgical system 1 of this embodiment includes a control device 10 including a processor 100, a driving device 20, and an endoscope 40. The control device 10 of this embodiment controls the driving device 20. The driving device 20 of this embodiment controls at least a medical manipulator 500, but may also control the endoscope 40, details of which will be described later with reference to Fig. 2.

[0011] The technique of the surgical system 1 of this embodiment can be applied to, for example, ESD (endoscopic submucosal dissection) treatment as described below, but does not prevent application to other treatments such as EMR (endoscopic mucosal resection). In addition, a part of the technique described below may be applied to other treatments. ESD stands for Endoscopic Submucosal Dissection, and EMR stands for Endoscopic Mucosal Resection. In the following description, the endoscope 40 of this embodiment is exemplified as a flexible endoscope that is mainly used in ESD, but does not prevent application of the technique of this embodiment to other endoscopes such as rigid endoscopes. In addition, since each configuration of the endoscope 40 of this embodiment can be widely applied to configurations that are well known as flexible endoscopes, detailed illustrations and descriptions are omitted below as appropriate.

[0012] The endoscope 40 of this embodiment includes an imager 42. The imager 42 includes an imaging sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) sensor, optical members, and functions as an imaging device. In this embodiment, an image captured by the imager 42 is called an endoscopic image, and the imager 42 captures return light from a subject irradiated by an illumination device (not shown) and outputs an image signal to the processor 100 of the control device 10 via a cable (not shown). The processor 100 generates a display image based on the image signal and outputs the display image to a display 610 (described later in FIG. 9, for example). The image signal includes a video signal, and the endoscopic image may be a still image of the video generated based on the video signal. The imager 42 of this embodiment may also be a 3D camera. In this way, the processor 100 can obtain a stereoscopic image as an endoscopic image. In this case, the display 610 may be compatible with the stereoscopic image. The display 610 and the like are compatible with stereoscopic images, and this means that the hardware of the display 610 is configured so that the user can recognize that the endoscopic image displayed on the display 610 has a three-dimensional effect. In addition, the fact that the user can recognize that the endoscopic image displayed on the display 610 has a three-dimensional effect is not limited to the fact that the user can recognize that the endoscopic image has a three-dimensional effect when viewing the display 610 with the naked eye, but also includes the fact that the user can recognize that the endoscopic image has a three-dimensional effect when viewing the display 610 through dedicated goggles or the like. In the following description, the surgeon who is the subject who operates the console 60 described later in FIG. 8 and the like is unified as the user, but for example, an assistant who assists the surgeon in his / her work may be treated as the user. Specifically, the assistant is, for example, a person who inserts the endoscope 40 into a non-specimen, or a person who replaces the treatment tool 50 described later. In addition, as described later, there may be a plurality of treatment tools 50, and may be specifically referred to as a first treatment tool 51, a second treatment tool 52, a third treatment tool 53, etc. as necessary. In addition, the lighting device (not shown) may have, for example, a plurality of lighting modes.For example, the lighting device includes multiple types of filters that transmit light of a desired wavelength, and the control device 10 performs a process of appropriately selecting a filter depending on the situation, and the light that has passed through the filter is irradiated onto the subject. This allows the user to perform the procedure smoothly. Note that a number of known lighting modes have been proposed, so detailed explanations are omitted.

[0013] The processor 100 of this embodiment is configured with the following hardware. The hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware may be configured with one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, ICs (Integrated Circuits), etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0014] In addition, for example, the processor 100 of the present embodiment can also operate based on the memory 130 not shown in FIG. 1 and the information stored in the memory 130. The information is, for example, a program and various data. The program may include, for example, a first trained model 141 described later in FIG. 24. The processor 100 can use a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or the like. The memory 130 may be a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a register, a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device. For example, the memory 130 stores computer-readable instructions, and the functions of each part of the control device 10 are realized as processing by the processor 100 executing the instructions. The instructions here may be instructions of an instruction set constituting a program, or may be instructions that instruct a hardware circuit of the processor 100 to operate. The memory 130 is also called a storage device.

[0015] The medical manipulator 500 includes at least an injection needle 516. Although the details will be described later in FIG. 2, there may be a plurality of medical manipulators 500, and they may be specifically referred to as a first medical manipulator 510, a second medical manipulator 520, a third medical manipulator 530, etc., as necessary. The injection needle 516 is used, for example, to inject a local injection liquid filled in a syringe 512 into the submucosal layer, which will be described later. The local injection liquid is, for example, physiological saline, but may also contain sodium hyaluronate, etc., consisting of a predetermined ratio. In this way, the viscosity of the local injection liquid can be increased. As a result, as will be described later in FIG. 22, the state in which the mucosa is elevated can be maintained. As a result, damage to the muscle layer can be minimized in the incision (step S610) described later in FIG. 14, etc. As a result, the possibility of complications occurring due to damage to the muscle layer can be reduced. The local injection liquid may also contain a predetermined pigment substance. The predetermined pigment substance is, for example, indigo carmine, etc. In this way, the submucosal layer where the local injection has been administered becomes bluish and transparent, so that the user can easily determine an area suitable for incision (step S610) which will be described later.

[0016] The driving device 20 of this embodiment will be described. The driving device 20 of this embodiment can be configured, for example, as shown in FIG. 2. The driving device 20 includes a first treatment tool driving device 21 that controls each part of the first treatment tool 51, a second treatment tool driving device 22 that controls each part of the second treatment tool 52, a third treatment tool driving device 23 that controls each part of the third treatment tool 53, and an endoscope driving device 24 that controls each part of the endoscope 40. Note that FIG. 2 is a conceptual diagram, and for convenience of explanation, some components such as the imager 42 and the syringe 512 are omitted from the illustration. The same applies to FIGS. 16 and 17 described later.

[0017] For convenience of explanation, the A-axis, UD-axis, and LR-axis are appropriately illustrated in FIG. 2 and subsequent figures as three mutually orthogonal axes. The direction of the A-axis is parallel to the longitudinal direction of the medical manipulator 500, based on the tip end portion, which is the tip of the insertion portion of the endoscope 40. The direction in which the medical manipulator 500 advances is the A1 direction, and the direction in which the medical manipulator 500 retreats is the A2 direction. In the following explanation, the forward and backward movements may be abbreviated to "advance and retreat". In other words, the direction of the A-axis is the direction along which the medical manipulator 500 advances and retreats. The direction along the UD-axis is called the UD-axis direction, and the direction along the LR-axis is called the LR-axis direction.

[0018] The first treatment tool 51 is, for example, a local injection needle. In this case, the first treatment tool 51 includes, for example, a first medical manipulator 510 and an injection needle 516 located at the tip of the first medical manipulator 510. In other words, the medical manipulator 500 in FIG. 1 corresponds to the first medical manipulator 510 in FIG.

[0019] The second treatment tool 52 is, for example, a grasping forceps. In this case, the second treatment tool 52 includes, for example, a second medical manipulator 520 and a grasping portion 522 located at the tip of the second medical manipulator 520.

[0020] The third treatment tool 53 is, for example, a high-frequency knife. In this case, the third treatment tool 53 includes, for example, a third medical manipulator 530. In the third treatment tool 53, a knife section is configured to be able to protrude from the tip of the third medical manipulator 530 as necessary. The knife section includes a power supply wire, a high-frequency electrode, and the like (not shown). For example, a high-frequency current is applied to the high-frequency electrode from a power supply device (not shown) via a power supply wire. In this state, the high-frequency electrode is brought into contact with a desired biological tissue, and the biological tissue is cauterized by thermal energy generated from the high-frequency electrode. By controlling the thermal energy, marking, incision (step S610), hemostasis, and the like (described later) are performed. Note that the shape of the knife section shown in FIG. 2 and the like is exemplified as a pole type, but is not limited thereto and may be a scalpel type, a needle type, a hook type, a scissors type, a tweezers type, or the like. That is, the tip of the third treatment tool 53 in this embodiment can be widely applied with a configuration related to a known high-frequency treatment tool.

[0021] In the surgical system 1 of this embodiment, at least the first treatment tool 51 is electrically controlled by a method described later. At least one of the second treatment tool 52, the third treatment tool 53, and the endoscope 40 may be further electrically controlled. In the following description, an example in which all of the second treatment tool 52, the third treatment tool 53, and the endoscope 40 are electrically controlled will be shown. Moreover, electrically means that the medical manipulator 500 is driven by an actuator such as a motor based on an electric signal for controlling the operation of the medical manipulator 500, etc. Furthermore, electrically driving the medical manipulator 500, etc. includes electrically driving the medical manipulator 500, etc. by the user manually operating the console 60 described later, in addition to electrically driving the medical manipulator 500, etc. by the processor 100.

[0022] In this manner, the local injection process (step S500) described later is realized by electrically controlling the first treatment tool 51 etc., and the process related to the local injection of ESD can be performed by automatic control of the first treatment tool 51. Automatic control of the first treatment tool 51 means that the processor 100, not the user, judges and controls each part included in the first treatment tool 51 etc., that is, the processor 100 controls each part included in the first treatment tool 51 etc. using a predetermined control algorithm.

[0023] In this manner, in the surgical system 1 of the present embodiment, the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 are inserted into the body together with the endoscope 40, and each treatment related to ESD is performed, but the endoscope 40 does not need to have three treatment tool channels. For example, the surgical system 1 of the present embodiment may further include an overtube 46. The overtube 46 may have a configuration having a plurality of treatment tool channels. For example, as shown in FIG. 2, the endoscope 40 has one treatment tool insertion port 44, the first treatment tool 51 is inserted into the treatment tool insertion port 44, and the second treatment tool 52 and the third treatment tool 53 are inserted into the two treatment tool channels in the overtube 46, respectively.

[0024] Also, as shown in FIG. 2, the surgical system 1 of this embodiment may further include a cap 48. Various types of caps of known shapes have been proposed, and for example, as shown in FIG. 3, the cap 48 includes an endoscope exit hole shown in B1 and a treatment tool exit hole shown in B2. The cap 48 in FIG. 3 is formed with one endoscope exit hole and two treatment tool exit holes corresponding to the example shown in FIG. 2. That is, as shown in FIG. 3, the first medical manipulator 510 protrudes from the endoscope exit hole shown in B1, and the second medical manipulator 520 and the third medical manipulator 530 protrude from the treatment tool exit holes shown in B2. For convenience of explanation, the cap 48 is illustrated so that the tip of the endoscope 40 protrudes from the endoscope exit hole shown in B1, but the cap 48 may be configured so that the tip of the endoscope 40 does not protrude. Also, although not illustrated for convenience, the cap 48 may further include tubes that form channels between each hole of the cap 48 and the driving device 20. In this way, the medical manipulator 500 can be operated smoothly.

[0025] 2 and 3, one treatment tool 50 protrudes from the cap 48 through the insertion part of the endoscope 40, and two treatment tools 50 protrude from the cap 48 through the overtube 46, but this is not limited thereto. For example, as shown in FIG. 4, three treatment tools 50 may protrude from the cap 48 through the overtube 46 to perform treatment such as ESD according to the method of this embodiment. The cap 48 in FIG. 4 is formed with one endoscope exit hole shown in B3 and three treatment tool exit holes shown in B4. Although not shown, for example, the endoscope 40 may have two treatment tool channels, and two treatment tools 50 may protrude from the cap 48 through the insertion part of the endoscope 40, and one treatment tool 50 may protrude from the cap 48 through the overtube 46. Although not shown, an example in which three treatment tools 50 protrude from the cap 48 through the insertion part of the endoscope 40 may be used.

[0026] In addition, in Figs. 2 and 3, the cap 48 is shown as being fitted to the overtube 46 and integrated therewith, but as shown in Fig. 5, for example, the cap 48 may be fitted to the endoscope 40 so that the endoscope 40 can be displaced independently of the overtube 46. For example, at the start of treatment, the endoscope 40, the overtube 46, and the cap 48 may be inserted into the body in an integrated state, and when the tip of the endoscope 40 approaches the location to be treated, the overtube 46 may be fixed by a balloon (not shown), and the endoscope 40 fitted with the cap 48 may be moved away from the overtube 46. In Fig. 2 and other figures, the size of the cap 48 is shown to be the same as or larger than the size of the overtube 46, but for example, the size of the cap 48 may be smaller than the size of the overtube 46. As a result, for example, the endoscope 40 fitted with the cap 48 can be driven inside the overtube 46.

[0027] 2 and the like, for example, the drive device 20 may further include an overtube drive device that drives the overtube 46. The overtube drive device can be configured with a drive device equivalent to the endoscope drive device 24.

[0028] The endoscope 40 and the treatment tool 50 of this embodiment include drive units according to the number of degrees of freedom required. For example, the second treatment tool drive device 22 includes a motor unit 220. The motor unit 220 includes, for example, a first bending motion drive section 221, a second bending motion drive section 222, an opening / closing motion drive section 223, a rolling motion drive section 224, and an advancing / retreating motion drive section 225.

[0029] The first bending operation drive unit 221 bends the second medical manipulator 520 in a direction along the UD axis by pulling or loosening a set of wires (not shown) based on a control signal received from the control device 10. As a result, the orientation of the gripper 522 is changed along the direction shown in D21. Similarly, the second bending operation drive unit 222 bends the second medical manipulator 520 in a direction along the LR axis based on a control signal received from the control device 10. As a result, the orientation of the gripper 522 is changed along the direction shown in D22.

[0030] The opening / closing operation drive unit 223 controls the opening / closing operation of the gripper 522. For example, one of the grip pieces rotates around the rotation axis shown in B21 along the direction shown in D23 based on a control signal received from the control device 10. Note that the gripper 522 shown in Fig. 6 is only an example, and a wide variety of known structures can be applied.

[0031] The roll motion drive unit 224 controls the roll rotation motion of the tip of the second medical manipulator 520. For example, based on a control signal received from the control device 10, the roll motion drive unit 224 causes the tip of the second medical manipulator 520 to roll in the direction indicated by D24.

[0032] The advance / retract action drive unit 225 controls the advance / retract action of the tip of the second medical manipulator 520. The advance / retract action drive unit 225 advances / retracts the second medical manipulator 520 along the A-axis based on a control signal received from the control device 10 by a drive mechanism including, for example, a linear motor.

[0033] The drive units of the third treatment tool driving device 23 and the endoscope driving device 24 can also be realized by a drive unit similar to the motor unit 220 of the second treatment tool driving device 22 described above. In the third treatment tool driving device 23, for example, a configuration equivalent to the roll motion driving section 224 described above may be omitted. In the endoscope driving device 24, for example, a driving unit 70 described later in Fig. 8 may slide in a predetermined direction relative to the floor to perform a function equivalent to the forward / backward motion driving section 225 described above.

[0034] 7, the first treatment tool driving device 21 may be configured to include a plunger driving unit 212 and an advance / retract operation driving unit 215. The plunger driving unit 212 controls the syringe pump 214 to press the plunger 514 included in the syringe 512 in the direction indicated by D16, which is the direction toward the tip side of the syringe 512. As a result, the local injection liquid in the syringe 512 is compressed, and the local injection liquid is discharged from the injection needle 516 by the compression force. Note that the direction indicated by D16 in FIG. 7 does not have to be the same as the A-axis direction shown in FIG. The advance / retract operation driving unit 215 controls the slider mechanism device 217. The slider mechanism device 217 is composed of a mechanism such as a linear motor or a rack and pinion, and includes a fixed unit 218 and a slider unit 219. The slider portion 219 is integrated with a part of the first medical manipulator 510, and advances and retreats in the direction indicated by D15 relative to the fixed portion 218 based on a control signal received from the control device 10. Note that the direction indicated by D15 in Fig. 7 is the same as the A-axis direction. This causes the injection needle 516 to advance and retreat along the A-axis direction.

[0035] More specifically, the surgical system 1 of this embodiment may be configured as shown in Fig. 8. In Fig. 8, the surgical system 1 includes the above-mentioned control device 10, and further includes a console 60 and a driving unit 70. The console 60 is wirelessly connected to the control device 10 by a communication method conforming to a wireless communication standard such as Wi-Fi (registered trademark), but may be wirelessly connected, for example, by wire. In the surgical system 1 of this embodiment, an endoscope 40 is inserted into the body of a subject (not shown) lying on a surgical table T, and a procedure such as ESD is performed.

[0036] 2, and electrically operates each part of the endoscope 40 based on a control signal from the control device 10. Although not shown, units corresponding to the first treatment tool driving device 21, the second treatment tool driving device 22, and the third treatment tool driving device 23 may be included in the driving unit 70 or may be provided separately from the driving unit 70, and the user may appropriately determine which.

[0037] The console 60 includes, for example, a display 610, a touch panel 620, a foot pedal 630, and a handle 640. The display 610 displays an endoscopic image captured by the imager 42 via the control device 10. The touch panel 620, which will be described in detail later with reference to FIG. 38, displays an endoscopic image like the display 610 and includes a function such as drawing. The touch panel 620 may display, for example, a part of the area including the center of the display 610. Note that there may be a plurality of foot pedals 630 and handles 640. For example, as shown in FIG. 9, the console 60 of this embodiment includes, as the foot pedals 630, a first foot pedal 631, a second foot pedal 632, and a third foot pedal 633. Note that there may be four or more foot pedals 630. Similarly, the console 60 of this embodiment includes, as the handles 640, a first handle 641 and a second handle 642. In this way, the user can use the two handles 640 to perform treatment while appropriately using the endoscope 40 and the plurality of treatment tools 50. Although not shown in the drawings, the console 60 may further include an operation unit for performing other operations in addition to the handle 640. The other operations include, for example, an air / water supply operation, an operation for changing the illumination mode of the light source device, and the like.

[0038] The direction shown in D1 in FIG. 9 is the direction along the direction in which the user faces the console 60, and is also referred to as the forward direction. The direction shown in D2 is the opposite direction to the direction shown in D1, and is also referred to as the rearward direction. The direction shown in D1 and the direction shown in D2 are collectively referred to as the front-rear direction. The direction shown in D3 is perpendicular to the front-rear direction, and is also referred to as the leftward direction. The direction shown in D4 is the opposite direction to the direction shown in D3, and is also referred to as the rightward direction. The direction shown in D3 and the direction shown in D4 are collectively referred to as the left-right direction. The same applies to FIG. 11, which will be described later.

[0039] For example, in the console 60 of this embodiment, the foot pedal 630 includes a foot switch (not shown), and when the user steps on the foot pedal 630, a control signal is transmitted from the foot switch to the control device 10. The control device 10 controls the endoscope 40 or the medical manipulator 500 based on a combination of a control signal based on the operation of the foot pedal 630 and a control signal based on the operation of a handle 640 (described later).

[0040] More specifically, for example, the memory 130 of the control device 10 stores a table shown in FIG. 10. For example, when the user operates the first foot pedal 631 and the first handle 641, the control device 10 transmits a control signal from the first handle 641 to the third treatment tool driving device 23. When the user operates the first foot pedal 631 and the second handle 642, the control device 10 transmits a control signal from the second handle 642 to the second treatment tool driving device 22. When the user operates the second foot pedal 632 and the first handle 641, the control device 10 transmits a control signal from the first handle 641 to the endoscope driving device 24. When the user operates the third foot pedal 633 and the first handle 641, the control device 10 transmits a control signal from the first handle 641 to the first treatment tool driving device 21.

[0041] 10, the control device 10 can operate the endoscope 40 by, for example, operating the first handle 641 on the right side while stepping on the second foot pedal 632 in the center of the console 60. Also, the user can operate the first treatment tool 51 by operating the first handle 641 on the right side while stepping on the third foot pedal 633 on the left side of the console 60. Also, the user can operate the second treatment tool 52 by operating the second handle 642 on the left side while stepping on the first foot pedal 631 on the right side of the console 60, and can operate the third treatment tool 53 by operating the first handle 641 on the right side.

[0042] More specifically, the handle 640 may be configured as shown in FIG. 11. In FIG. 11, the handle 640 includes a first part 651, a second part 652, and a third part 653. The first part 651 and the second part 652 are connected via a joint shown in B61. The second part 652 and the third part 653 are connected via a joint shown in B62. Although FIG. 11 shows only the second handle 642, the first handle 641 is similar to the second handle 642.

[0043] The first part 651 can be displaced in the direction shown in D62 and the direction shown in D65. The direction shown in D62 is the same as the left-right direction described above in FIG. 9. The direction shown in D65 is the same as the front-rear direction described above in FIG. 9. For example, when the user holds the second part 652, the user moves the second part 652 along the direction shown in D65, so that the first part 651 is displaced in the front-rear direction via the joint shown in B61. Similarly, when the user holds the second part 652, the user moves the second part 652 along the direction shown in D62, so that the first part 651 is displaced in the left-right direction via the joint shown in B61. In addition, the user can rotate the second part 652 in the direction shown in D61 around a rotation axis provided at the joint shown in B61. In addition, the user can roll the third part 653 in the direction shown in D64 around the longitudinal direction of the third part 653 as an axis.

[0044] When the user operates the handle 640, the medical manipulator 500 advances and retreats, bends, rolls, etc. based on a control signal output from a sensor (not shown) or the like. For example, assume that the user operates the second handle 642 while stepping on the first foot pedal 631 as described above to operate the second treatment tool 52. For example, when the user rotates the second part 652 in the direction shown in D61 of FIG. 11, the second medical manipulator 520 bends in the direction shown in D21 of FIG. 6. Also, when the user operates the second handle 642 so that the first part 651 is displaced in the direction shown in D62 of FIG. 11, the second medical manipulator 520 bends in the direction shown in D22 of FIG. 6. Also, when the user rolls the third part 653 in the direction shown in D64 of FIG. 11, the second medical manipulator 520 rolls in the direction shown in D24 of FIG. 6. 6. Also, for example, when the user operates the second handle 642 so that the first part 651 is displaced in the direction shown by D65 in FIG. 11, the second medical manipulator 520 advances or retreats in the direction shown by D25 in FIG.

[0045] Although not shown, the handle 640 may further include an operation unit such as a button. For example, a high-frequency current may be applied to the high-frequency electrode of the third treatment tool 53 by operating a button included in the first handle 641 while the user is stepping on the first foot pedal 631. Also, for example, a high-frequency current may be controlled to be opened and closed by operating a button included in the second handle 642 while the user is stepping on the first foot pedal 631.

[0046] Further, for example, the control device 10 may be configured to enable or disable reception of a control signal transmitted from the handle 640. Enabling reception of a control signal transmitted from the handle 640 means transmitting a corresponding control signal to each part of the drive device 20 based on the control signal received from the handle 640. Disabling reception of a control signal transmitted from the handle 640 means discarding an instruction based on the control signal received from the handle 640. Specifically, for example, in addition to the table shown in FIG. 10 described above, the memory 130 of the control device 10 stores a table shown in FIG. 12. For example, as described above in FIG. 6, since the second medical manipulator 520 performs any of forward / backward movement, bending, and roll rotation, reception of a control signal based on any of the operations of the first part 651, the second part 652, and the third part 653 is set to be enabled, as shown in B71 of FIG. 12.

[0047] The third medical manipulator 530 may be designed to move forward and backward and bend but not roll. In this case, as shown in B72 of Fig. 12, reception of a control signal based on the operation of the first part 651 and the second part 652 is set to be valid, but reception of a control signal based on the operation of the third part 653 is set to be invalid, as shown in B73 of Fig. 12.

[0048] Also, for example, as described above, since the endoscope 40 can move forward and backward, bend, and roll, reception of a control signal based on the operation of any of the first part 651, the second part 652, and the third part 653 is set to be valid, as shown in B74 of Fig. 12. Also, from the table of Fig. 10, when the second foot pedal 632 is stepped on, a drive mechanism to which a control signal based on the operation of the second handle 642 is to be sent is not set, and therefore reception of a control signal based on the operation of the second handle 642 is set to be invalid, as shown in B75 of Fig. 12.

[0049] 7, the first treatment tool driving device 21 may be configured so that the first treatment tool 51 only advances and retreats. When the third foot pedal 633 is stepped on, a driving mechanism to which a control signal based on the operation of the second handle 642 is sent is not set. In this case, as shown in B76 of FIG. 12, reception of a control signal based on the operation of the first part 651 in the forward / rearward direction is enabled, and reception of a control signal based on the operation of the first part 651 in the left / right direction is disabled. Also, as shown in B77 of FIG. 12, reception of other control signals is disabled.

[0050] It should be noted that the combination of the operation of the foot pedal 630 and the operation of the handle 640 is not limited to the above. For example, when the drive device 20 further includes the above-mentioned overtube drive device, the contents of the table in FIG. 10 may be changed as follows. For example, when the user operates the second foot pedal 632 and the second handle 642, the control device 10 may transmit a control signal from the second handle 642 to the first treatment tool drive device 21. Also, when the user operates the third foot pedal 633 and the first handle 641, the control device 10 may transmit a control signal from the first handle 641 to the overtube drive device.

[0051] By using the console 60 configured in this manner, as shown in FIG. 13, for example, the user operates the treatment tool 50 appropriately while watching the display 610, and performs treatment such as ESD on the treatment target shown in C1. The treatment target can also be called a lesion. The lesion here refers to a part that appears to be in a state different from a normal state in appearance, and is not necessarily limited to those caused by disease. The lesion is, for example, a tumor, but is not limited to this and may be a polyp, inflammation, diverticulum, etc. Note that the treatment target of ESD is an early tumor. An early tumor is, for example, a tumor whose growing range remains within the mucosal layer or submucosal layer, as will be described later in FIG. 20. Note that the display in FIG. 13 is merely an example, and it is not necessary that the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 are displayed on the display 610 at the same time.

[0052] A flow of treatment to which the method of this embodiment is applied will be described with reference to Fig. 14. The user sets the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 (step S10). Setting the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 means that three treatment tool channels are formed by the combination of the endoscope 40 or the endoscope 40 and the overtube 46, and the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 are inserted into each treatment tool channel.

[0053] The user then sets the local injection position (step S100), the target amount (step S200), and the direction information (step S300). The local injection position refers to the position where the injection needle 516 is inserted, and may be simply referred to as the "position". The target amount is the target value of the injection amount of the local injection liquid to be injected into the local injection position. For example, the user sets the position where the injection needle 516 is inserted around the lesion, the angle of the injection needle 516 to be directed toward the position, and the target amount of the local injection liquid to be injected after the injection needle 516 is inserted.

[0054] Although not shown in FIG. 14, marking is performed after step S300. The marking forms a predetermined mark for performing a local injection process (step S500), an incision (step S610), and the like, which will be described later. The predetermined mark is formed by performing minute cauterization around the lesion, for example, using the above-mentioned third treatment tool 53. Note that the marking may be performed by, for example, segmenting an area related to the lesion in the captured endoscopic image.

[0055] Thereafter, the processor 100 performs a local injection process (step S500). The details of the local injection process (step S500) will be described later with reference to FIG. 18. The processor 100 performs a process (step S600) to determine whether or not the local injection process (step S500) has been performed for all positions set in step S100. If the processor 100 determines that the local injection process (step S500) has been performed for all positions set in step S100 (YES in step S600), the user performs an incision (step S610). The incision (step S610) is performed using, for example, the third treatment tool 53, and the incision (step S610) can be performed smoothly by, for example, grasping the affected area with the grasping portion 522 of the second treatment tool 52. On the other hand, if the result of step S600 is NO, the user inserts the injection needle 516 into a position where the local injection process (step S500) has not been performed, and the processor 100 performs the local injection process (step S500) again.

[0056] Thereafter, the user performs dissection (step S620). For example, the user removes the lesion floating from the submucosa by incision (step S610). For example, the second medical manipulator 520 is retracted to a position outside the body while the lesion is being held by the gripping portion 522 of the second treatment tool 52, to collect the lesion. Alternatively, the lesion may be collected by replacing any one of the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 with a dedicated treatment tool for collecting the lesion, such as a net treatment tool. If bleeding occurs, step S620 may include a step of cauterizing the bleeding site using the third treatment tool 53 to stop bleeding, for example. If the size of the lesion is larger than the diameter of the channel of the second medical manipulator 520, the endoscope 40 and the medical manipulator 500 may be retracted to a position outside the body while the lesion is being held by the gripping portion 522.

[0057] In order to realize the local injection process (step S500) of this embodiment, three treatment tool channels are not required. For example, even if there are two treatment tool channels, the method of this embodiment can be applied. In that case, ESD may be performed according to the flow shown in FIG. 15. In FIG. 15, the same process as in FIG. 14 will not be described.

[0058] In Fig. 15, the user sets the second treatment tool 52 and the third treatment tool 53 (step S12). That is, two treatment tool channels are formed by the endoscope 40 or the combination of the endoscope 40 and the overtube 46, and the second treatment tool 52 and the third treatment tool 53 are inserted into each treatment tool channel. For example, as shown in Fig. 16, the third treatment tool 53 is inserted into the treatment tool insertion port 44 of the endoscope 40, and the second treatment tool 52 is inserted into the treatment tool channel provided in the overtube 46. Then, steps S100, S200, and S300 are performed in the same manner as in Fig. 14, and marking is performed using the third treatment tool 53.

[0059] After that, the user replaces the first treatment tool 51 with the third treatment tool 53. That is, as shown in FIG. 17, the first treatment tool 51 is inserted into the treatment tool insertion port 44 of the endoscope 40, and the second treatment tool 52 is continuously inserted into the treatment tool channel provided in the overtube 46. Then, the processor 100 performs a local injection process (step S500) and step S600, which will be described later. After that, if YES is determined in step S600, the user replaces the first treatment tool 51 with the third treatment tool 53. That is, if YES is determined in step S600, the local injection has ended, so there is no need to use the first treatment tool 51, and it is sufficient to perform incision (step S610) and dissection (step S620) using the second treatment tool 52 and the third treatment tool 53.

[0060] The user can perform ESD treatment by selecting the flow in Fig. 14 or the flow in Fig. 15 as appropriate depending on the situation. Since steps S400 and S602 are added to Fig. 15 compared to Fig. 14, the user can complete the treatment in a shorter time by selecting the flow in Fig. 14. In addition, the flow of a modified example described later in Fig. 46 can be applied to the case including three treatment tool channels.

[0061] Also, although the flow is not illustrated, for example, if incision (step S610) can be performed using only the third treatment tool 53 without using the second treatment tool 52, ESD may be performed using an endoscope 40 including one treatment tool channel.

[0062] The local injection process (step S500) will be described. The local injection process (step S500) is performed at a timing after starting control of the first medical manipulator 510 based on the direction information set in step S300 for the local injection position set in step S100. In other words, execution of step S500 starts at a timing when the injection needle 516 pierces the inner wall and the control of the syringe pump 214 to push the plunger 514 in the direction shown in D16 in FIG. 7 is started.

[0063] The processor 100 acquires an endoscopic image (step S502). For example, the processor 100 acquires an endoscopic image captured by the imager 42 at the timing of making a first judgment (step S510) described below. The processor 100 then makes the first judgment (step S510). The processor 100 then judges whether the judgment result of the first judgment (step S510) is OK or NG (step S531).

[0064] When the processor 100 judges that the result of the first judgment (step S510) is NG (NO in step S531), the processor 100 changes the position of the injection needle 516 (step S540) and performs step S502 again. In step S540, for example, the processor 100 controls the first treatment tool driving device 21 so that the injection needle 516 at the tip of the first medical manipulator 510 is pulled out by a predetermined length.

[0065] The first judgment (step S510) as a predetermined judgment according to the method of this embodiment will be described using the flowchart of FIG. 19. Hereinafter, the first judgment as a predetermined judgment will be simply referred to as the first judgment. The processor 100 judges whether the local injection is successful or not based on the endoscopic image (step S512). That is, when the endoscopic image as input data is input to the control device 10, the processor 100 outputs the judgment result of whether the local injection is successful or not. For example, although not shown, the user sets the periphery of the area where the injection needle 516 is inserted in the endoscopic image acquired in step S502 as a ROI (region of interest). Then, the processor 100 judges whether the local injection liquid is injected or not. Note that ROI is an abbreviation of Region of Interest.

[0066] Specific effects of the local injection process (step S500) will be described with reference to Figures 20, 21, and 22. Figure 20 is a cross-sectional view conceptually showing the timing when the injection needle 516 is inserted into the inner wall of the digestive tract, such as the stomach, esophagus, and large intestine, and the local injection process (step S500) is started. The inner wall is composed of a mucosal layer shown at L1, a submucosal layer shown at L2, a muscular layer shown at L3, and the like. Note that the serosal membrane and the like outside the muscular layer are not shown in Figure 20. The lesion shown at C11 in Figure 20 is an early stage tumor as described above, and is assumed to be within the range of the mucosal layer shown at L1.

[0067] As shown in FIG. 20, the injection needle 516 is stuck into the inner wall in a direction inclined with respect to a direction perpendicular to the direction along the wall surface of the inner wall. For example, as described above in FIG. 7, when the first treatment tool driving device 21 can only move forward and backward with respect to the first medical manipulator 510, the processor 100 adjusts the forward and backward direction of the injection needle 516 by controlling the endoscope 40 to bend in step S300 described above. In this way, the possibility that the injection needle 516 will penetrate the inner wall can be reduced. In particular, since the wall of the large intestine is thinner than the wall of the stomach, etc., it is required that the injection needle 516 does not stick deeply into the wall surface of the inner wall. In addition, since the submucosa is made of fat and is soft, it is generally difficult to stick the injection needle 516 so as to remain within the range of the submucosa of L2, and at the timing when the local injection process (step S500) is started, the tip of the injection needle 516 usually reaches a part of the muscle layer.

[0068] Since the muscle layer is a hard layer made of muscle fibers, even if the plunger driving unit 212 drives the syringe pump 214, the local injection liquid does not enter the muscle layer. In other words, no change is observed in the set ROI. In this case, the processor 100 judges NG in the first judgment (step S510). As a result, the processor 100 judges NO in step S531 and changes the position of the injection needle 516 in step S540. More specifically, the processor 100 controls the forward / backward operation driving unit 215 so as to move the injection needle 516 in the direction of A2 by a predetermined amount. In other words, the injection needle 516 is pulled out from the inner wall by a predetermined amount in step S540. If the processor 100 judges NG in the second first judgment (step S510), step S540 is performed again.

[0069] By performing step S540 a predetermined number of times, the tip of the injection needle 516 is located in the submucosal layer shown at L2, for example, as shown in FIG. 21. Since the syringe pump 214 continues to push the plunger 514, the local injection liquid is discharged from the injection needle 516 at the timing shown in FIG. 21, resulting in a state as shown in FIG. 22. In FIG. 22, the mucosal layer shown at L11, the submucosal layer shown at L12, and the lesion shown at C21 are raised by the local injection liquid shown at B81. The muscle layer shown at L13 in FIG. 22 is in approximately the same state as the muscle layer shown at L3 in FIG. 20 and FIG. 21. As a result, although not shown, a change occurs in which each tissue is raised in the set ROI.

[0070] In this case, the processor 100 judges OK in the first judgment (step S510) again. Then, the processor 100 judges YES in step S531, and the flow of the local injection process (step S500) ends. This means that local injection is completed at one position. After that, for example, in the incision (step S610), the user cauterizes and removes the position indicated by B82 with the third treatment tool 53, so that the lesion indicated by C21 is raised above the submucosal layer, and dissection (step S620) can be performed.

[0071] From the above, the surgical system 1 of this embodiment includes an endoscope 40 including an imager 42 that captures an endoscopic image, a medical manipulator 500 having an injection needle 516 at its tip, a driving device 20 that controls the medical manipulator 500 to control the position of the injection needle 516, and a processor 100. The processor 100 controls the syringe 512 connected to the injection needle 516 to inject a target amount of local injection liquid, obtains an endoscopic image of the treatment target from the imager 42, and performs a predetermined judgment (first judgment (step S510)) to judge whether the local injection is successful or not using the endoscopic image. In addition, the processor 100 controls the driving device 20 to change the position of the injection needle 516 based on the judgment result of the predetermined judgment (NO in step S531) (step S540).

[0072] In this way, the process related to local injection can be automatically controlled. As described above, ESD is a complicated procedure using the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53, and it places a large burden on the user. In this respect, by applying the method of this embodiment, the burden of the process related to local injection using the first treatment tool 51 on the user is reduced. This allows the user to concentrate on using the second treatment tool 52 and the third treatment tool 53, so that the treatment can be performed smoothly. In addition, when three treatment tool channels are included, the process of replacing the first treatment tool 51 and the third treatment tool 53 is eliminated as described above, so that the time of the operation can be shortened.

[0073] The method of the present embodiment may be realized by the processor 100. That is, the processor 100 of the present embodiment controls the endoscope 40 including the imager 42 that captures an endoscopic image, and the driving device 20 that controls the position of the injection needle 516 by controlling the medical manipulator 500 having the injection needle 516 at the tip. The processor 100 of the present embodiment controls the syringe 512 connected to the injection needle 516 to inject a target amount of local injection liquid, obtains an endoscopic image showing the treatment target from the imager 42, performs a predetermined judgment (first judgment (step S510)) to judge whether the local injection is successful or not using the endoscopic image, and controls the driving device 20 to change the position of the injection needle 516 based on the judgment result of the predetermined judgment (NO in step S531) (step S540). By doing so, the same effect as above can be obtained.

[0074] The technique of this embodiment may be realized by a control method. That is, the control method of this embodiment is a control method for controlling an endoscope 40 including an imager 42 for capturing an endoscopic image, and a driving device 20 for controlling the position of an injection needle 516 by controlling a medical manipulator 500 having an injection needle 516 at its tip. The control method of this embodiment controls the syringe 512 connected to the injection needle 516 to inject a target amount of local injection liquid, obtains an endoscopic image showing the treatment target from the imager 42, performs a predetermined judgment (first judgment (step S510)) for judging whether the local injection is successful or not using the endoscopic image, and controls the driving device 20 to change the position of the injection needle 516 based on the judgment result of the predetermined judgment (NO in step S531) (step S540). By doing so, the same effect as above can be obtained.

[0075] The method of this embodiment is not limited to the above, and various modifications such as adding other features are possible. For example, the above-mentioned first judgment (step S510) may be a processing example shown in the flowchart of FIG. 23. In FIG. 23, the processor 100 acquires an endoscopic image inferred using the first trained model 141 (step S514). For example, the processor 100 infers what kind of endoscopic image can be acquired if the local injection is successful based on the endoscopic image captured immediately before starting the local injection process (step S500). Thereafter, the processor 100 compares the inferred endoscopic image with the endoscopic image actually acquired from the imager 42 (step S516). The first judgment (step S510) shown in FIG. 23 can be realized by configuring the control device 10 as shown in the configuration example shown in FIG. 24.

[0076] 24, the control device 10 includes an input unit 110, a processor 100 including an inference unit 102, an output unit 120, and a memory 130 that stores a first trained model 141. That is, the processor 100 functions as the inference unit 102 by reading the first trained model 141 from the memory 130 and executing a program related to the first trained model 141. Note that the inference unit 102 can also execute programs related to a second trained model 142, a third trained model 143, a fourth trained model 144, a fifth trained model 145, and a sixth trained model 146, which will be described later.

[0077] The first trained model 141 of this embodiment is a program module generated by performing machine learning as supervised learning. In other words, the first trained model 141 is generated by supervised learning based on a data set in which input data and correct labels are associated with each other. The same is true for the second trained model 142, the third trained model 143, the fourth trained model 144, the fifth trained model 145, and the sixth trained model 146 described later.

[0078] In addition, in the first trained model 141 of this embodiment, at least a part of the model includes a neural network. Although not shown, the neural network has an input layer to which data is input, an intermediate layer that performs calculations based on the output from the input layer, and an output layer that outputs data based on the output from the intermediate layer. The number of intermediate layers is not particularly limited. The number of nodes included in each layer in the intermediate layer is not particularly limited. A node included in a given layer in the intermediate layer is connected to a node in an adjacent layer. A weighting coefficient is set for each connection. Each node multiplies the output of the previous node by the weighting coefficient to obtain a sum of the multiplication results. Furthermore, each node adds a bias to the sum and applies an activation function to the sum to obtain the output of the node. This process is performed sequentially from the input layer to the output layer to obtain the output of the neural network. Note that various functions such as a sigmoid function and a ReLU function are known as activation functions, and they can be widely applied in this embodiment. In addition, the second trained model 142, the third trained model 143, the fourth trained model 144, the fifth trained model 145, and the sixth trained model 146 described below also include neural networks.

[0079] The input unit 110 is an interface for receiving input data from the outside. For example, in step S514, the input unit 14 functions as the input unit 110 by inputting image data of an endoscopic image in which a lesion to be treated is captured before local injection to the input unit 110.

[0080] The output unit 120 is an interface that transmits to the outside the data estimated by the inference unit 102. For example, in step S514, the output unit 120 functions as the output unit 120 by outputting data of an endoscopic image inferred by the inference unit 102 as output data from the first trained model 141.

[0081] The machine learning of the first trained model 141 is performed by, for example, a learning device 80. Fig. 25 is a block diagram showing an example configuration of the learning device 80. The learning device 80 includes, for example, a processor 800, a memory 830, and a communication unit 860, and the processor 800 includes a machine learning unit 802.

[0082] The processor 800 controls input and output of data between each functional unit such as the memory 830 and the communication unit 860. The processor 800 can be realized by hardware similar to the processor 100 described above in FIG. 1. The processor 800 executes various arithmetic processing based on a predetermined program read from the memory 830, an operation input signal from an operation unit not shown in FIG. 25, and the like, and controls data input and output operations with the surgical system 1. The predetermined program here includes a machine learning program not shown. That is, the processor 800 functions as the machine learning unit 802 by appropriately reading and executing the machine learning program and necessary data from the memory 830. In addition to the machine learning program not shown, the memory 830 stores a first training model 841 and first training data 851. The memory 830 can be realized by a semiconductor memory similar to the memory 130 described above in FIG. 1.

[0083] The communication unit 860 is a communication interface capable of communicating with the surgical system 1 by a predetermined communication method. The predetermined communication method is, for example, the aforementioned Wi-Fi (registered trademark) or the like, but is not limited thereto and may be a communication method conforming to a wired communication standard such as USB. As a result, the learning device 80 transmits the machine-learned first trained model 141 to the surgical system 1, and the surgical system 1 can update the first trained model 141. Note that FIG. 25 shows an example in which the learning device 80 and the surgical system 1 are separate, but this does not prevent the surgical system 1 from being configured as a configuration example including a learning server equivalent to the learning device 80.

[0084] Although not shown, the first training data 851 is a data set in which an endoscopic image showing a lesion as a treatment target before local injection and an endoscopic image showing a lesion whose shape has changed due to local injection are correct labels. As described above in FIG. 22, the correct label is an endoscopic image showing the periphery of the injection needle 516 being punctured due to successful local injection, which shows a three-dimensional change in shape. The machine learning unit 802 inputs a large number of first training data 851 to the first training model 841, thereby updating the first training model 841 to the first learned model 141.

[0085] As described above, in step S514 of FIG. 23, an endoscopic image showing a lesion before local injection is input to the input unit 110 as input data, and the inference unit 102 infers an endoscopic image showing a lesion after local injection, and outputs it via the output unit 120. Then, in step S516, the processor 100 compares the endoscopic image inferred in step S514 with the endoscopic image actually acquired in step S502 to determine whether local injection has been successful. If the similarity of the images is equal to or greater than a predetermined threshold, the result of the first judgment (step S510) is OK, and the processor 100 judges YES in step S531. On the other hand, if the similarity of the images is less than the predetermined threshold, the result of the first judgment (step S510) is NG, and the processor 100 judges NO in step S531, and performs steps S540 and thereafter.

[0086] The method of the first judgment (step S510) is not limited to the above. For example, the first training data 851 may be a data set in which an endoscopic image after the start of the local injection process (step S500) and information indicating whether the local injection is successful or not are used as a correct answer label. The machine learning unit 802 inputs the first training data 851 to the first training model 841 as a learning stage, and updates it as the first trained model 141. For example, in the learning stage, an endoscopic image is input to the first training model 841, and the first training model 841 outputs a determination result as to whether the local injection is successful or not. Then, the machine learning unit 802 performs feedback to optimize the parameters of the neural network included in the first training model 841 based on the error between the output result and the correct answer label. This updates the first trained model 141. Note that the techniques related to the learning stages are similar to those for the second trained model 142, the third trained model 143, the fourth trained model 144, the fifth trained model 145, and the sixth trained model 146 described below, so they will not be explained further below.

[0087] Then, in the first judgment (step S510), the endoscopic image acquired in step S502 is input to the input unit 110, the processor 100 reads the first trained model 141 from the memory 130, and the inference unit 102 infers the success or failure of the local injection based on the input endoscopic image. If the output unit 120 outputs information indicating that the local injection is successful, the judgment result of the first judgment (step S510) is OK, and the processor 100 judges YES in step S531. On the other hand, if the output unit 120 outputs information indicating that the local injection is not successful, the judgment result of the first judgment (step S510) is NG, and the processor 100 judges NO in step S531, and performs steps S540 and onward.

[0088] Thus, the surgical system 1 of this embodiment includes the first trained model 141 trained based on learning data (first training data 851) including an endoscopic image showing a treatment target into which a local injection liquid is injected. The processor 100 uses the first trained model 141 to determine whether the shape change of the treatment target is excessive or insufficient from the endoscopic image (step S516), and when it is determined that the shape change of the treatment target is insufficient (NO in step S532), it controls the drive device 20 to pull out the injection needle 516 by a predetermined amount (step S540). In this way, the local injection process can be automatically controlled using machine learning based on the endoscopic image.

[0089] Machine learning of the second trained model 142 described later can be performed in the same manner using the learning device 80. In this case, the second training model 842 and the second training data 852 are stored in the memory 830 of the learning device 80. The processor 800 performs machine learning on the second training model 842 using the second training data 852, and the second training model 842 is updated as the second trained model 142 and output to the surgery system 1. Machine learning of the third trained model 143 described later can be performed in the same manner using the learning device 80. In this case, the memory 830 of the learning device 80 stores the third training model 843 and the third training data 853. The processor 800 performs machine learning on the third training model 843 using the third training data 853, and the third training model 843 is updated as the third trained model 143 and output to the surgery system 1. Machine learning of the fourth trained model 144 described later can be performed in the same manner using the learning device 80. In this case, the fourth training model 844 and the fourth training data 854 are stored in the memory 830 of the learning device 80. Then, the processor 800 performs machine learning on the fourth training model 844 using the fourth training data 854, and the fourth training model 844 is updated as the fourth trained model 144 and output to the surgery system 1. Also, machine learning of the fifth trained model 145 described later can be performed in the same manner using the learning device 80. In this case, the fifth training model 845 and the fifth training data 855 are stored in the memory 830 of the learning device 80. Then, the processor 800 performs machine learning on the fifth training model 845 using the fifth training data 855, and the fifth training model 845 is updated as the fifth trained model 145 and output to the surgery system 1. Also, machine learning of the sixth trained model 146 described later can be performed in the same manner using the learning device 80. In this case, the sixth training model 846 and the sixth training data 856 are stored in the memory 830 of the learning device 80. Then, the processor 800 performs machine learning on the sixth training model 846 using the sixth training data 856, and the sixth training model 846 is updated as the sixth learned model 146 and output to the surgical system 1.

[0090] Also, for example, the local injection process (step S500) of this embodiment may be as shown in the flowchart of FIG. 26. In this case, the second judgment (step S520) is performed as the predetermined judgment of this embodiment. Hereinafter, the second judgment as the predetermined judgment is simply described as the second judgment. After performing the above-mentioned step S502, the processor 100 acquires the control value of the injection amount of the local injection liquid (step S504). For example, the processor 100 acquires information on the control value transmitted by the plunger driving unit 212 to the syringe pump 214. Thereafter, the processor 100 performs the second judgment (step S520) and checks the judgment result (OK or NG) of the second judgment (step S532). If the judgment result of the second judgment is NG (NO in step S532), the processor 100 performs the above-mentioned step S540 and performs step S502 again.

[0091] The second judgment (step S520), for example, as shown in the flowchart of Fig. 27, judges whether the local injection is successful or not based on the endoscopic image acquired in step S502 and the control value of the injection amount of the local injection liquid acquired in step S504 (step S522). That is, when the endoscopic image and the control value of the injection amount of the local injection liquid are input as input data to the control device 10, the processor 100 outputs the judgment result of the success or failure of the local injection. In other words, step S522 in Fig. 27 is different from step S512 in Fig. 19 in that input data used for judging the success or failure of the local injection is added.

[0092] For example, although not shown, a physical model of the mucosa and submucosa is stored in the memory 130. Then, the processor 100 calculates a first region, which is a region that will be raised when the local injection liquid of the injection amount corresponding to the control value enters the submucosa, based on the physical model. The processor 100 also extracts a second region, which is a region that is actually raised, from the endoscopic image. Then, the processor 100 obtains the degree of agreement between the first region and the second region, and if the degree of agreement is equal to or greater than a predetermined reference value, the local injection is successful, and the second judgment (step S520) is judged to be OK. As a result, the processor 100 judges YES in step S532 and ends the local injection process (step S500). On the other hand, if the degree of agreement is less than the predetermined reference value, the processor 100 judges NG in the second judgment (step S520) because the shape change of the treatment target can be considered to be insufficient. As a result, the processor 100 judges NO in step S532, performs the above-mentioned step S540, and changes the position of the injection needle 516. After that, in the second step S522 again, if the second region changes and the degree of coincidence between the first region and the second region becomes equal to or greater than a predetermined reference value, the local injection has been successful by changing the position of the injection needle 516, and therefore the second judgment (step S520) judges OK.

[0093] From the above, in the surgical system 1 of this embodiment, the processor 100 judges the success or failure of the local injection in a predetermined judgment (second judgment (step S520)) by further using the control value of the injection amount of the local injection liquid, and controls the drive device 20 to change the position of the injection needle 516 based on the judgment result of the predetermined judgment (NO in step S532, step S540). In this way, the local injection process can be automatically controlled based on the control value of the injection amount of the local injection liquid and the endoscopic image.

[0094] Also, for example, the second judgment (step S520) may be performed as in the processing example shown in the flowchart of FIG. 28. In FIG. 28, the processor 100 acquires an endoscopic image inferred using the first trained model 141 (step S524). Step S524 in FIG. 28 has the same wording as step S514 in FIG. 23, but the data set used for inference is different. As shown in FIG. 29, the inference unit 102 infers an endoscopic image including a raised lesion based on the control value of the injection amount of the local injection liquid using the first trained model 141, using the control value of the injection amount of the local injection liquid acquired in step S504 as input data. Note that the input data may include, for example, information on the local injection position as metadata. Then, for example, as shown in E11, the data of the endoscopic image inferred by the inference unit 102 becomes output data. In other words, the data set used for machine learning of the first trained model 141 in the second judgment (step S520) of Figure 28 differs from the first judgment (step S510) of Figure 23 in that it further includes a control value for the injection amount of local injection liquid in addition to the endoscopic image.

[0095] 29, the manner in which input data is input to the input unit 110, the processor 100 reads the first trained model 141 from the memory 130, the inference unit 102 makes an inference based on the input data, and the inferred output data is output via the output unit 120 is simply illustrated using only the input data, the first trained model 141, and the output data. The same applies to Figures 32, 35, 40, and 43 described below.

[0096] After that, the processor 100 compares the inferred endoscopic image with the endoscopic image acquired from the imager 42 (step S526). That is, the endoscopic image shown at E11 in FIG. 29 is compared with the endoscopic image shown at E12 in which the injection needle 516 is actually inserted and local injection is performed. Note that a process may be added to exclude a portion of the endoscopic image shown at E12 in which the injection needle 516 and the like are reflected from the comparison target. Then, for example, when the similarity between the endoscopic image shown at E11 and the endoscopic image shown at E12 is less than a predetermined threshold, the processor 100 judges NG in the second judgment (step S520) since the shape change of the treatment target is insufficient. In this case, the processor 100 judges NO in step S532 and performs the above-mentioned step S540. Then, again in step S522, if the similarity between the endoscopic image shown in E11 and the endoscopic image shown in E12 becomes equal to or greater than a predetermined threshold, the processor 100 judges the second judgment (step S520) to be OK since the local injection was successful, and then judges the second judgment (step S532) to be YES, thereby terminating the local injection processing (step S500).

[0097] The first training data 851 in this case is a data set in which, for example, data on a target amount of injection fluid for local injection is associated with endoscopic image data showing a lesion into which the target amount of injection fluid is injected as a correct answer label, as shown in Fig. 30. The machine learning unit 802 updates the first trained model 141 by inputting such first training data 851 to the first training model 841.

[0098] For example, the first training data 851 may further include endoscopic image data showing the lesion before the local injection liquid is injected, in addition to the above data. That is, the first training data 851 may be a data set in which data consisting of a combination of data on the target amount of the local injection liquid and endoscopic image data showing the lesion before the local injection liquid is injected is associated with endoscopic image data showing the lesion into which the target amount of the local injection liquid is injected as a correct answer label. Then, the inference unit 102 uses the first trained model 141 trained based on such first training data 851 to infer an endoscopic image including a raised lesion based on the control value of the injection amount of the local injection liquid acquired in step S504 and the endoscopic image data showing the lesion before the local injection liquid is injected.

[0099] Also, for example, although illustration is omitted, the first training data 851 may be a data set in which the data of the target amount of the injection liquid of the local injection is associated with the value of the area of ​​the region raised after the local injection as a correct answer label. Alternatively, the value of the volume of the part raised after the local injection may be the correct answer label. Then, the inference unit 102 uses the first trained model 141 trained based on such first training data 851 to infer the area of ​​the raised region, etc., based on the control value of the injection amount of the local injection liquid acquired in step S504. Then, the processor 100 performs a process of extracting the raised region from the endoscopic image, a process of determining the degree of agreement between the area of ​​the extracted region, etc., and the area of ​​the inferred region, etc., and a process of determining whether the degree of agreement is equal to or greater than a predetermined reference value. If the degree of agreement is equal to or greater than the predetermined reference value, the processor 100 determines OK in the second determination (step S520), and then determines YES in step S532, and ends the local injection process (step S500).

[0100] Thus, the surgical system 1 of this embodiment includes the first trained model 141 trained based on learning data (first training data 851) including the injection amount of the local injection liquid and an endoscopic image showing the treatment target into which the local injection liquid corresponding to the injection amount is injected. The processor 100 uses the first trained model 141 to determine whether the shape change of the treatment target is excessive or insufficient relative to the control value from the endoscopic image, and if the shape change of the treatment target is insufficient (NO in step S532), controls the drive device 20 to pull out the injection needle 516 by a predetermined amount (step S540). In this way, the local injection process can be automatically controlled using the control value of the injection amount of the local injection liquid and machine learning using the endoscopic image.

[0101] Also, for example, the success or failure of local injection may be determined by performing measurement by a sensor provided around the syringe 512 in combination with the first judgment (step S510) described above in FIG. 19 or the second judgment (step S520) described above in FIG. 27. As described above in FIG. 20, when the tip of the injection needle 516 is located in the muscle layer shown at L3 in FIG. 20, the local injection liquid does not enter. Therefore, even if the syringe pump 214 pushes the plunger 514 based on the control value transmitted from the plunger driving unit 212 to the syringe pump 214, the force of the syringe pump 214 pushing the plunger 514 and the resistance force that the syringe pump 214 receives from the plunger 514 are balanced, and the plunger 514 appears to be stationary in appearance.

[0102] Therefore, although detailed illustration is omitted, for example, a force sensor for measuring the resistance force received from the plunger 514 is provided at the position indicated by B11 in FIG. 7 in the syringe pump 214. Then, the processor 100 judges whether or not the shape of the treatment target has changed due to the injection of the local injection liquid based on the endoscopic image as described above, and measures the detection value of the force sensor. When the tip of the injection needle 516 is located in the muscle layer, the resistance force received from the plunger 514 is considered to be equal to the upper limit value of the force with which the syringe pump 214 presses the plunger 514. Therefore, for example, a value obtained by multiplying the upper limit value by a predetermined ratio is set as a first predetermined value, and when the resistance force measured by the force sensor is equal to or greater than the first predetermined value, it can be judged that the tip of the injection needle 516 is located in the muscle layer. This makes it possible to further support the judgment result when, for example, the judgment result of the first judgment (step S510) or the second judgment (step S520) is NG.

[0103] On the other hand, if the resistance measured by the force sensor is less than the first predetermined value, the plunger 514 is moving forward, and it can be determined that the tip of the injection needle 516 is located in the submucosal layer shown in L2 of FIG. 21. As a result, the processor 100 determines YES as the determination result of the second determination (step S520), and the flow of the local injection process (step S500) ends. That is, the surgical system 1 of this embodiment includes a force sensor capable of detecting the resistance when injecting the local injection liquid into the treatment target. If the shape change of the treatment target is insufficient and the detection value of the force sensor is equal to or greater than the first predetermined value (NO in step S531, NO in step S532), the processor 100 controls the drive device 20 to pull out the injection needle 516 by a predetermined amount (step S540). In this way, the success or failure of the local injection can be determined more accurately based on the measurement result of the resistance from the syringe 512.

[0104] Alternatively, a position sensor capable of detecting the amount of displacement of the plunger 514 may be used, although illustration is omitted. The plunger 514 can be displaced within the range shown in D18 of FIG. 7. The position sensor here may be a distance measuring sensor or an encoder. As described above, when the tip of the injection needle 516 is located in the muscle layer shown in L3 of FIG. 20, it is considered that the plunger 514 has hardly moved. Therefore, for example, the detection value of the position sensor corresponding to the amount of displacement that can be treated as the plunger 514 not substantially moving forward is set as a second predetermined value. Then, when the detection value of the position sensor to be measured is less than the second predetermined value, it can be determined that the plunger 514 has not substantially moved forward and therefore the tip of the injection needle 516 is located in the muscle layer. This allows the determination result to be further supported when the determination result of the first determination (step S510) or the second determination (step S520) is NG.

[0105] That is, the surgical system 1 of this embodiment includes a position sensor capable of detecting the displacement amount of the plunger 514. When the shape change of the treatment target is insufficient and the detection value of the position sensor is less than the second predetermined value (NO in step S531, NO in step S532), the processor 100 controls the drive device 20 to pull out the injection needle 516 by a predetermined amount (step S540). In this way, the success or failure of local injection can be more accurately determined based on the measurement result of the movement amount of the plunger 514.

[0106] Also, for example, instead of the above-mentioned step S100, a first determination process (step S110) may be performed to determine the position to insert the injection needle 516 by machine learning. An example of the first determination process (step S110) will be described with reference to the flowchart of Fig. 31. The processor 100 uses the second trained model 142 to determine the position to insert the injection needle 516 based on the endoscopic image (step S118).

[0107] For example, as shown at E21 in Fig. 32, an endoscopic image showing a lesion to be treated is input as input data to the input unit 110. Then, the inference unit 102 performs inference using the second trained model 142, and as shown at E22, an endoscopic image to which information on the local injection position has been added is output from the output unit 120. More specifically, in the endoscopic image shown at E22, position information shown at F21, position information shown at F22, position information shown at F23, position information shown at F24, position information shown at F25, and position information shown at F26 have been added around the lesion.

[0108] 32, order information indicating the order in which the injection needle 516 is inserted is output as output data from the output unit 120. For example, the position shown in F21 is displayed as the position for inserting the first injection needle 516, the position shown in F22 as the position for inserting the second injection needle 516, and the position shown in F23 as the position for inserting the third injection needle 516. The information indicating the order shown in E23 is also displayed as the position shown in F24 as the position for inserting the fourth injection needle 516, the position shown in F25 as the position for inserting the fifth injection needle 516, and the position shown in F26 as the position for inserting the sixth injection needle 516.

[0109] As shown in FIG. 33, the second trained model 142 is generated by performing machine learning on the second training model 842 using the second training data 852 as a data set. The second training data 852 is a data set in which information on local injection positions is associated as a correct answer label with an endoscopic image showing a lesion based on past cases. The second training data 852 includes a large amount of data consisting of combinations of lesions of various shapes and local injection positions for the lesions. In addition, the second training data 852 may further include order information as a correct answer label. For example, the data shown in G20 in FIG. 33 is data based on a case in which the injection needle 516 was inserted into the lesion in the order of positions shown in G21, G22, G23, and G24.

[0110] For the above reasons, the surgical system 1 of this embodiment includes a second trained model 142 trained based on training data including an endoscopic image showing a treatment target and position information for inserting the injection needle 516. The processor 100 performs a first determination process (step S110) for determining a position for inserting the injection needle 516 based on the endoscopic image using the second trained model 142. In this manner, the position for inserting the injection needle 516 can be automatically determined in ESD treatment or the like.

[0111] Also, for example, instead of the above-mentioned step S200, a second determination process (step S220) may be performed to determine the target amount of locally injected liquid by machine learning. An example of the second determination process (step S220) will be described with reference to the flowchart of FIG. 34. The processor 100 determines the target amount of locally injected liquid at each position based on the endoscopic image and position information by using the third trained model 143 (step S226). For example, as shown at E31 and E32 in FIG. 35, data consisting of the endoscopic image, position information, and order information output by the above-mentioned step S118 is input to the input unit 110. That is, the data output from the output unit 120 by the inference related to step S110 is input to the input unit 110 as it is.

[0112] Then, the inference unit 102 reads out the third trained model 143 to perform inference, and in addition to the endoscopic image and position information shown at E33 in Fig. 35, the target amount of the local injection liquid at each position is output as output data from the output unit 120 as shown at E34. For example, the target amount shown at F131 is displayed for the position shown at F31, the target amount shown at F132 is displayed for the position shown at F32, the target amount shown at F133 is displayed for the position shown at F33, the target amount shown at F134 is displayed for the position shown at F34, the target amount shown at F135 is displayed for the position shown at F35, and the target amount shown at F136 is displayed for the position shown at F36.

[0113] As shown in FIG. 36, the third trained model 143 is generated by performing machine learning on the third training model 843 using the third training data 853 as a data set. In the third training data 853, information on the endoscopic image shown in G32 is associated with the endoscopic image shown in G31 as a correct answer label. The endoscopic image shown in G31 includes an image of a lesion and information on the local injection position shown in G131. The endoscopic image shown in G32 is an image after local injection is performed at the local injection position shown in G131. More specifically, it is an endoscopic image in which information on the injection amount of the local injection liquid shown in G132 is associated with image information on the lesion raised by the injection amount shown in G133.

[0114] From the above, the surgical system 1 of this embodiment includes a third trained model 143 trained based on training data (third training data 853) including an endoscopic image showing a treatment target, position information for inserting the injection needle 516, and a target amount of local injection liquid corresponding to the position information. The processor 100 uses the third trained model 143 to perform a second determination process (step S220) for determining a target amount of local injection liquid at each position for inserting the injection needle 516 based on the endoscopic image and the position information determined in the first determination process (step S110). In this way, the target amount of local injection liquid at each position for inserting the injection needle 516 can be automatically determined.

[0115] Note that there may be a plurality of third trained models 143. Specifically, for example, subjects may be classified into a plurality of categories in consideration of gender, age, and the like, and the third trained models 143 corresponding to each category may be stored in the memory 130. In this case, the second determination process (step S220) may be performed as an example of a process shown in the flowchart of FIG. 37. The processor 100 acquires information on the subject (step S222), and selects the corresponding third trained model 143 based on the acquired subject information (step S224). More specifically, the user inputs information that the subject is a woman in her 40s from a medical record or the like to the control device 10, and stores the information in the memory 130. Then, in step S222, the processor 100 acquires information that the subject is a woman in her 40s. Then, in step S224, the processor 100 reads out the third trained model 143 corresponding to a woman in her 40s from the memory 130. Thereafter, the processor 100 performs the processing of the above-mentioned step S226 using the third trained model 143 read in step S224.

[0116] Although not shown, for example, in FIG. 35, the information shown in E33 may be three-dimensional shape information changed by performing local injection at each position, instead of the amount of local injection of the local injection liquid at each position. In other words, the target index is how much the surrounding area where the injection needle 516 is inserted is raised by performing local injection at each position where the injection needle 516 is inserted. In this case, the third training data 853 may be associated with three-dimensional shape information changed by performing local injection, instead of the amount of local injection of the local injection liquid shown in G132 of FIG. 36, in the data set described above in FIG. 36.

[0117] Also, the endoscopic image shown in E21 of FIG. 32 may be acquired using the touch panel 620 of the console 60. Specifically, for example, as shown in FIG. 38, a lesion shown in C21 is displayed on the display 610, and the lesion is also displayed on the touch panel 620. At this time, the user operates the touch panel 620 to surround the periphery of the lesion with an icon made of a predetermined figure shown in B91. Note that the predetermined figure is an ellipse in FIG. 38, but it may be a polygon, or the user may be able to create any shape by using a drawing function. Also, the icon made of a predetermined figure shown in B91 may be displayed on the display 610 as shown in B92. Then, through the first determination process (step S110) and the second determination process (step S220) described above, the position information shown in E32 of FIG. 35 and the information on the injection amount of the local injection liquid shown in E33 may be displayed on the display 610. From the above, in the surgical system 1 of this embodiment, the processor 100 determines, based on an endoscopic image, the position in the treatment target to which the injection needle 516 is inserted and the target amount of local injection liquid corresponding to the position to which the injection needle 516 is inserted. In this way, a system can be constructed that automatically determines the position to which the injection needle 516 is inserted and the injection amount of local injection liquid at that position by acquiring an endoscopic image.

[0118] Moreover, the direction information in step S300 in Fig. 14 etc. may be set using machine learning. Specifically, for example, the processor 100 may perform a third determination process (step S330) shown in the flowchart in Fig. 39 instead of step S300.

[0119] The processor 100 acquires output data from the above-mentioned third trained model 143 (step S332). That is, the data output from the output unit 120 by the inference related to step S220 is input as it is to the input unit 110. After that, the processor 100 determines directional information of the injection needle 516 using the fourth trained model 144 (step S334).

[0120] For example, data shown at E41 and E42 in FIG. 40 is input to the input unit 110. The data shown at E41 and E42 in FIG. 40 corresponds to the data shown at E22 and E23 in FIG. 32. The inference unit 102 then performs inference using the fourth trained model 144. As a result, as shown at E43, data in which the local injection position included in the data of E41 is associated with injection needle direction information, which is the orientation of the injection needle 516 when performing the above-mentioned local injection process (step S500), is output via the output unit 120.

[0121] 41, the fourth trained model 144 is generated by performing machine learning on the fourth training model 844 using the fourth training data 854 as a data set. The fourth training data 854 is a data set in which an endoscopic image including information on a local injection position is associated with injection needle direction information corresponding to the position as a correct answer label.

[0122] Returning to the description of FIG. 39, after performing the above-mentioned step S334, the processor 100 determines the trajectory information of the injection needle 516 based on the direction information of the injection needle 516 (step S339). That is, in order to make the position information on the image acquired by the first determination process (step S110), the second determination process (step S220), and the third determination process (step S330) correspond to the coordinate information of the endoscope 40 and the like in the actual treatment, a calculation process of the parameters for controlling the driving device 20 is performed. In this way, the position for performing the local injection process (step S500), the amount of the local injection liquid, and the driving information of the endoscope 40 and the first medical manipulator 510 are obtained, and the control device 10 controls the driving device 20 so as to correspond to this information, thereby realizing automatic control of the local injection.

[0123] Also, instead of steps S100 and S200 described above with reference to Fig. 14 and the like, the position where local injection is performed and the target amount of local injection liquid at that position may be set by performing, for example, the processing example shown in Fig. 42. The processor 100 acquires region information (step S710). For example, as described above, the user uses the touch panel 620 to specify the periphery of the lesion to be treated shown in C31 with a predetermined figure shown in H10, and the processor 100 acquires the region related to the predetermined figure as region information.

[0124] After performing step S710, the processor 100 measures the area of ​​the lesion included in the region information (step S720). After performing step S710, the processor 100 also acquires a predetermined data set shown in H20 based on the region information (step S730). Details of step S730 will be described later, but the predetermined data set is, for example, a data set in which a target amount of injection liquid and a predetermined image showing a shape change caused by the protuberance corresponding to the target amount are associated. In other words, when the target amount of locally injected liquid is small, the predetermined image is associated small, and when the target amount of locally injected liquid is large, the predetermined image is associated large.

[0125] The processor 100 performs step S740 based on steps S720 and S730, and ends the flow. Step S740 sets the local injection position and the target amount of the local injection liquid based on the area measured in step S720 and the predetermined data set acquired in step S730.

[0126] Then, as shown in H30, the size and arrangement of the predetermined image are determined so that the entire lesion shown in C31 will be raised. This makes it possible to grasp the amount and position of the local injection fluid required to raise the entire lesion.

[0127] Step S730 will be conceptually explained using Fig. 43. When the data of the region information acquired in step S710 is input to the input unit 110, the processor 100 reads out the fifth trained model 145 from the memory 130. Then, the inference unit 102 outputs data of the tissue feature amount related to the region information via the output unit 120 based on the fifth trained model 145. The tissue feature amount is, for example, the viscoelasticity of the tissue, the tissue thickness, etc.

[0128] The output data on the tissue features is input again to the input unit 110, and the processor 100 reads out the sixth trained model 146 from the memory 130. The inference unit 102 then outputs a predetermined data set corresponding to the tissue features via the output unit 120 based on the sixth trained model 146.

[0129] 44, the fifth trained model 145 is generated by performing machine learning on the fifth training model 845 using the fifth training data 855 as a data set. The fifth training data 855 is a data set in which information on the feature amount of tissue including a lesion is associated as a correct answer label with respect to an endoscopic image showing the tissue.

[0130] As shown in FIG. 45, the sixth trained model 146 is generated by performing machine learning on the sixth training model 846 using the sixth training data 856 as a data set. The sixth training data 856 is a data set in which the following predetermined data is associated as a correct answer label with respect to feature amount information of tissue. The predetermined data is, for example, information on the height and size of the raised part shown in E61, information on the position where the local injection was performed, information on the amount of the local injection liquid, etc., extracted from an endoscopic image shown in E60 showing that the tissue including the lesion has been raised due to the local injection.

[0131] As a modified example of the method of this embodiment, ESD may be performed according to the treatment flow shown in FIG. 46. The above-mentioned FIG. 14 shows a treatment flow in which local injection processing (step S500) is performed on the entire periphery of the lesion to be treated, followed by incision (step S610), whereas FIG. 46 shows a treatment flow in which local injection processing (step S500) and incision (step S610) are repeatedly performed locally. More specifically, after setting the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 (step S10), the processor 100 determines one local injection position by the above-mentioned step S100. Thereafter, the processor 100 performs steps S200, S300, local injection processing (step S500), and incision (step S610) for the determined one local injection position. In the case of the treatment flow of FIG. 46, since the position where the local injection is performed is one, the incision (step S610) can be performed immediately after the local injection process (step S500). After that, the user judges whether the local injection process (step S500) and the incision (step S610) have been performed for all positions (step S614). For example, the user judges from the endoscopic image whether the lesion where the incision (step S610) has been performed is floating above the submucosa. Then, when the user judges that the local injection process (step S500) and the incision (step S610) have been performed for all positions (YES in step S614), the user performs the peeling (step S620) as in FIG. 14. On the other hand, when the user judges that the lesion is not floating above the submucosa, the user judges NO in step S614 and performs step S100 again.

[0132] The first decision process (step S110) described in Fig. 31 etc., the second decision process (step S220) described in Fig. 34 etc., and the third decision process (step S330) described in Fig. 39 etc. may be applied to the treatment flow of Fig. 46. That is, although not shown, Fig. 46 may be a treatment flow in which, after step S10, the first decision process (step S110), the second decision process (step S220), the third decision process (step S330), the local injection process (step S500), and the incision (step S610) are repeated.

[0133] A treatment example in the modified example will be specifically described. In step S10, the user sets the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 as shown in FIG. 2 (step S10). Then, the user operates the first handle 641 while stepping on the second foot pedal 632 of the console 60 to operate the endoscope 40 so that the lesion to be treated, shown in C41, is displayed on the display 610, as shown in the screen example shown in J10 of FIG. 47. Then, the user operates the touch panel 620 to select a part of the lesion, as shown in J11. The processor 100 performs a first determination process (step S110) based on the endoscopic image in the selected range, and determines the position to perform the local injection process (step S500). Then, the processor 100 performs a second determination process (step S220) to set the target amount of local injection liquid at the position. Then, the processor 100 performs a third determination process (step S330) to determine the direction information for inserting the injection needle 516 into the position.

[0134] The processor 100 may further perform a process of presenting at least one result of the first determination process (step S110), the second determination process (step S220), and the third determination process (step S330) to the user and asking the user to determine whether the result is OK or NG. For example, in the example screen of J20 in Fig. 47, position information on the position where the local injection process (step S500) is performed, shown at J21, information on the target amount of the local injection liquid, shown at J22, and information on the direction information for inserting the injection needle 516, shown at J23, are displayed on the display 610.

[0135] The screen example shown in J20 in FIG. 47 can be realized by, for example, making the third determination process (step S330) the process example shown in the flowchart in FIG. 48. The processor 100 performs steps S322 and S334 as in FIG. 39, and then performs a process of presenting the determined various information (step S336). Then, a process of determining whether or not there is an OK instruction from the user (step S338) is performed. For example, the processor 100 determines whether it is OK or NG based on a control signal transmitted from a switch or the like (not shown) of the console 60. If there is an OK instruction from the user (YES in step S338), the processor 100 performs the above-mentioned step S339. After step S339 is performed, the user operates the first handle 641 while stepping on the first foot pedal 631 of the console 60 to perform marking at the position shown in J21 using the third treatment tool 53. Although not shown in the flow chart, if an instruction of NG is given by the user (NO in step S338), the flow of the third determination process (step S330) is terminated, and then step S100 is executed again.

[0136] Thereafter, the user inserts the injection needle 516 into the marked position, and local injection processing (step S500) is performed. For example, the user operates the first handle 641 while stepping on the second foot pedal 632 of the console 60 to operate the endoscope 40 so that the tip of the endoscope 40 faces the position indicated by J21. The user then operates the first handle 641 while stepping on the third foot pedal 633 of the console 60 to advance the first medical manipulator 510 and insert the injection needle 516 into the marked position. Alternatively, the driving information of the endoscope 40 and the driving information of the first medical manipulator 510 required to insert the injection needle 516 may be programmed in step S339, and the process of inserting the injection needle 516 may be automatically controlled. Then, the process of the local injection liquid entering the submucosa is automatically controlled by the local injection processing (step S500). This allows the user to observe the success or failure of local injection by looking at the screen example shown in J30 of FIG. 49, thereby reducing the burden on the user.

[0137] Thereafter, if the user determines that the local injection has been successful, he or she operates the second treatment tool 52 by operating the second handle 642 while stepping on the first foot pedal 631 of the console 60, and operates the third treatment tool 53 by operating the first handle 641. As a result, as shown in the example screen of J40 in Fig. 49, with the raised lesion being grasped by the grasping portion 522, the desired location is incised by the third treatment tool 53.

[0138] In this manner, the steps shown in the screen examples of Figures 47 and 49 are repeated. Then, when the entire lesion is raised above the submucosal layer, the user operates the second handle 642 while stepping on the first foot pedal 631 of the console 60, and retracts the second medical manipulator 520 to the outside of the body while holding the lesion with the holding part 522, thereby recovering the excised lesion.

[0139] In the example screen of J10 in FIG. 47, the user operates the touch panel 620 to determine the position where local injection is performed, but the present invention is not limited to this, and the processor 100 may automatically determine the position where local injection is performed. In this case, the first determination process (step S110) may be, for example, the process example shown in the flowchart of FIG. 50. After performing step S112, the processor 100 performs a process of determining whether or not a region requiring local injection exists (step S114). When the processor 100 determines that a region requiring local injection exists (YES in step S114), it performs a process of designating a predetermined region (step S116). Step S116 can be realized, for example, by the following method. For example, a position sensor is included in the tip of the endoscope 40, and the processor 100 acquires position information of the tip of the endoscope 40. The processor 100 then compares the endoscopic image of the lumen surface estimated from the position information with the endoscopic image of the lumen surface actually displayed, and if they are different, it is determined that a lesion may exist, and designates the corresponding region as a predetermined region. After that, the processor 100 performs the above-mentioned step S118.

[0140] Also, the timing to perform step S114 may be automatically determined. For example, polling may be performed for a certain period of time after the local injection process (step S500), and step S114 in the first determination process (step S110) may be executed again after the certain period of time has elapsed. Alternatively, the processor 100 may determine the timing when the local injection is completed when it is determined from the endoscopic image captured by the 3D camera that the part that was raised by the local injection has changed to a shape close to flat, and may perform step S114. Alternatively, the processor 100 may determine from the endoscopic image whether the color of the submucosa reflects the color of the dye contained in the local injection liquid, and may determine that the local injection is completed when the color density of the dye becomes less than a certain value. Alternatively, the third treatment tool 53 may include a position sensor, and the processor 100 may determine the timing when the third treatment tool 53 moves a certain distance away from the position of the treatment target as the timing when the local injection is completed. In this way, step S114 is automatically performed, and therefore the looping process in the treatment flow of FIG. 46 is automatically performed. In other words, since the local injection using the first treatment tool 51 is completely automated, the user can concentrate on incision (step S610) and dissection (step S620) using the second treatment tool 52 and the third treatment tool 53.

[0141] Although the embodiments to which the present disclosure is applied and their modifications have been described above, the present disclosure is not limited to each embodiment or its modifications as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist of the disclosure. In addition, various disclosures can be formed by appropriately combining multiple components disclosed in each of the above-mentioned embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Furthermore, components described in different embodiments or modifications may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with that different term in any part of the specification or drawings. [Explanation of symbols]

[0142] 1...surgical system, 10...control device, 14...input section, 20...drive device, 21...first treatment tool drive device, 22...second treatment tool drive device, 23...third treatment tool drive device, 24...endoscope drive device, 40...endoscope, 42...imager, 44...treatment tool insertion port, 46...overtube, 48...cap, 50...treatment tool, 51...first treatment tool, 52...second treatment tool, 53...third treatment tool, 60...console, 70...driving unit, 80...learning device, 100...processor, 102...inference section, 110...input section, 1 20...output unit, 130...memory, 141...first trained model, 142...second trained model, 143...third trained model, 144...fourth trained model, 145...fifth trained model, 146...sixth trained model, 212...plunger drive unit, 214...syringe pump, 215...advance and retreat operation drive unit, 217...slider mechanism device, 218...fixed unit, 219...slider unit, 220...motor unit, 221...first bending operation drive unit, 222...second bending operation drive unit, 223...opening and closing operation drive unit, 22 4...Roll motion drive unit, 225...Advance and retreat motion drive unit, 500...Medical manipulator, 510...First medical manipulator, 512...Syringe, 514...Plunger, 516...Injection needle, 520...Second medical manipulator, 522...Grip, 530...Third medical manipulator, 610...Display, 620...Touch panel, 630...Foot pedal, 631...First foot pedal, 632...Second foot pedal, 633...Third foot pedal, 640...Handle, 641...First handle, 642...Second 2 handle, 651...first part, 652...second part, 653...third part, 800...processor, 802...machine learning unit, 830...memory, 841...first training model, 842...second training model, 843...third training model, 844...fourth training model, 845...fifth training model, 846...sixth training model, 851...first training data, 852...second training data, 853...third training data, 854...fourth training data, 855...fifth training data, 856...sixth training data, 860...communication unit, T...operating table

Claims

1. an endoscope including an imager for capturing an endoscopic image; A medical manipulator having an injection needle at its tip; a drive unit that controls the medical manipulator to control the position of the injection needle; A processor; Including, The processor, Controlling the syringe connected to the injection needle to inject a target amount of local injection liquid; acquiring the endoscopic image showing the treatment target from the imager; A surgical system characterized by using the endoscopic image to make a predetermined judgment as to whether local injection has been successful or not, and controlling the drive device to change the position of the injection needle based on the result of the predetermined judgment.

2. 2. The surgical system of claim 1, A first trained model trained based on training data including the endoscopic image showing the treatment target into which the local injection liquid is injected, The processor, Using the first trained model, determine whether a shape change of the treatment target is excessive or insufficient from the endoscopic image; a surgical system that controls the drive device so as to withdraw the injection needle a predetermined amount when it is determined that the change in shape of the treatment target is insufficient.

3. 2. The surgical system of claim 1, The processor, In the predetermined judgment, a control value of the injection amount of the local injection liquid is further used to judge whether the local injection has been successful or not; The surgical system controls the drive device so as to change the position of the injection needle based on the result of the predetermined judgment.

4. The surgical system of claim 3, A first trained model trained based on training data including the injection amount of the local injection liquid and the endoscopic image showing the treatment target into which the local injection liquid corresponding to the injection amount is injected, The processor, Using the first trained model, determine whether a change in shape of the treatment target is excessive or insufficient with respect to the control value from the endoscopic image; A surgical system comprising: a driving device that controls the driving device so as to withdraw the injection needle by a predetermined amount when the change in shape of the treatment target is insufficient.

5. In the surgical system of claim 1 or 3, a force sensor capable of detecting a resistance force when the local injection liquid is injected into the treatment target, The processor, A surgical system comprising: a surgical instrument that controls the drive device to withdraw the injection needle by a predetermined amount when there is insufficient change in the shape of the treatment target and the detection value of the force sensor is equal to or greater than a predetermined value.

6. In the surgical system of claim 1 or 3, a position sensor capable of detecting a displacement amount of the plunger; The processor, A surgical system comprising: a surgical instrument that controls the drive device so as to withdraw the injection needle by a predetermined distance when the change in shape of the treatment target is insufficient and the detection value of the position sensor is less than a predetermined value.

7. 2. The surgical system of claim 1, The processor, A surgical system characterized in that a position at which the injection needle is to be inserted into the treatment target and the target amount of the local injection liquid corresponding to the position at which the injection needle is to be inserted are determined based on the endoscopic image.

8. The surgical system of claim 7, A second trained model trained based on training data including the endoscopic image showing the treatment target and position information for inserting the injection needle, The processor, A surgical system comprising: a first determination process that uses the second learned model to determine a position to insert the injection needle based on the endoscopic image.

9. The surgical system of claim 8. a third trained model trained based on training data including the endoscopic image showing the treatment target, the position information for inserting the injection needle, and the target amount of the local injection liquid corresponding to the position information; The processor, A surgical system characterized by performing a second determination process using the third learned model to determine the target amount of the local injection liquid at each position where the injection needle is inserted, based on the endoscopic image and the position information determined in the first determination process.

10. A processor for controlling an endoscope including an imager for capturing an endoscopic image and a driving device for controlling a medical manipulator having an injection needle at a tip thereof to thereby control a position of the injection needle, Controlling the syringe connected to the injection needle to inject a target amount of local injection liquid; acquiring the endoscopic image showing the treatment target from the imager; A processor that performs a predetermined judgment using the endoscopic image to judge whether local injection has been successful or not, and controls the drive device to change the position of the injection needle based on the result of the predetermined judgment.

11. 1. A control method for controlling an endoscope including an imager that captures an endoscopic image and a driving device that controls a position of an injection needle by controlling a medical manipulator having an injection needle at a tip end, comprising: Controlling the syringe connected to the injection needle to inject a target amount of local injection liquid; acquiring the endoscopic image showing the treatment target from the imager; A control method comprising: making a predetermined judgment using the endoscopic image to judge whether local injection has been successful; and controlling the drive device to change the position of the injection needle based on the judgment result of the predetermined judgment.

Images