Medical Manipulator System

Abstract

To provide a medical manipulator system that automatically moves the position of an endoscope according to the progress of a procedure. [Solution] The medical manipulator system 5 includes an endoscope 40 including an imager 42 that captures endoscopic images, a medical manipulator 500 that protrudes beyond the distal end surface of the endoscope 40, a drive unit 20 that controls the endoscope 40 and the medical manipulator 500, and a processor 100. The processor 100 acquires information about a working area, which is the range in which the medical manipulator 500 can work, by calculating the information about the multiple working areas for each of multiple positions of the endoscope 40. The processor 100 also drives and controls the endoscope 40 to a first position corresponding to a first working area selected from the multiple working areas, based on the area to be treated, three-dimensional shape information of the lumen, and three-dimensional shape information of the endoscope 40.

Description

[Technical Field]

[0001] The present invention relates to a medical manipulator system, a processor, a control method, and the like. [Background technology]

[0002] There are known surgical systems that control medical instruments such as endoscopes with robotic arms, etc. Patent Document 1 discloses a flexible robotic endoscope system that stores information on the initial position for performing treatments such as ESD in a memory and moves an insertion shaft based on the stored position information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 048194 Summary of the Invention [Problem to be solved by the invention]

[0004] The technique disclosed in Patent Document 1 does not take into consideration driving the endoscope to the optimum position each time as the procedure progresses. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to a medical manipulator system including an endoscope including an imager that captures endoscopic images, a medical manipulator that protrudes beyond the distal end surface of the endoscope, a drive device that controls the endoscope and the medical manipulator, and a processor, wherein the processor acquires the endoscopic image of a treatment target from the imager, recognizes a region of the treatment target from the endoscopic image, acquires three-dimensional shape information of a lumen, acquires three-dimensional shape information of the endoscope, calculates information about a working region, which is a range in which the medical manipulator can work, for each of a plurality of positions in the movable range of the endoscope, thereby acquiring information about a plurality of working regions, and drive-controls the endoscope to a first position corresponding to a first working region selected from the plurality of working regions based on the region of the treatment target, the three-dimensional shape information of the lumen, and the three-dimensional shape information of the endoscope.

[0006] Another aspect of the present disclosure relates to a processor that controls an endoscope including an imager that captures endoscopic images, a medical manipulator that protrudes beyond the distal end surface of the endoscope, and a drive device that controls the endoscope and the medical manipulator, the processor acquiring the endoscopic image showing a treatment target from the imager, recognizing an area of ​​the treatment target from the endoscopic image, acquiring three-dimensional shape information of a lumen, acquiring three-dimensional shape information of the endoscope, calculating information about a working area within which the medical manipulator can work for each of a plurality of positions within the movable range of the endoscope, thereby acquiring information about a plurality of working areas, and driving and controlling the endoscope to a first position corresponding to a first working area selected from the plurality of working areas based on the area of ​​the treatment target, the three-dimensional shape information of the lumen, and the three-dimensional shape information of the endoscope.

[0007] Another aspect of the present disclosure relates to a control method for controlling an endoscope including an imager that captures endoscopic images, a medical manipulator that protrudes beyond the distal end surface of the endoscope, and a drive device that controls the endoscope and the medical manipulator, the control method comprising: acquiring an endoscopic image showing a treatment target from the imager; recognizing an area of ​​the treatment target from the endoscopic image; acquiring three-dimensional shape information of a lumen; acquiring three-dimensional shape information of the endoscope; calculating information about a working area, which is an area in which the medical manipulator can work, for each of a plurality of positions within the movable range of the endoscope, thereby acquiring information about a plurality of working areas; and driving and controlling the endoscope to a first position corresponding to a first working area selected from the plurality of working areas based on the area of ​​the treatment target, the three-dimensional shape information of the lumen, and the three-dimensional shape information of the endoscope. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a medical manipulator system. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a driving device. [Figure 3] FIG. 10 is a diagram illustrating an example including three treatment instrument channels. [Figure 4] FIG. 10 is another diagram illustrating an example including three treatment tool channels. [Figure 5] FIG. 10 is another diagram illustrating an example including three treatment tool channels. [Figure 6] FIG. 4 is a diagram for explaining a configuration example of a second treatment tool driving device. [Figure 7] A diagram illustrating an example of the configuration of a surgical system. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of a console. [Figure 9] FIG. 1 illustrates a drive mechanism controlled by a foot pedal and a handle. [Figure 10] FIG. 2 is a diagram illustrating an example of the configuration of a handle. [Figure 11] FIG. 10 is a diagram illustrating an example of settings for a control signal transmitted from a console. [Figure 12]FIG. 10 is a diagram illustrating an example of a screen displayed on the console display. [Figure 13] 10 is a flowchart illustrating an example of a procedure to which the technique of this embodiment is applied. [Figure 14] 10 is a flowchart illustrating an example of processing according to the method of the present embodiment. [Figure 15] 10 is a flowchart illustrating an example of processing related to acquisition of three-dimensional information of a treatment target. [Figure 16] 10 is a flowchart illustrating setting the position and order of work areas. [Figure 17] FIG. 10 is a block diagram illustrating another example of the configuration of an endoscope. [Figure 18] FIG. 4A is a diagram illustrating an example of a second sensor, FIG. 4B is a diagram illustrating another example of the second sensor, and FIG. 4C is a diagram illustrating an example of an optical fiber used in the second sensor. [Figure 19] FIG. 10 is a diagram illustrating a standard for measuring position information of a treatment target. [Figure 20] FIG. 2 is a diagram illustrating the relationship between a reference coordinate system and the tip of an endoscope. [Figure 21] FIG. 1A is a diagram illustrating position information of a treatment target measured from a first sensor, and FIG. 1B is a diagram illustrating position information of a treatment target measured from the tip of an endoscope. [Figure 22] 10A and 10B are diagrams illustrating the relationship between the coordinates of a treatment target in an endoscopic image and the coordinates of the tip of an endoscope. [Figure 23] 10 is a flowchart illustrating an example of processing related to calculation of the movable range of the tip of the endoscope. [Figure 24] FIG. 3 is a diagram illustrating the range of motion of an endoscope. [Figure 25] (A) and (B) are different diagrams illustrating the range of motion of the endoscope. [Figure 26] FIG. 10 is another diagram illustrating the range of motion of the endoscope. [Figure 27] 10 is a flowchart illustrating an example of a process for calculating a first working area. [Figure 28] FIG. 4A is a diagram illustrating the relationship between a first spatial region and an imaging range of an endoscopic image, and FIG. 4B is a diagram illustrating an example of a predetermined range of an endoscopic image. [Figure 29]FIG. 10 is a diagram for explaining an example of calculation of a first working area. [Figure 30] FIG. 10 is another diagram illustrating an example of the operation of the first working area. [Figure 31] 10(A) and 10(B) are diagrams illustrating an example of the relationship between the direction of the tip of the overtube, the direction of the tip of the endoscope, and the direction along the inner wall of the lumen. [Figure 32] 10A and 10B are diagrams for explaining examples of angles formed between the direction in which the treatment tool advances and the direction of the distal end of the treatment tool. [Figure 33] 10A and 10B are diagrams illustrating examples of angles at which the distal end of a treatment tool points toward a treatment target. [Figure 34] 10A and 10B are diagrams illustrating another example of the relationship between the orientation of the tip of the overtube, the orientation of the tip of the endoscope, and the orientation along the inner wall of the lumen. [Figure 35] FIG. 10 is a diagram illustrating an example of setting a first set of first working areas. [Figure 36] 10 is a flowchart illustrating an example of a process for setting a first second working area group. [Figure 37] FIG. 10 is a diagram illustrating an example of setting a first second working area group. [Figure 38] 10A, 10B, and 10C are diagrams illustrating examples of the process in step S256. [Figure 39] FIG. 10 is a diagram for explaining another example of the process to be performed in step S256. [Figure 40] 10 is a flowchart illustrating an example of a process for determining an operation mode. [Figure 41] 10 is a flowchart illustrating an example of a process for setting a first third working area group. [Figure 42] 10 is another flowchart illustrating an example of the process of setting the first third working area group. [Figure 43] FIG. 10 is a diagram for explaining a method for setting the first third working area group. [Figure 44] FIG. 10 is another diagram illustrating a method for setting the first third working area group. [Figure 45] FIG. 10 is another diagram illustrating a method for setting the first third working area group. [Figure 46] 10 is a flowchart illustrating an example of a process for calculating a second working area. [Figure 47] FIG. 10 is a diagram illustrating a method for calculating a second working area. [Figure 48] (A) and (B) are other diagrams illustrating the calculation method for the second working area. [Figure 49] FIG. 10 is another diagram illustrating the method for calculating the second working area. [Figure 50] FIG. 10 is another diagram illustrating the method for calculating the second working area. [Figure 51] 10 is a flowchart illustrating an example of a process for setting a second set of second working areas. [Figure 52] 10 is a flowchart illustrating a processing example of setting a second third working area group. [Figure 53] FIG. 10 is a diagram illustrating an example of a second third working area set setting method. [Figure 54] FIG. 10 is another diagram illustrating an example of a second third working area set setting method. [Figure 55] FIG. 10 is a diagram illustrating another example of a method for setting the second third working area group. [Figure 56] FIG. 10 is another diagram for explaining another example of a method for setting the second third working area set. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below. Note that the embodiments described below do not unduly limit the scope of the claims, and not all of the configurations described in the embodiments are necessarily essential components.

[0010] An example of the configuration of a medical manipulator system 5 of this embodiment will be described using Figure 1. The medical manipulator system 5 of this embodiment includes a control device 10 including a processor 100, a drive device 20, and an endoscope 40. The control device 10 of this embodiment controls the drive device 20. The drive device 20 of this embodiment controls the endoscope 40 and a medical manipulator 500. As will be described later, there may be multiple medical manipulators 500, and these may be specifically referred to as a first medical manipulator 510, a second medical manipulator 520, a third medical manipulator 530, etc., as necessary.

[0011] The technique of the medical manipulator system 5 of this embodiment can be applied to procedures related to, for example, ESD (Endoscopic Submucosal Dissection), as will be described later, but this does not prevent its application to other procedures, such as EMR (Endoscopic Mucosal Resection). Furthermore, part of the technique described below may be applied to other procedures. Note that ESD stands for Endoscopic Submucosal Dissection, and EMR stands for Endoscopic Mucosal Resection. In the following description, the endoscope 40 of this embodiment is exemplified as a medical flexible endoscope primarily used in ESD, but this does not prevent the technique of this embodiment from being applied to other endoscopes.

[0012] Furthermore, since each component of the endoscope 40 of this embodiment can be a widely applicable configuration that is well known as a flexible endoscope, detailed illustrations and explanations will be omitted below as appropriate. Each component of the endoscope 40 refers, for example, to the insertion section of the endoscope 40. The insertion section includes a bending section and a tip section. The tip section of the endoscope includes an imager 42, which will be described later, but may also include a cap 48, which will be described later in detail. Hereinafter, the tip section of the endoscope 40 will be abbreviated as the "endoscope tip section," the insertion section of the endoscope 40 will be abbreviated as the "endoscope insertion section," and the bending section of the endoscope 40 will be abbreviated as the "endoscope bending section."

[0013] The endoscope 40 of this embodiment includes an imager 42. The imager 42 includes an imaging sensor, such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) sensor, optical components, and functions as an imaging device. In this embodiment, an image captured by the imager 42 is referred to as an endoscopic image. The imager 42 captures returned light from a subject illuminated by an illumination device (not shown) and outputs an image signal to the processor 100 of the control device 10 via a cable (not shown). The processor 100 generates a display image based on the image signal and outputs the display image to, for example, a display 610 (described later in FIG. 8 ). The image signal includes a video signal, and the endoscopic image may be a still image of the video generated based on the video signal. As described below, there may be multiple treatment tools 50, which may be specifically referred to as a first treatment tool 51, a second treatment tool 52, a third treatment tool 53, and so on, as necessary. The illumination device (not shown) may have, for example, multiple illumination modes. For example, the lighting device may include multiple types of filters that control the lighting device and transmit light of desired wavelengths, and the control device 10 performs processing to appropriately select a filter depending on the situation, and the light that has passed through the filter is irradiated onto the subject. This allows the user to perform the procedure smoothly. Note that since many known methods have been proposed for the lighting mode, detailed explanations will be omitted.

[0014] The processor 100 of this embodiment is configured with the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured with one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, ICs (Integrated Circuits), etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0015] Furthermore, for example, the processor 100 of this embodiment can operate based on a memory (not shown) (hereinafter simply referred to as "memory") and information stored in the memory. The information may be, for example, a program and various data. The processor 100 may use a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or the like. The memory may be a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a register, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the functions of each unit of the control device 10 are realized as processes by the processor 100 executing the instructions. The instructions here may be instructions from an instruction set constituting a program, or instructions instructing the hardware circuitry of the processor 100 to operate. The memory is also referred to as a storage device.

[0016] Furthermore, in this embodiment, the processor 100 acquires three-dimensional shape information of the lumen and three-dimensional shape information of the endoscope 40. A specific method for acquiring this information will be described later.

[0017] The drive device 20 of this embodiment will be described. The drive device 20 of this embodiment can be configured, for example, as shown in Fig. 2. The drive device 20 includes a first treatment tool drive device 21 that controls each part of the first treatment tool 51, a second treatment tool drive device 22 that controls each part of the second treatment tool 52, a third treatment tool drive device 23 that controls each part of the third treatment tool 53, and an endoscope drive device 24 that controls each part of the endoscope 40. Note that Fig. 2 is a conceptual diagram, and for ease of explanation, some components, such as the imager 42, are not shown.

[0018] 2, 3, 4, 5, and 6 are also shown as appropriate in the right-handed three-dimensional coordinate axes, including the A-axis, UD-axis, and LR-axis. The direction of the A-axis is parallel to the longitudinal direction of the medical manipulator 500, with the tip of the endoscope as the reference point. The direction in which the medical manipulator 500 advances is referred to as the A1 direction, and the direction in which the medical manipulator 500 retreats is referred to as the A2 direction. In the following description, forward and backward movement may be abbreviated as "advance and retreat." In other words, the direction of the A-axis is the direction in which the medical manipulator 500 advances and retreats. The direction along the UD-axis is referred to as the UD-axis direction, and the direction along the LR-axis is referred to as the LR-axis direction. The coordinate axes shown in FIGS. 2, 3, 4, 5, and 6 are shown for the convenience of explaining the drive unit 20, and do not necessarily correspond to the coordinate axes shown in FIG. 20 and subsequent figures.

[0019] The first treatment tool 51 is, for example, a local injection needle. In this case, the first treatment tool 51 includes, for example, a first medical manipulator 510 and an injection needle 516 located at the tip of the first medical manipulator 510. The injection needle 516 is used to inject, for example, a local injection liquid filled in a syringe (not shown) into the submucosal layer. The local injection liquid is, for example, physiological saline, but may also contain sodium hyaluronate or the like in a predetermined proportion. This increases the viscosity of the local injection liquid, thereby maintaining a high elevation of the mucosa. This minimizes damage to the muscle layer in the first procedure (step S10) and the second procedure (step S20) described later in Figure 13 and subsequent steps. This reduces the possibility of complications associated with muscle layer damage. The local injection liquid may also contain a predetermined pigment substance. The predetermined pigment substance is, for example, indigo carmine. This makes the submucosal layer, after local injection, bluish and transparent, allowing the user to appropriately define the treatment area described later.

[0020] The second treatment tool 52 is, for example, a grasping forceps. In this case, the second treatment tool 52 includes, for example, a second medical manipulator 520 and a grasping portion 522 located at the tip of the second medical manipulator 520.

[0021] The third treatment tool 53 is, for example, a high-frequency knife. In this case, the third treatment tool 53 includes, for example, a third medical manipulator 530. The third treatment tool 53 is configured so that a knife portion can protrude from the tip of the third medical manipulator 530 as needed. The knife portion includes a power supply wire, a high-frequency electrode, and the like (not shown). For example, a high-frequency current is applied to the high-frequency electrode from a power supply device (not shown) via a power supply wire. In this state, by contacting the high-frequency electrode with desired biological tissue, the biological tissue is cauterized by thermal energy generated from the high-frequency electrode. By controlling the thermal energy, marking, incision, hemostasis, and the like (described below) can be performed. Note that while the shape of the knife portion shown in FIG. 2 and other figures is pole-shaped, it may also be scalpel-shaped, needle-shaped, hook-shaped, scissors-shaped, tweezers-shaped, and the like. In other words, the tip of the third treatment tool 53 of this embodiment can be configured in a wide variety of ways related to known high-frequency treatment tools.

[0022] As shown in FIG. 2 , the medical manipulator system 5 of this embodiment is electrically controlled. More specifically, the second treatment tool 52, the third treatment tool 53, and the endoscope 40 are all electrically controlled. Note that in this embodiment, the first treatment tool 51 does not necessarily need to be electrically controlled, but the first treatment tool 51 may also be electrically controlled. Furthermore, "electrically driven" refers to the medical manipulator 500 being driven by an actuator such as a motor based on an electrical signal for controlling the operation of the medical manipulator 500, etc. Furthermore, electrically driving the medical manipulator 500, etc., includes electrically driving the medical manipulator 500, etc., based on a determination by the processor 100, as well as electrically driving the medical manipulator 500, etc., by a user manually operating a console 60, which will be described later.

[0023] In this way, electrically controlling the medical manipulator system 5 realizes the techniques described below, and some of the steps relating to the first procedure (step S10) and the second procedure (step S20) described below can be performed by automatic control of the endoscope 40. Automatic control of the endoscope 40 means that the processor 100, not the user, makes the decisions to control each part included in the endoscope 40, etc., in other words, the processor 100 controls each part included in the endoscope 40, etc., using a predetermined control algorithm.

[0024] As described above, in the medical manipulator system 5 of this embodiment, the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 are inserted into the body together with the endoscope 40 to perform each treatment related to ESD, but the endoscope 40 does not need to have three treatment tool channels. For example, the medical manipulator system 5 of this embodiment may further include an overtube 46. The overtube 46 may be configured to have multiple treatment tool channels. For example, as shown in FIG. 2 , the endoscope 40 has one treatment tool insertion port 44, and the first treatment tool 51 is inserted into the treatment tool insertion port 44, and the second treatment tool 52 and the third treatment tool 53 are inserted into two treatment tool channels in the overtube 46, respectively.

[0025] As shown in FIG. 2, the medical manipulator system 5 of this embodiment may further include a cap 48. Various caps of known shapes have been proposed, and for example, as shown in FIG. 3, the cap 48 includes an endoscope exit hole indicated by B1 and a treatment instrument exit hole indicated by B2. The cap 48 in FIG. 3 is formed with one endoscope exit hole and two treatment instrument exit holes, corresponding to the example shown in FIG. 2. That is, as shown in FIG. 3, the first medical manipulator 510 protrudes from the endoscope exit hole indicated by B1, and the second medical manipulator 520 and the third medical manipulator 530 protrude from the treatment instrument exit holes indicated by B2. For convenience of explanation, the cap 48 is illustrated as if the tip of the endoscope 40 protrudes from the endoscope exit hole indicated by B1, but the cap 48 may be configured so that the tip of the endoscope 40 does not protrude. Although not shown for convenience, the cap 48 may further include tubes that form channels between each hole of the cap 48 and the drive device 20. This allows the medical manipulator 500 to operate smoothly.

[0026] 2 and 3, one treatment tool 50 protrudes from the cap 48 through the insertion portion of the endoscope 40, and two treatment tools 50 protrude from the cap 48 through the overtube 46. However, this is not limiting. For example, as shown in FIG. 4, three treatment tools 50 may protrude from the cap 48 through the overtube 46 to perform treatment according to the method of this embodiment. The cap 48 in FIG. 4 has one endoscope exit hole shown in B3 and three treatment tool exit holes shown in B4. Although not shown, the endoscope 40 may have two treatment tool channels, and two treatment tools 50 may protrude from the cap 48 through the endoscope insertion portion, and one treatment tool 50 may protrude from the cap 48 through the overtube 46. Although not shown, three treatment tools 50 may protrude from the cap 48 through the endoscope insertion portion.

[0027] 2 and 3, the cap 48 is shown as being fitted to the overtube 46 and integrated therewith. However, as shown in FIG. 5, for example, the cap 48 may be fitted to the endoscope 40, allowing the endoscope 40 to be displaced independently of the overtube 46. For example, at the start of treatment, the endoscope 40, the overtube 46, and the cap 48 may be inserted into the body in an integrated state. When the distal end of the endoscope approaches the location to be treated, the overtube 46 may be fixed with a balloon (not shown), and the endoscope 40 fitted with the cap 48 may be moved away from the overtube 46. Furthermore, although the size of the cap 48 is shown as being the same as or larger than the overtube 46 in FIG. 2 and other figures, the size of the cap 48 may be smaller than the overtube 46. This allows, for example, the endoscope 40 fitted with the cap 48 to be driven inside the overtube 46.

[0028] In this way, when the endoscope distal end includes the cap 48, the relationship between the position of the endoscope 40 and the positions of the respective medical manipulators 500 can be made to correspond to the relationship between the position of the endoscope exit hole formed in the cap 48 and the position of the respective treatment instrument exit holes in a plan view seen from the direction along the A axis. Alternatively, for example, multiple types of caps 48 with different relationships between the position of the endoscope exit hole and the position of the respective treatment instrument exit holes may be prepared, allowing the user to select an appropriate cap 48 depending on the treatment. This changes the positional relationship between the imager 42 and the respective medical manipulators 500, thereby changing the position at which each treatment instrument 50 is visible in the endoscopic image, thereby enabling the acquisition of an endoscopic image suitable for the treatment. For example, if the display described later in FIG. 12 is possible, the cap 48 shown in FIG. 5 can be selected. However, it may be assumed that the display shown in FIG. 12 is not always possible due to the location of the treatment target, etc. In such cases, the user can change the position at which the treatment instrument 50 is visible in the endoscopic image by selecting a different type of cap 48, thereby enabling the treatment to be performed more smoothly.

[0029] Furthermore, although not shown, the medical manipulator system 5 of this embodiment may be configured not to include the cap 48 but to include an overtube 46 including multiple tubes that form the endoscope insertion portion channel and multiple treatment instrument channels. In this case, in a plan view seen from the direction along the A axis, the relationship between the positions of the tubes that form the endoscope channel and the positions of the tubes that form each treatment instrument channel corresponds to the relationship between the position of the endoscope 40 and the position of each medical manipulator 500. Furthermore, multiple types of overtubes 46 that differ in the relationship between the position of the endoscope insertion portion channel and the positions of the multiple treatment instrument channels may be prepared, allowing the user to select an appropriate overtube 46 depending on the treatment. This may provide the same effect as when multiple types of caps 48 are available for selection.

[0030] 2 and the like, the drive device 20 may further include, for example, an overtube drive device that drives the overtube 46. The overtube drive device can be configured as a drive device equivalent to the endoscope drive device 24.

[0031] The endoscope 40 and the treatment tool 50 of this embodiment include drive units according to the number of degrees of freedom required. For example, the second treatment tool drive device 22 includes a motor unit 220. The motor unit 220 includes, for example, a first bending motion drive section 221, a second bending motion drive section 222, an opening / closing motion drive section 223, a rolling motion drive section 224, and an advancing / retreating motion drive section 225.

[0032] The first bending operation drive unit 221 bends the second medical manipulator 520 in a direction along the UD axis by pulling or loosening a set of wires (not shown) based on a control signal received from the control device 10. As a result, the orientation of the gripping unit 522 is changed along the direction indicated by D21. Similarly, the second bending operation drive unit 222 bends the second medical manipulator 520 in a direction along the LR axis based on a control signal received from the control device 10. As a result, the orientation of the gripping unit 522 is changed along the direction indicated by D22.

[0033] The opening / closing operation drive unit 223 controls the opening / closing operation of the gripper 522. For example, one of the grip pieces rotates around the rotation axis indicated by B21 in the direction indicated by D23 based on a control signal received from the control device 10. Note that the gripper 522 shown in Fig. 6 is just one example, and a wide variety of known structures can be applied.

[0034] The roll motion drive unit 224 controls the roll rotation motion of the tip of the second medical manipulator 520. For example, based on a control signal received from the control device 10, the roll motion drive unit 224 causes the tip of the second medical manipulator 520 to roll in the direction indicated by D24.

[0035] The advancing / retreating operation drive unit 225 controls the advancing / retreating operation of the tip of the second medical manipulator 520. The advancing / retreating operation drive unit 225 causes the second medical manipulator 520 to advance / retreat along the A-axis based on a control signal received from the control device 10 by a drive mechanism including, for example, a linear motor.

[0036] The drive units of the third treatment tool drive device 23 and the endoscope drive device 24 can also be realized by a drive unit similar to the motor unit 220 of the second treatment tool drive device 22 described above. In the third treatment tool drive device 23, for example, a configuration equivalent to the roll motion drive unit 224 described above may be omitted. In the endoscope drive device 24, for example, a driving unit 70 described later in Fig. 7 may slide in a predetermined direction relative to the floor, thereby fulfilling a function equivalent to the advance / retreat motion drive unit 225 described above.

[0037] The method of this embodiment may also be realized as a surgical system 1 including a medical manipulator system 5. As shown in FIG. 7 , the surgical system 1 includes the aforementioned control device 10 and endoscope 40, as well as a console 60 and a driving unit 70. The console 60 is wirelessly connected to the control device 10 using a communication method conforming to a wireless communication standard such as Wi-Fi (registered trademark), but may also be connected by wire, for example. In the surgical system 1 of this embodiment, an endoscope 40 is inserted into the body of a subject (not shown) lying on an operating table TA, and a procedure, etc., which will be described later with reference to FIG. 13 etc., is performed.

[0038] 2, and electrically operates each part of the endoscope 40 based on a control signal from the control device 10. Although not shown, units corresponding to the first treatment tool driving device 21, the second treatment tool driving device 22, and the third treatment tool driving device 23 may be included in the driving unit 70 or may be provided separately from the driving unit 70, and this can be determined appropriately by the user.

[0039] The console 60 includes, for example, a display 610, a touch panel 620, foot pedals 630, and a handle 640. The display 610 displays an endoscopic image captured by the imager 42 via the control device 10. The touch panel 620 displays the endoscopic image similarly to the display 610 and also includes functions such as drawing. The touch panel 620 may display a partial area, including the center of the display 610. Note that there may be multiple foot pedals 630 and handles 640. For example, as shown in FIG. 8 , the console 60 of this embodiment includes, as the foot pedals 630, a first foot pedal 631, a second foot pedal 632, and a third foot pedal 633. Note that there may be four or more foot pedals 630. Similarly, the console 60 of this embodiment includes, as the handle 640, a first handle 641 and a second handle 642. In this manner, a user can use the two handles 640 to perform treatment while appropriately selecting the endoscope 40 and multiple treatment tools 50. Although not shown, the console 60 may further include an operation unit for performing other operations in addition to the handle 640. Examples of other operations include air and water supply operations, changing the illumination mode of the light source device, and the like.

[0040] The direction indicated by D1 in FIG. 8 is the direction along which the user faces the console 60, and is also referred to as the forward direction. The direction indicated by D2 is the opposite direction to the direction indicated by D1, and is also referred to as the rearward direction. The direction indicated by D1 and the direction indicated by D2 together are also referred to as the front-to-back direction. The direction indicated by D3 is perpendicular to the front-to-back direction, and is also referred to as the leftward direction. The direction indicated by D4 is the opposite direction to the direction indicated by D3, and is also referred to as the rightward direction. The direction indicated by D3 and the direction indicated by D4 together are also referred to as the left-to-right direction. The same applies to FIG. 10, which will be described later.

[0041] In the console 60 of this embodiment, the foot pedal 630 includes a foot switch (not shown), and when a user steps on the foot pedal 630, the foot switch transmits a control signal to the control device 10. The control device 10 controls the endoscope 40 or the medical manipulator 500 based on a combination of a control signal based on the operation of the foot pedal 630 and a control signal based on the operation of a handle 640 (described later).

[0042] 9 is stored in the memory included in the control device 10. For example, when the user operates the first foot pedal 631 and the first handle 641, the control device 10 transmits a control signal from the first handle 641 to the third treatment tool driving device 23. When the user operates the first foot pedal 631 and the second handle 642, the control device 10 transmits a control signal from the second handle 642 to the second treatment tool driving device 22. When the user operates the second foot pedal 632 and the first handle 641, the control device 10 transmits a control signal from the first handle 641 to the endoscope driving device 24. When the user operates the third foot pedal 633 and the first handle 641, the control device 10 transmits a control signal from the first handle 641 to the first treatment tool driving device 21.

[0043] 9 is stored in the memory included in the control device 10, so that, for example, the user can operate the endoscope 40 by operating the first handle 641 on the right side while stepping on the second foot pedal 632 located in the center of the console 60. The user can also operate the first treatment tool 51 by operating the first handle 641 on the right side while stepping on the third foot pedal 633 located on the left side of the console 60. The user can also operate the second treatment tool 52 by operating the second handle 642 on the left side while stepping on the first foot pedal 631 located on the right side of the console 60, and can also operate the third treatment tool 53 by operating the first handle 641 on the right side.

[0044] The handle 640 may be configured in more detail as shown in FIG. 10. In FIG. 10, the handle 640 includes a first part 651, a second part 652, and a third part 653. The first part 651 and the second part 652 are connected via a joint shown by B61. The second part 652 and the third part 653 are connected via a joint shown by B62. Although FIG. 10 only shows the second handle 642, the first handle 641 is configured in the same way as the second handle 642.

[0045] The first part 651 can be displaced in the directions indicated by D62 and D65. The direction indicated by D62 is the same as the left-right direction described above in FIG. 8. The direction indicated by D65 is the same as the front-back direction described above in FIG. 8. For example, when a user holds the second part 652 and moves the second part 652 in the direction indicated by D65, the first part 651 is displaced in the front-back direction via the joint indicated by B61. Similarly, when a user holds the second part 652 and moves the second part 652 in the direction indicated by D62, the first part 651 is displaced in the left-right direction via the joint indicated by B61. The user can also rotate the second part 652 in the direction indicated by D61 around a rotation axis provided in the joint indicated by B61. The user can also roll the third part 653 in the direction indicated by D64 around the longitudinal direction of the third part 653.

[0046] When the user operates the handle 640, the medical manipulator 500 moves forward and backward, bends, rolls, etc. based on a control signal output from a sensor (not shown) or the like. For example, as described above, assume that the user operates the second handle 642 while stepping on the first foot pedal 631 to operate the second treatment tool 52. For example, when the user rotates the second part 652 in the direction shown by D61 in FIG. 10, the second medical manipulator 520 bends in the direction shown by D21 in FIG. 6. Also, for example, when the user operates the second handle 642 so that the first part 651 is displaced in the direction shown by D62 in FIG. 10, the second medical manipulator 520 bends in the direction shown by D22 in FIG. 6. Also, for example, when the user rolls the third part 653 in the direction shown by D64 in FIG. 10, the second medical manipulator 520 rolls in the direction shown by D24 in FIG. 6. 6. Also, for example, when the user operates the second handle 642 so that the first part 651 is displaced in the direction indicated by D65 in FIG. 10, the second medical manipulator 520 moves forward or backward in the direction indicated by D25 in FIG.

[0047] Although not shown, the handle 640 may further include an operation unit such as a button. For example, a high-frequency current may be applied to the high-frequency electrode of the third treatment tool 53 by operating a button included in the first handle 641 while the user is stepping on the first foot pedal 631. Furthermore, for example, a high-frequency current may be controlled to be opened or closed by operating a button included in the second handle 642 while the user is stepping on the first foot pedal 631.

[0048] Furthermore, for example, the control device 10 may be configured to enable or disable reception of a control signal transmitted from the handle 640. Enabling reception of a control signal transmitted from the handle 640 means transmitting a corresponding control signal to each component of the drive device 20 based on the control signal received from the handle 640. Disabling reception of a control signal transmitted from the handle 640 means discarding an instruction based on the control signal received from the handle 640. Specifically, for example, in addition to the table shown in FIG. 9 described above, a table shown in FIG. 11 is stored in the memory of the control device 10. For example, as described above in FIG. 6, since the second medical manipulator 520 performs any of advance / retreat, bending, and roll rotation, reception of a control signal based on any of the operations of the first part 651, the second part 652, and the third part 653 is set to be enabled, as shown in B71 in FIG. 11.

[0049] The third medical manipulator 530 may be configured to move forward and backward and bend but not roll. In this case, as shown in B72 of Fig. 11, reception of control signals based on the operation of the first part 651 and the second part 652 is enabled, but reception of control signals based on the operation of the third part 653 is disabled, as shown in B73 of Fig. 11.

[0050] Furthermore, for example, as described above, since the endoscope 40 can move forward and backward, bend, and roll, reception of a control signal based on operation of any of the first part 651, the second part 652, and the third part 653 is set to be valid, as shown in B74 of Fig. 11. Furthermore, from the table of Fig. 9, it can be seen that when the second foot pedal 632 is depressed, no drive mechanism is set to be the destination of a control signal based on operation of the second handle 642, and therefore reception of a control signal based on operation of the second handle 642 is set to be invalid, as shown in B75 of Fig. 11.

[0051] Furthermore, for example, the first treatment tool drive device 21 may be configured so that the first treatment tool 51 only moves forward and backward. Furthermore, when the third foot pedal 633 is depressed, no drive mechanism is set as the destination of a control signal based on operation of the second handle 642. In this case, as shown by B76 in Fig. 11, reception of a control signal based on operation of the first part 651 in the forward and backward directions is enabled, and reception of a control signal based on operation of the first part 651 in the left and right directions is disabled. Furthermore, as shown by B77 in Fig. 11, reception of other control signals is disabled.

[0052] It should be noted that the combination of the operation of the foot pedal 630 and the operation of the handle 640 is not limited to the above. For example, if the drive device 20 further includes the above-mentioned overtube drive device, the breakdown of the table in Fig. 9 may be changed as follows. For example, when the user operates the second foot pedal 632 and the second handle 642, the control device 10 may transmit a control signal from the second handle 642 to the first treatment tool drive device 21. Furthermore, when the user operates the third foot pedal 633 and the first handle 641, the control device 10 may transmit a control signal from the first handle 641 to the overtube drive device.

[0053] With the console 60 configured in this manner, as shown in FIG. 12, for example, the user operates the treatment tool 50 appropriately while looking at the display 610, and performs treatment on the treatment target shown in C1.

[0054] In this embodiment, the treatment target is a lesion, but the treatment target may also include a region including the lesion and a predetermined margin. The term "lesion" as used herein refers to a region that appears to be different from a normal state in appearance and is not necessarily limited to a region caused by a disease. The lesion may be, for example, a tumor, but is not limited to this and may also be a polyp, inflammation, diverticulum, etc. The predetermined margin refers to the region between the region marked (step S2) described below in FIG. 13 and the lesion region, but may also be a region other than the lesion that is elevated by the local injection (step S4) described below. The treatment target in this embodiment is an early-stage tumor. An early-stage tumor is a tumor whose growth is limited to the mucosal layer or submucosal layer.

[0055] In this embodiment, the treatment target will be explained while being illustrated as appropriate, but in order to make it easier to understand the purpose of this embodiment, unless otherwise specified, illustrations of parts of the tissue that have been changed by marking (step S2) and local injection (step S4) will be omitted.

[0056] 12 is merely an example, and it is not necessary for the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 to be simultaneously displayed on the display 610. However, when the method of this embodiment is applied to the second procedure (step S20) described later, it is desirable that the second treatment tool 52 and the third treatment tool 53 be simultaneously displayed.

[0057] The medical manipulator system 5 of this embodiment can be applied to the treatment shown in the flowchart of FIG. 13, for example. For example, a user performs marking (step S2). Marking (step S2) is a process of forming a predetermined mark around a lesion area in a lumen for performing a first procedure (step S10) or a second procedure (step S20), which will be described later. The predetermined mark is formed by a technique of minutely cauterizing the tissue surface with a third treatment tool 53, which will be described later. Alternatively, marking (step S2) may be performed by, for example, segmenting a region related to the lesion in a captured endoscopic image.

[0058] Thereafter, the user performs local injection (step S4). Specifically, for example, the user injects the local injection liquid into the submucosal layer so that the area where marking (step S2) was performed is raised. The local injection (step S4) is performed using the first treatment tool 51 described above.

[0059] Thereafter, the user performs the first procedure (step S10). The first procedure (step S10) is, for example, a circumferential incision. Specifically, for example, the submucosal layer that has been raised by the local injection (step S4) is incised using the third treatment tool 53 along the direction of the markers formed by the marking (step S2). In this way, damage to the muscle layer can be minimized.

[0060] The user then performs the second procedure (step S20). The second procedure (step S20) is, for example, ESD (endoscopic submucosal dissection). ESD stands for Endoscopic Submucosal Dissection. Although the outer periphery of the lesion is dissected from the lumen wall by the first procedure (step S10) described above, this does not necessarily mean that the inside of the lesion is dissected from the lumen wall. Therefore, for example, the user may grasp the lesion with the second treatment tool 52 and further incise the lesion with the third treatment tool 53 so that the inside of the lesion area is dissected from the lumen wall. Although not shown in the flowchart, a procedure similar to the local injection (step S4) may be performed as needed before starting the second procedure (step S20). This allows the entire lesion to be suspended from the submucosa, thereby more appropriately performing the second procedure (step S20). For example, if the size of the lesion is small, the first procedure (step S10) may be omitted and the treatment of FIG. 13 may be performed using only the second procedure (step S20).

[0061] In the following, when an explanation is given that applies to both the first procedure (step S10) and the second procedure (step S20), the first procedure (step S10) and the second procedure (step S20) may be collectively referred to simply as "procedures."

[0062] Thereafter, the user performs hemostasis and recovery (step S40). For example, the user cauterizes the bleeding site caused by the first procedure (step S10) or the second procedure (step S20) using the third treatment tool 53. The user then recovers the lesion by, for example, grasping the lesion with the gripping portion 522 of the second treatment tool 52 and retracting the second medical manipulator 520 to a position outside the body. The user may also recover the lesion by replacing any one of the first treatment tool 51, the second treatment tool 52, and the third treatment tool 53 with a dedicated treatment tool for recovering the lesion, such as a net treatment tool. If the size of the lesion is larger than the diameter of the channel of the second medical manipulator 520, the user may also retract the endoscope 40 and the medical manipulator 500 to a position outside the body while grasping the lesion with the gripping portion 522.

[0063] In the following description, circumferential incision is exemplified as the first procedure (step S10), and submucosal dissection is exemplified as the second procedure (step S20), but the procedures to which the method of this embodiment can be applied are not limited to these. For example, all or part of the methods described below may be applied to marking (step S2), local injection (step S4), etc., or may be applied to other procedures.

[0064] An example of processing according to the method of this embodiment will be described using the flowchart of FIG. 14. The processing shown in FIG. 14 can be applied to both the first procedure (step S10) and the second procedure (step S20) of FIG. 13. However, the detailed processing content of step S200 when the processing of FIG. 14 is applied to the first procedure (step S10) is partially different from the detailed processing content of step S200 when the processing of FIG. 14 is applied to the second procedure (step S20). Therefore, hereinafter, after a general description of the case where the processing of FIG. 14 is applied to the first procedure (step S10), an example of applying the processing of FIG. 14 to the second procedure (step S20) will be described. Furthermore, when the processing of FIG. 14 is applied to the second procedure (step S20), differences from the description of the case where the processing of FIG. 14 is applied to the first procedure (step S10) will be mainly described.

[0065] The processor 100 acquires three-dimensional shape information of the treatment target (step S100). A more detailed processing example of step S100 will be described later. Next, the processor 100 sets the positions and order of the working areas (step S200). As will be described later in detail, step S200 sets N working areas and sets the order of the N working areas.

[0066] The working area in this embodiment refers to an area related to the range within which the medical manipulator 500 can work. The working area refers to a predetermined range of the endoscopic image that satisfies both the range within which the treatment tool 50 associated with the medical manipulator 500 is visible and the range of motion of the medical manipulator 500. The predetermined range is a given range suitable for arranging the treatment tool 50 associated with the medical manipulator 500. In other words, it can also be said to be a range within which it is considered convenient for the user to be able to visually recognize the treatment tool 50 when performing a procedure. Specific details of the predetermined range will be described later with reference to FIG. 28 and other figures. Hereinafter, the working area when the method of this embodiment is applied to the first procedure (step S10) will be more specifically referred to as the "first working area." In other words, the "first working area" refers to the range in which the third medical manipulator 530 can move when the first procedure (step S10) is performed, and the range in which the user can visually recognize the third treatment tool 53 in the endoscopic image to the extent that the user can perform the first procedure (step S10). Similarly, the working area when the method of this embodiment is applied to the second procedure (step S20) is more specifically referred to as the "second working area," and details will be described later with reference to FIG. 46. Note that when both the "first working area" and the "second working area" apply, they are collectively referred to as the "working area."

[0067] The working area in this embodiment is position information related to the movable range of the medical manipulator 500 and is set in association with the position information of the endoscope 40. That is, theoretically, the number of working areas can be set equal to the number of position information that the endoscope 40 can assume. As will be described in detail later, in this embodiment, only a number of working areas appropriate for performing a procedure are selected, and an order is assigned to the selected working areas. Hereinafter, the working area selected by the user to perform the first procedure will be referred to as the first working area. More generally, the working area selected to perform the Nth procedure will be referred to as the Nth working area. Also, hereinafter, the Nth working area when the method of this embodiment is applied to the first procedure (step S10) will be specifically referred to as the “first Nth working area,” and the Nth working area when the method of this embodiment is applied to the second procedure (step S20) will be specifically referred to as the “second Nth working area.” Note that when both the “first Nth working area” and the “second Nth working area” apply, they will be collectively simply referred to as the “Nth working area.”

[0068] The processor 100 then performs a process (step S310) to determine whether or not a working area required for treatment is present. If the processor 100 determines that a working area required for treatment is present (YES in step S310), it performs a process (step S320) to determine whether or not an Nth working area has been selected. If the processor 100 determines that the Nth working area has been selected (YES in step S320), it controls the position of the endoscope 40 to the Nth position corresponding to the Nth working area (step S330). On the other hand, if the processor 100 determines that the Nth working area has not been selected (NO in step S320), it performs step S320 again. In other words, the processor 100 automatically controls the endoscope driving device 24 to move the endoscope 40 to positions corresponding to each of the ordered working areas, the number of times corresponding to each of the ordered working areas.

[0069] As described above, the position information of the working area is associated with the position of the endoscope 40. More specifically, the position information of the working area and the position information of the endoscope 40 are associated and stored in memory. Hereinafter, a position related to the position information of the endoscope 40 associated with the position information of the first working area will be referred to as the "first position." Similarly, a position related to the position information of the endoscope 40 associated with the position information of the Nth working area will be referred to as the Nth position. Hereinafter, the Nth position when the technique of this embodiment is applied to the first procedure (step S10) will be specifically referred to as the "first Nth position," and the Nth position when the technique of this embodiment is applied to the second procedure (step S20) will be specifically referred to as the "second Nth position." Note that when a position corresponds to both the "first Nth position" and the "second Nth position," it will be collectively referred to simply as the "Nth position."

[0070] For example, after the first working area to the Nth working area are set in step S200, which will be described later, the user designates the first working area and performs the procedure by operating the operation unit of the console 60, etc. At this time, the processor 100 controls the drive device 20 based on the stored position information of the first position, and automatically moves the endoscope 40 to the first position. In other words, after the user designates the first working area and moves the endoscope 40 to the first position, in order to handle the third treatment tool 53, the user only needs to operate the first handle 641 while stepping on the first foot pedal 631, and there is no need to operate the other foot pedals 630 and handle 640.

[0071] After the user has completed the procedure at the first position, the processor 100 moves the endoscope 40 from the first position to the second position by, for example, operating an operation unit (not shown). In this case, too, the user does not need to operate the first handle 641 while stepping on the second foot pedal 632.

[0072] The flowchart of Fig. 14 is applied to each of the first procedure (step S10) and the second procedure (step S20) of Fig. 13. Therefore, when the procedure of Fig. 13 is performed, the processing of the flowchart of Fig. 14 is executed twice. Therefore, when performing step S330 in the first procedure (step S10), the processor 100 controls the drive of the endoscope 40 to a first first position corresponding to the first first working area, and then when performing step S330 in the second procedure (step S10), the processor 100 controls the drive of the endoscope 40 to a second first position corresponding to the second first working area.

[0073] FIG. 15 is a flowchart showing a more detailed example of the processing of step S100 in FIG. 14. The processor 100 acquires an endoscopic image (step S110). For example, the imager 42 captures an image of the inner wall of a lumen containing a treatment target. The processor 100 then acquires information about the endoscopic image from the imager 42. As will be described in detail later, step S100 includes processing for recognizing the treatment target area from the captured endoscopic image. Recognizing the treatment target area means, in other words, recognizing the boundary between the treatment target area and an area other than the treatment target area. Hereinafter, the boundary line relating to the boundary of the treatment target area recognized by the processor 100 in step S110 will be referred to as the "treatment target boundary line" or simply as the "boundary line." The treatment target boundary line serves as an incision line that is the target of incision by the third treatment tool 53.

[0074] In the following description, the imager 42 will be described as an imager that captures a two-dimensional image, but the imager 42 may be a three-dimensional camera such as a stereo camera. In this case, some of the processing described below can be omitted, although detailed description will be omitted.

[0075] The processor 100 then acquires three-dimensional shape information of the lumen (step S120). The three-dimensional shape information of the lumen includes three-dimensional shape information related to the region to be treated. Step S120 can be achieved using a first sensor 410 or the like (not shown in FIG. 1), and details will be described later.

[0076] The processor 100 then acquires three-dimensional information of the endoscope 40 (step S130). The three-dimensional shape information of the endoscope 40 refers to three-dimensional shape information of the portion of the endoscope 40 that is inserted into the body, and more specifically, refers to three-dimensional shape information of the insertion section and the tip of the endoscope 40. Step S130 can be achieved by using a second sensor 420 or the like that is not shown in FIG. 1. This allows the processor 100 to acquire three-dimensional data that specifies the position and posture of the tip of the endoscope.

[0077] Thereafter, the processor 100 calculates three-dimensional shape information of the treatment target in the reference coordinate system (step S140). For example, the processor 100 converts the two-dimensional coordinates of the treatment target in the endoscopic image into three-dimensional shape information using the three-dimensional information of the lumen acquired in step S120 and parameters of the imager 42; a detailed method will be described later. Note that, although a detailed description of the reference coordinate system will be given later, the values ​​of the position information related to the data acquired in steps S110, S120, and S130 are all based on different measurement standards, and therefore, step S140 includes a process of unifying the measurement standards and converting the values ​​into values ​​based on the unified standard. Then, based on the converted three-dimensional shape information of the treatment target, three-dimensional position information is set that identifies the location to be incised by the user using the third treatment tool 53 in the first procedure (step S10), for example.

[0078] Fig. 16 is a flowchart showing a more detailed processing example of step S200 in Fig. 14. The processor 100 acquires basic information of the medical manipulator system 5 (step S210). Thereafter, the processor 100 calculates the movable range of the tip of the endoscope (step S220) and calculates the working area (step S230). Thereafter, the processor 100 sets a first working area group (step S240). For example, the processor performs a process of assigning the working area calculated in step S230 to each of the movable ranges calculated in step S220.

[0079] In this embodiment, a set of multiple working areas is referred to as a working area group. The set of multiple working areas assigned to each movable range of the endoscope tip in step S240 is referred to as a "first working area group." A set of multiple working areas that may be suitable for performing a procedure is referred to as a "second working area group." A set of multiple working areas for performing a procedure is referred to as a "third working area group." Since the number of working areas included in the first working area group may be enormous at the stage of performing step S240, the number of working areas required for the procedure is determined step by step. As will be described in detail later, for example, by performing the processing of step S250, processor 100 selects multiple working areas from the first working area group that may be suitable for performing a procedure and sets the second working area group. Processor 100 also selects multiple working areas from the second working area group that will actually be used for performing a procedure and sets the third working area group. In other words, the first to Nth working areas are included in the third working area group.

[0080] Thereafter, the operation mode is determined (step S260). The methods of steps S210, S220, S230, S240, S250, and S260 will be described in more detail below.

[0081] As described above, the medical manipulator system 5 of this embodiment includes an endoscope 40 including an imager 42 that captures endoscopic images, a medical manipulator 500 that protrudes beyond the distal end surface of the endoscope 40, a drive unit 20 that controls the endoscope 40 and the medical manipulator 500, and a processor 100. The processor 100 acquires an endoscopic image of the treatment target from the imager 42, recognizes the area of ​​the treatment target from the endoscopic image, acquires three-dimensional shape information of the lumen, and acquires three-dimensional shape information of the endoscope 40. The processor 100 also acquires information about the working area, which is the range in which the medical manipulator 500 can work, for each of multiple positions of the endoscope 40, thereby acquiring information about the multiple working areas. The processor 100 also drives and controls the endoscope 40 to a first position corresponding to a first working area selected from the multiple working areas, based on the area of ​​the treatment target, the three-dimensional shape information of the lumen, and the three-dimensional shape information of the endoscope 40.

[0082] As described above, the medical manipulator system 5 of this embodiment includes the imager 42, the medical manipulator 500, the drive unit 20, and the processor 100, and therefore can be applied to a medical endoscope 40. Furthermore, the processor 100 calculates information related to the working area, which is the range within which the medical manipulator 500 can work, and therefore the user can grasp whether the treatment tool 50 associated with the medical manipulator 500 will reach the treatment target, etc. Furthermore, the processor 100 controls the drive of the endoscope 40 to the first position corresponding to the selected first working area, thereby eliminating the burden on the user in operating the endoscope 40.

[0083] Among procedures using an endoscope 40, circumferential incision, ESD, and the like involve making an incision around the periphery of a lesion, which requires multiple changes to the position of the endoscope tip, placing a significant burden on the user. The method disclosed in International Publication No. 2017 / 048194 does not take into consideration driving the endoscope 40 to the optimal position each time the procedure progresses. In this regard, by applying the method of the present embodiment, the endoscope 40 is automatically driven and controlled to an Nth position corresponding to a preselected Nth working area, thereby shortening the time required to adjust the position of the endoscope 40 and reducing the burden on the user during the procedure. This allows the user to concentrate on making an incision or the like on the treatment target during the procedure.

[0084] The method of the present embodiment may be realized by a processor 100. That is, the processor 100 of the present embodiment controls an endoscope 40 including an imager 42 that captures endoscopic images, a medical manipulator 500 that protrudes beyond the distal end surface of the endoscope 40, and a driving device 20 that controls the endoscope 40 and the medical manipulator 500. The processor 100 also acquires an endoscopic image of the treatment target from the imager 42, recognizes the area of ​​the treatment target from the endoscopic image, acquires three-dimensional shape information of the lumen, and acquires three-dimensional shape information of the endoscope 40. The processor 100 also acquires information about a plurality of working areas by calculating information about the working area, which is the range in which the medical manipulator 500 can work, for each of a plurality of positions in the movable range of the endoscope 40. The processor 100 also drives and controls the endoscope 40 to a first position corresponding to a first working area selected from the plurality of working areas based on the area of ​​the treatment target, the three-dimensional shape information of the lumen, and the three-dimensional shape information of the endoscope 40. By doing so, the same effect as above can be obtained.

[0085] The technique of this embodiment may also be realized by a control method. That is, the control method of this embodiment controls an endoscope 40 including an imager 42 that captures endoscopic images, a medical manipulator 500 that protrudes beyond the distal end of the endoscope 40, and a driving device 20 that controls the endoscope 40 and the medical manipulator 500. The control method of this embodiment acquires an endoscopic image of the treatment target from the imager 42, recognizes the area of ​​the treatment target from the endoscopic image, acquires three-dimensional shape information of the lumen, and acquires three-dimensional shape information of the endoscope 40. The control method of this embodiment acquires information about multiple working areas by calculating information about the working area, which is the range in which the medical manipulator 500 can work, for each of multiple positions in the movable range of the endoscope 40. The control method of this embodiment also drives and controls the endoscope 40 to a first position corresponding to a first working area selected from the multiple working areas based on the area of ​​the treatment target, the three-dimensional shape information of the lumen, and the three-dimensional shape information of the endoscope 40. This achieves the same effects as described above.

[0086] The processor 100 may also acquire information about a working area, which is a range that is within a predetermined range of the endoscopic image and is virtually set based on the range in which the treatment tool 50 associated with the medical manipulator 500 is shown and the movable range of the medical manipulator 500. In this way, the endoscope 40 can be controlled to be driven to a position that corresponds to the working area that is set based on the range in which the treatment tool 50 is shown in the endoscopic image and the movable range of the medical manipulator 500.

[0087] The above-described technique may also be realized as a control method. That is, the control method of this embodiment may acquire information about a working area, which is a range within a predetermined range of an endoscopic image and which is virtually set based on the range in which the treatment tool 50 associated with the medical manipulator 500 is captured and the movable range of the medical manipulator 500. By doing so, the same effect as above can be obtained.

[0088] Furthermore, the processor 100 may select a first working area and a second working area from the plurality of working areas, move the endoscope 40 to a first position corresponding to the selected first working area, and then move the endoscope 40 to a second position corresponding to the selected second working area. In this way, the burden of the procedure on the user can be further reduced.

[0089] Furthermore, the above-described technique may be realized as a control method. That is, the control method of this embodiment may select a first working area and a second working area from a plurality of working areas, move the endoscope 40 to a first position corresponding to the selected first working area, and then move the endoscope 40 to a second position corresponding to the selected second working area. By doing so, the same effect as above can be obtained.

[0090] Furthermore, the processor 100 may drive and control the position of the endoscope 40 to a first first position corresponding to a first first working area selected from a plurality of first working areas during a first procedure (step S10). Furthermore, the processor 100 may drive and control the position of the endoscope 40 to a second first position corresponding to a second first working area selected from a plurality of second working areas during a second procedure (step S20). In this way, a medical manipulator system 5 can be constructed in which, in a treatment in which a plurality of different procedures are performed in succession, different working areas are set according to each procedure, and the endoscope 40 is driven and controlled to a position corresponding to the set working area.

[0091] The above-described technique may also be realized as a control method. That is, the control method of this embodiment may drive and control the position of endoscope 40 to a first first position corresponding to a first first working area selected from a plurality of first working areas in a first procedure (step S10). The control method of this embodiment may drive and control the position of endoscope 40 to a second first position corresponding to a second first working area selected from a plurality of second working areas in a second procedure (step S20). In this way, the same effect as above can be obtained.

[0092] The following describes in more detail the techniques of steps S110, S120, S130, and S140 in Fig. 15. For example, as shown in Fig. 17, the endoscope 40 can realize the technique related to the processing example in Fig. 15 by including a first sensor 410 and a second sensor 420 in addition to the imager 42 described above. For example, the processor 100 can realize step S110 by acquiring an endoscopic image from the imager 42, can realize the processing related to step S120 by acquiring three-dimensional shape information of the lumen from the first sensor 410, and can realize the processing related to step S130 by acquiring three-dimensional shape information of the endoscope 40 from the second sensor 420.

[0093] In step S110, for example, the processor 100 performs a process of segmenting a treatment target region based on the acquired endoscopic image. Although not shown, step S110 may be performed using a machine-learned trained model. The trained model has been trained to segment and output a region suitable for performing the first procedure (step S10) or the second procedure (step S20) in the endoscopic image when an endoscopic image of a lesion area after local injection (step S4) is input. The trained model for step S110 may be a convolutional neural network (CNN) used in the image recognition field. Alternatively, a user may set a treatment target region using a function of the touch panel 620 shown in FIG. 8 . For example, the user can determine the boundary of the treatment target by operating the touch panel 620 to form an incision for the first procedure (step S10) while viewing the endoscopic image displayed on the touch panel 620. Furthermore, the treatment target segmented by the processor 100 may be displayed on the touch panel 620, and the user may be able to use the touch panel 620 to appropriately modify the area of ​​the displayed treatment target.

[0094] In step S110, the processor 100 performs a process of storing the position information of the region to be treated in memory. A specific example will be described later with reference to Figs. 21(B) and 22.

[0095] The method of step S120 will now be described in more detail. More specifically, the first sensor 410 is a distance sensor that detects distance images using, for example, a TOF (Time Of Flight) method. Distance sensors measure distance by measuring the time of flight of light. The first sensor 410 detects the distance from the tip of the endoscope to the inner wall of the lumen for each pixel. Position information of each point on the inner wall of the body, i.e., three-dimensional shape information of the lumen, can be calculated from the distance for each pixel detected by the first sensor 410 and position information and posture information of the tip of the endoscope.

[0096] The first sensor 410 is, for example, a LIDAR (Light Detection and Ranging) type. The first sensor 410 in the LiDAR system may be a distance sensor that detects distance by detecting a distance from a laser source. The first sensor 410 in the LiDAR system includes, for example, a laser light source, an optical sensor, etc. The laser light source irradiates the inner wall of the lumen with laser light, and the optical sensor detects the laser light that hits the inner wall of the lumen and bounces back, thereby measuring the time it takes for the laser light to hit the inner wall of the lumen and bounce back. This makes it possible to measure the distance and direction to the inner wall of the lumen. The processor 100 can calculate three-dimensional information of the lumen, for example, by acquiring information on the distance to the inner wall of the lumen as point cloud data.

[0097] The above-mentioned TOF method and LIDAR method are well known techniques and therefore will not be illustrated. As described above, the medical manipulator system 5 of this embodiment includes a first sensor 410, and the processor 100 acquires three-dimensional shape information of the lumen from the first sensor 410. In this way, the first sensor 410 acquires three-dimensional shape information of the lumen. A medical manipulator system 5 can be constructed.

[0098] Furthermore, the first sensor 410 may be a distance sensor using the TOF method or the LIDAR method. In this way, a medical manipulator system 5 can be constructed that acquires three-dimensional shape information of a lumen using a distance sensor using the TOF method or the LIDAR method.

[0099] The method for calculating the 3D information of the lumen is not limited to the above. For example, the processor 100 may acquire the 3D shape information of the lumen from the position information of pixels in 3D space calculated by triangulation from the position information of pixels of feature points included in two images captured simultaneously using a stereo camera. The processor 100 may also calculate the position of each feature point using photometric stereo images. In this case, multiple illumination windows are provided at the tip of the endoscope. The multiple illumination lights emitted from the multiple illumination windows can be selectively switched and emitted by controlling the driving of multiple light-emitting diodes for illumination provided in a light source device (not shown). The state of the shadow part in the image of the lumen wall changes depending on the switching of the illumination light. Therefore, the distance to the shadow part of the lumen wall can be calculated based on the amount of change. In other words, the 3D shape information of the lumen can be acquired based on photometric stereo from the image of the shadow area in the captured image obtained by illuminating with multiple selectively operated illumination units.

[0100] The method of step S130 will be described in more detail. For example, the second sensor 420 is a curved shape observation sensor that uses a magnetic field for sensing. More specifically, the second sensor 420 includes, for example, multiple source coils 422, a current generating device (not shown), a first insertion shape calculation device (not shown), and an antenna (not shown). For example, as shown in FIG. 18A, multiple source coils 422 are provided at predetermined intervals in the insertion section of the endoscope 40. The current generating device (not shown) outputs a sinusoidal current to each source coil 422 in order, starting with the source coil 422 closest to the distal end of the endoscope. Each source coil 422 generates a magnetic field using the current. The first insertion shape calculation device detects the magnetic field emitted from each source coil 422 via an antenna (not shown) and acquires position information for each source coil 422 based on the strength of the detected magnetic field. Furthermore, the first insertion shape calculation device generates insertion shape information of the insertion portion of the endoscope 40 based on the acquired position information of each of the multiple source coils 422, and transmits the generated insertion shape information to the processor 100. Note that, for example, the processor 100 may function as the second insertion shape calculation device, and various modifications are possible.

[0101] Furthermore, for example, second sensor 420 may be a curved shape observation sensor that uses a sensing method that utilizes distortion. As will be described later, second sensor 420 includes, for example, optical fiber 424 and a second insertion shape calculation device. Although detailed illustration is omitted, the second insertion shape calculation device includes, for example, a predetermined light source device, a reflected light receiver, a spectroscope, a spectrum analyzer, a curvature calculation device, and a curved shape calculation device.

[0102] The second sensor 420 includes a plurality of optical fibers 424. For example, as conceptually shown in E1 of Fig. 18(B), the endoscope insertion portion includes optical fibers 424A, 424B, 424C, and 424D. Note that, hereinafter, these may be collectively referred to simply as optical fibers 424.

[0103] As shown in FIG. 18C, the optical fiber 424 is configured such that the core portion indicated by E5 is inserted into the cladding portion indicated by E6. In the optical fiber 424 included in the second sensor 420, the core portion is connected to an FBG (Fiber Bragg Grating) 426 (hereinafter referred to as FBG 426). ) The FBG 426 includes a Bragg grating portion 428 whose refractive index changes periodically. As a result, when light of a certain wavelength is incident, reflected diffracted light is obtained by the Bragg grating portion 428. The specific wavelength is also called the Bragg wavelength. When the endoscope insertion portion is straight, the value of the specific wavelength and the value of the wavelength of the reflected diffracted light are equal. On the other hand, for example, when the endoscope insertion portion is bent, distortion occurs in the Bragg grating portion 428, and the value of the wavelength of the reflected diffracted light becomes larger or smaller than the value of the specific wavelength depending on the magnitude of the distortion. Therefore, by configuring the FBG 426 in this embodiment so that Bragg grating portions 428 are included at predetermined intervals and each Bragg grating portion 428 has a different refractive index, information about the direction and degree of distortion at the position corresponding to each Bragg grating portion 428 can be obtained.

[0104] For example, when a predetermined incident light from a light source device shown in E2 in FIG. 18(B) is incident on an optical fiber 424, reflected diffracted light is captured by the optical fiber 424 into a reflected light receiver shown in E3. The predetermined incident light includes light based on a specific wavelength corresponding to each Bragg grating 428. A predetermined spectral distribution can be obtained by detecting the predetermined reflected diffracted light captured by the reflected light receiver with a spectrometer. The spectral analysis device detects the peak of the predetermined reflected diffracted light based on the obtained predetermined spectral distribution and determines the wavelength of the detected peak. Curvature data is calculated based on the wavelength value of the detected peak and the value of the specific wavelength. The curved shape calculation device calculates curved shape data of the endoscope insertion section based on the acquired curvature data and transmits it to the processor 100. The processor 100 may function as a part of the second insertion shape calculation device.

[0105] Furthermore, it is desirable to arrange the optical fiber 424A and the optical fiber 424B so that they face each other along the direction of the dashed line indicated by E4 in Fig. 18(B). This makes it possible to obtain curved shape data for a case where, for example, when the endoscope insertion portion is bent, the optical fiber 424A is located on either the inner or outer periphery of the curved portion, and the optical fiber 424B is located on the other. This makes it possible to obtain three-dimensional shape information of the endoscope insertion portion with greater accuracy. Similarly, it is desirable to arrange the optical fiber 424C and the optical fiber 424D so that they face each other in a direction perpendicular to the dashed line indicated by E4 in Fig. 18(B).

[0106] From the above, the medical manipulator system 5 of this embodiment includes the second sensor 420, and the processor 100 acquires the three-dimensional shape information of the endoscope 40 from the second sensor 420. In this way, the three-dimensional shape information of the endoscope 40 is acquired by the second sensor 420. A medical manipulator system 5 for obtaining the above can be constructed.

[0107] The second sensor 420 may be a curved shape observation sensor using a sensing method that uses a magnetic field or a curved shape observation sensor using a sensing method that uses distortion. In this way, a medical manipulator system 5 can be constructed that acquires three-dimensional shape information of the endoscope 40 using a curved shape observation sensor using a sensing method that uses a magnetic field or distortion.

[0108] The second sensor 420 of this embodiment may be a curvature sensor that observes the shape by a sensing method using, for example, ultrasonic waves, or may be a curvature sensor that observes the shape by a sensing method using an X-ray absorbing material.

[0109] Next, the calculation method for step S140 will be conceptually explained using Figures 19, 20, 21, and 22. Position information is information whose measured value differs depending on the measurement standard. Specifically, for example, in the above-mentioned step S110, imager 42 included in the tip of the endoscope acquires position information of the treatment target as coordinate information of the endoscopic image. Furthermore, in step S120, first sensor 410 acquires three-dimensional shape information of the lumen. As a result, three-dimensional shape information of the treatment target present on the inner wall of the lumen is acquired. In this case, the value of the position information of the treatment target acquired in step S110 and the value of the position information of the treatment target acquired in step S120 will differ for the reasons described above.

[0110] Therefore, in this embodiment, the coordinate system that serves as the reference for the measurement values ​​used in the calculation of step S140 will be referred to as the "reference coordinate system." Hereinafter, the three-dimensional coordinate axes of the left-handed system based on the base end of endoscope 40, i.e., the three-dimensional coordinate axes of the left-handed system in the reference coordinate system, will be referred to as the X0-axis, Y0-axis, and Z0-axis, and the three-dimensional coordinate axes of the left-handed system based on the tip of the endoscope will be referred to as the X1-axis, Y1-axis, and Z1-axis. Furthermore, the three-dimensional coordinate axes of the left-handed system based on the first sensor 410 will be referred to as the X2-axis, Y2-axis, and Z2-axis. In this case, as shown in FIG. 19 , the position coordinates of the treatment target when the base end of endoscope 40 is used as the reference, i.e., the position coordinates of the treatment target in the reference coordinate system, can be expressed as (x0, y0, z0). Similarly, the position coordinates of the treatment target when the tip of the endoscope is used as the reference can be expressed as (x1, y1, z1), and the position coordinates of the treatment target region when the first sensor 410 is used as the reference can be expressed as (x2, y2, z2).

[0111] For example, the transformation matrix M(12) is used between the position information (x1, y1, z1) and the position information (x2, y2, z2).

number

number

number

[0112] For example, if the inner wall of the lumen is the inner wall of the large intestine, the relationship between the X0-axis, Y0-axis, and Z0-axis and the X1-axis, Y1-axis, and Z1-axis is as shown in Figure 20, and the orientations of the X1-axis, Y1-axis, and Z1-axis constantly change depending on the progress of the endoscope insertion portion, the treatment status, etc. In this embodiment, three-dimensional shape information of the endoscope 40 is periodically acquired in step S130, so that the relationship between the X0-axis, Y0-axis, and Z0-axis and the X1-axis, Y1-axis, and Z1-axis can be grasped, and position information of the treatment target can be acquired as values ​​in the reference coordinate system. In other words, the transformation matrix M(01) of equation (2) can be obtained in step S130.

[0113] Next, a method for indicating the position of a treatment target imaged in an endoscopic image in a reference coordinate system will be described. For example, as conceptually shown in Fig. 21(A), three-dimensional shape information of a lumen is acquired by a first sensor 410, and position information of the treatment target shown in E10 is also acquired. Since this information is acquired based on the first sensor 410, for example, the position coordinates of a position shown in E11, which is included in the region of the treatment target shown in E10, can be expressed as (x2, y2, z2).

[0114] Meanwhile, the treatment target is also imaged by an imager 42 included near the tip of the endoscope. In this case, position information of the treatment target can be displayed relative to the tip of the endoscope. Therefore, for example, as shown in FIG. 21(B), if a coordinate system consisting of X1, Y1, and Z1 axes is set, the position information of the position shown in E21 can be expressed as (x1, y1, z1). The position shown in E21 in FIG. 21(B) is the same as the position shown in E11 in FIG. 21(A), but because the measurement standards are different, the values ​​of (x2, y2, z2) in FIG. 21(A) are different from the values ​​of (x1, y1, z1) in FIG. 21(B).

[0115] Furthermore, when an image of the treatment target is captured by the imager 42, the image is formed at a position that is a distance t away in the +X1 direction from the lens position shown in E20 of FIG. 21(B). Note that t is This is basic information of the imager 42, which will be described later, and is a known value. In other words, it can be assumed that the plane to be treated is located at a distance t in the +X1 direction from the lens position. Therefore, if the treatment target is captured at the position shown in E23 in the virtual image shown in E22, the treatment target shown in E21 will be located on an extension of the line connecting the lens position shown in E20 and the position shown in E23 (the line of the angle of view of the imager 42). In other words, when the lens position shown in E20 is used as a reference, the position information of the treatment target shown in E21 can be found by scaling the position information related to the position shown in E23.

[0116] Furthermore, when conceptually shown in a perspective view, the screen shown at E22 in FIG. 21B becomes the screen shown at E32 in FIG. 22. Here, the position shown at E30 in FIG. 22 corresponds to the position of the lens shown at E20 in FIG. 21B, and the position shown at E31 in FIG. 22 corresponds to the position shown at E23 in FIG. 21B. Furthermore, the X3 axis and Y3 axis in FIG. 22 are coordinate axes of the screen shown at E32. That is, the position information of the treatment target stored in the above-mentioned step S110 is composed of a value based on the X3 axis and a value based on the Y3 axis in FIG. 22. For example, the position information related to the position shown at E31 in FIG. 22 can be expressed as position information (x3, y3) on the endoscopic image.

[0117] In addition, in FIG. 22, the position information of the lens shown in E30 is expressed by coordinates (xα, yα, zα) based on a coordinate system consisting of the X1 axis, the Y1 axis, and the Z1 axis.

[0118] From the above, there is the following relationship between (x1, y1, z1) in Figure 21(B) and (x3, y3) in Figure 22:

number

[0119] Also, if the right-hand side of equation (1) is equal to the right-hand side of equation (4),

number

[0120] In this way, the position information acquired by the first sensor 410 and the position coordinates of the tip of the endoscope can be converted into values ​​based on the reference coordinate system. Then, in step S140, the processor 100 converts the values ​​of the position information that identify the treatment target into values ​​based on the reference coordinate system, and stores the set of converted values ​​in memory.

[0121] In this embodiment, the values ​​of the treatment target and the information acquired in steps S110, S120, and S130 are all converted into the reference coordinate system. That is, the calculation processes performed by the processor 100 in Figs. 23 and 27, which will be described later, are performed based on values ​​converted into values ​​measured from the reference coordinate system.

[0122] Step S210 will be described in more detail. The basic information of the medical manipulator system 5 includes, for example, basic information of the endoscope 40, basic information of the imager 42, and basic information of the medical manipulator 500. The basic information of the endoscope 40 includes information such as the degrees of freedom of the endoscope 40, the length, inner diameter, and outer diameter of the endoscope insertion portion. The basic information of the imager 42 includes, for example, information such as the position and angle of view of the imager 42. The basic information of the medical manipulator 500 includes, for example, information such as the degrees of freedom, dimensions, and operating range of the medical manipulator 500.

[0123] Step S210 may be performed, for example, by the user operating an operating unit not shown, or, for example, when an endoscope 40 or the like is connected to the driving device 20, information about the connected endoscope 40 or the like may be automatically acquired, and can be realized in various ways.

[0124] Step S220 will be described in more detail using Figures 23, 24, 25, and 26. More specifically, step S220 is performed as shown in the processing example of the flowchart in Figure 23. Processor 100 acquires the movable range of the tip of the endoscope (step S222). The movable range of the tip of the endoscope refers to the entire range of positions and postures that the tip of the endoscope can take. Processor 100 then excludes areas that do not overlap with the lumen from the range acquired in step S222 (step S224).

[0125] Step S222 will be conceptually explained using Figures 24 and 25. Note that, for the X0 axis, Y0 axis, and Z0 axis appropriately shown in Figure 24 and subsequent figures, a reference coordinate system is set so that the plane formed by the boundary of the treatment target is parallel to the X0-Y0 plane, in order to facilitate understanding of the gist of this embodiment. Also, for simplicity of explanation, the direction along the inner wall of the lumen is assumed to be parallel to the X0 direction. Also, as in Figures 12, 21(A), and 21(B), the posture of the endoscope 40 is assumed to be adjusted so that the treatment target is displayed at the bottom of the endoscopic screen. Furthermore, unless otherwise specified, the endoscope 40 is assumed to advance in a direction parallel to the +X0 direction.

[0126] The endoscope 40 shown in the figures used in the following description is a representative of the endoscope 40 described above in FIG. 5. That is, the following description will use an endoscope 40 in which the overtube 46 and the cap 48 are integrated and the endoscope 40 can be displaced independently of the overtube 46. The endoscope 40 used in the following description is configured so that three treatment tools 50 (a first treatment tool 51, a second treatment tool 52, and a third treatment tool 53) can protrude through the cap 48 or the endoscope tip. That is, when the overtube 46 is fixed with a balloon or the like (not shown), the endoscope tip protrudes and can move forward or bend based on the tip of the overtube 46, and when the endoscope tip is fixed, the treatment tools 50 protrude and can move forward or bend based on the endoscope tip. In the figures used in the following description, of the components included in the endoscope tip, the cap 48 is shown as a representative of the endoscope tip, and other components are omitted from the illustration as appropriate.

[0127] For example, in the case of an endoscope 40 in which the cap 48 does not move independently of the overtube 46, the technique of this embodiment can be applied by replacing control of the position of the endoscope 40 with control of the position of the overtube 46. This is because it is important to control the position of the part including the medical manipulator 500 so that the treatment tool 50 can appropriately approach the treatment target. Therefore, although an example of controlling the position of the endoscope 40 is exemplified below, the technique of this embodiment can be expanded and considered as controlling the position of the medical manipulator system 5.

[0128] For example, assume that the overtube 46 and the cap 48 are integrated, as shown at E40 in Fig. 24. In this case, the user fixes the overtube 46 with a balloon (not shown) or the like, and while stepping on the second foot pedal 632, operates the first part 651 of the first handle 641 along the direction shown at D65 in Fig. 10 to control the endoscope driving device 24 and perform an advancement operation to advance the endoscope 40. As a result, for example, as shown at E41 in Fig. 24, the position of the endoscope tip can be moved further away in the +X0 direction from the tip position of the overtube 46.

[0129] 24, the user can control the endoscope driving device 24 by operating the first part 651 of the first handle 641 in the direction shown in D62 of Fig. 10 while stepping on the second foot pedal 632, thereby performing a bending operation that bends the endoscope bending portion in a direction along the Y0 direction. In other words, the user can perform an operation that combines the above-mentioned forward movement and bending operation on the endoscope bending portion, and therefore can move the position of the endoscope tip away from the position of the tip of the overtube 46, as shown in E42 or E43 of Fig. 24.

[0130] In other words, when considering forward movement in the +X0 direction and bending movement along the Y0 direction, the range that the endoscope tip can reach from the position of the tip of the overtube 46 is conceptually shown as the range E51 in Fig. 25(A). Furthermore, the user can control the endoscope driving device 24 and perform a bending movement that bends the endoscope bending section in the Z0 direction by operating the second part 652 of the first handle 641 in the direction shown in D61 in Fig. 10 while pressing the second foot pedal 632. Thus, when the bending movement of the endoscope bending section in the Z0 direction is also taken into consideration, the movable range of the endoscope tip can be roughly illustrated as the range shown in E52 in Fig. 25(B).

[0131] The range shown in E52 in FIG. 25(B) is not data obtained by measurement, but data obtained by the endoscope 40 This can be obtained from the structural data of each part that constitutes the structure. The structural data is included in the data acquired in step S210.

[0132] Step S224 will be conceptually explained. For example, suppose the range indicated by E61 in FIG. 26 is the range calculated in step S222, and the region between the dotted lines indicated by E62 and E63 in FIG. 26 is the region of the lumen. It is not conceivable that the tip of the endoscope will be driven into a region other than the lumen. Therefore, in step S224, the processor 100 excludes a region that does not overlap with the lumen from the range calculated in step S222. Specifically, for example, the processor 100 performs processing to obtain a set of positional information in which the three-dimensional shape information of the lumen acquired in step S120 matches the positional information of the range calculated in step S222. This determines the range indicated by E64 in FIG. 26 and a region other than the range indicated by E64. Then, in step S224, the processor 100 excludes a region other than the range indicated by E64 from the range indicated by E61 in FIG. 26. As a result, the range indicated by E64 in FIG. 26 is calculated as the movable range of the tip of the endoscope.

[0133] Step S230 will be described in more detail with reference to Figures 27, 28, 29, and 30. More specifically, step S230 is performed as in the processing example shown in the flowchart of Figure 27. The processor 100 sets a first spatial region based on the operating range of the third treatment tool 53 (step S231-A). Thereafter, the processor 100 calculates a first working region based on the first spatial region (step S231-B).

[0134] Step S231-A will be conceptually explained. As described above, the third medical manipulator 530 that drives the third treatment tool 53 is configured to be able to advance and retreat and bend in the same way as the endoscope 40. Therefore, the processor 100 calculates the first spatial region, which is the range in which the third treatment tool 53 can mechanically operate, based on the data acquired in step S210. Since the mechanism for advancing and retreating and bending of the third medical manipulator 530 is the same as that for the endoscope 40, the outline of the first spatial region is the range shown by the solid line at E72 in FIG. 28(A), similar to the range shown at E52 in FIG. 25. Note that the imager 42 captures an image. The outline of the possible range is shown by the dotted line in E71 in Fig. 28(A). Note that, as shown in Fig. 28(A), the range shown in E71 and the range shown in E72 are conceptual and do not specifically specify the size of the range.

[0135] Step S231-B will be conceptually explained. As described above, the processor 100 calculates, as the first working area, a range that is a predetermined range of the endoscopic image and satisfies both the range in which the treatment tool 50 of the medical manipulator 500 is visible and the first spatial area. For example, a user performs the first procedure (step S10) while looking at the endoscopic image, but simply being able to visually recognize the third treatment tool 53 at an arbitrary position within the endoscopic image does not necessarily mean that the first procedure (step S10) can be performed appropriately, and it is desirable that the third treatment tool 53 be visible within a predetermined range of the endoscopic image.

[0136] Therefore, in step S231-B, the processor 100 sets a predetermined range, which is a given range suitable for positioning the treatment tool 50 of the medical manipulator 500, based on the endoscopic image. The predetermined range can be determined appropriately depending on how the treatment target and the inner wall are imaged in the endoscopic image. As described above, assuming that the plane formed by the boundary of the treatment target is parallel to the X0-Y0 plane, the range suitable for treatment may be, for example, a rectangular area in the lower two-thirds of the endoscopic image, or a rectangular area in the lower half of the endoscopic image. The ratio may be determined arbitrarily by the user. Furthermore, the predetermined range does not have to be a rectangular area and may be, for example, an arc-shaped area. Alternatively, for example, if a rectangular endoscopic image is considered to be a clock face, the predetermined range may be an area between a first time and a second time, centered on the center of gravity of the frame of the endoscopic image. For example, in FIG. 28(B), the dotted rectangular area shown by E81 is the endoscopic image, and the area shown by the solid line in E82 is an example in which the area between the 4 o'clock direction and the 8 o'clock direction when the center of gravity of the frame of the endoscopic image shown by E83 is set as the predetermined range. The 4 o'clock direction refers to the 4 o'clock direction when the endoscopic image is considered to be the face of a 12-hour clock. Similarly, the 8 o'clock direction refers to the 8 o'clock direction when the endoscopic image viewed from the distal end of the endoscope is considered to be the face of a 12-hour clock. Note that FIG. 28(B) is an example, and the first time is not limited to 4 o'clock, nor is the second time limited to 8 o'clock. When the range shown by the solid line in E82 in FIG. 28(B) is set as the predetermined range, the processor 100 determines the overlapping range of the range shown by the solid line in E72 in FIG. 28(A) and the range shown by the solid line in E82 in FIG. 28(B) as the first range. The processor 100 calculates the working area of ​​the third medical manipulator 530 to be equal to the working area of ​​the first medical manipulator 530. In this way, in the medical manipulator system 5 of this embodiment, the processor 100 acquires information about the working area, which is a virtually set range, by superimposing a range in the endoscopic image that is within a given range suitable for the placement of the treatment tool 50 associated with the medical manipulator 500 and in which the treatment tool 50 associated with the medical manipulator 500 is shown on the medical manipulator 500, on the movable range of the medical manipulator 500. In this way, the endoscope 40 can be driven and controlled to a position that corresponds to the working area that is set based on the range in which the treatment tool 50 is shown on the endoscopic image and the movable range of the third medical manipulator 530 so that the user can easily perform the procedure.

[0137] The first spatial region calculated in step S231-A is reduced in step S231-B. For example, when the first spatial region is viewed in the X0-Y0 plane, it is assumed that the first spatial region calculated in step S231-A is represented by the region shown in E91 in Fig. 29. Then, in step S231-B, the region shown in E91 is reduced to the region shown in E92, and the region shown in E92 corresponds to the first working region.

[0138] In the following description, for example, the direction indicated by E93 may be conveniently referred to as the "direction in which the first working area faces." The direction indicated by E93 is the same as the direction in which the third medical manipulator 530 moves forward and backward as indicated by E94. In other words, the direction in which the first working area faces is the same as the direction in which the tip of the endoscope faces when the position of the endoscope 40 is controlled based on the first working area.

[0139] Furthermore, the first spatial area shown at E91 in FIG. 29 and the first working area shown at E92 are illustrated with shapes such that the direction in which the third medical manipulator 530 advances and retreats is the longitudinal direction, but this is merely an example, and the specific size of the first working area is determined by the relationship between the components included in the endoscope 40.

[0140] Furthermore, when step S231-B is performed, the processor 100 may store in memory the position information that identifies the first working area and the position information of the tip of the endoscope when step S231-B is performed, in association with each other. Hereinafter, for convenience of explanation, a dot indicating the position of the tip of the endoscope may be added to the cap 48 as shown at E95 in Fig. 29, but the user may decide as appropriate where to specify the position of the tip of the endoscope.

[0141] Using Figure 30, the processing of step S231-B when the first spatial region is viewed in the X0-Z0 plane will be conceptually described. For example, as shown in Figure 30(A), assume that the lens of the imager 42 is located at the position indicated by E101, the medical manipulator 500 protrudes from the position indicated by E102, and the predetermined range described above in Figure 28(B) is set to, for example, a rectangular region in the lower half of the endoscopic image. In this case, the region indicated by E103 is calculated as the first spatial region. Furthermore, the region indicated by E104 does not overlap with the predetermined range, and is therefore excluded from the first spatial region.

[0142] Also, for example, as shown in Fig. 30(B), suppose that the lens of the imager 42 is located at the position indicated by E111, the medical manipulator 500 protrudes from the position indicated by E112, and the predetermined range described above in Fig. 28(B) is set to, for example, a rectangular region in the lower half of the endoscopic image. In this case, the region indicated by E113 is calculated as the first spatial region. Furthermore, the region indicated by E114 does not overlap with the predetermined range, and is therefore excluded from the first spatial region.

[0143] As described above, the degree of reduction of the first spatial region in the Z0 direction varies in step S231-B depending on the relative positional relationship between the position of the lens of the imager 42 and the protruding position of the medical manipulator 500. For example, if the position of the treatment target and the structure of the lumen in which the treatment target is located are known from pre-treatment examination results, the user can grasp in advance information such as how the treatment target will appear in the endoscopic image and the extent to which reduction of the first spatial region in the Z0 direction is acceptable. This allows the user to determine the relationship between the position of the endoscope 40 and the positions of each medical manipulator 500 before inserting the endoscope 40 into the body based on the grasped information. For example, as shown in FIG. 30(A), for a treatment tool 50 for which a certain degree of reduction of the first spatial region in the Z0 direction is acceptable, the position of the endoscope 40 and the protruding position of the medical manipulator 500 may be relatively close to each other, and therefore, the medical manipulator 500 may be passed through the treatment tool insertion port 44. On the other hand, as shown in Figure 30(B), for a treatment tool 50 that requires minimal reduction in the first spatial region in the Z0 direction, it is desirable to make the position of the imager 42 and the protruding position of the medical manipulator 500 relatively far apart, so the medical manipulator 500 can be passed through a tube that forms a treatment tool channel outside the endoscope 40.

[0144] Furthermore, as described above, the user may prepare multiple types of caps 48 having different relationships between the position of the endoscope exit hole and the position of each treatment tool exit hole, and select an appropriate cap 48 from the multiple types of caps 48. For example, if the user wants to make the position of the imager 42 and the protruding positions of each medical manipulator 500 relatively far apart, the user may select a cap 48 in which the distance between the position of the endoscope exit hole and the position of each treatment tool exit hole is relatively far apart.

[0145] Similarly, when a medical manipulator system 5 that does not include the cap 48 is employed, multiple types of overtubes 46 that differ in the relationship between the position of the endoscope insertion portion channel and the positions of the multiple treatment instrument channels may be prepared, and the user may select an appropriate overtube 46 from the multiple types of overtubes 46. For example, if the user wants to make the position of the imager 42 and the protruding positions of the respective medical manipulators 500 relatively far apart, the user may select an overtube 46 in which the distance between the position of the endoscope insertion portion channel and the positions of the respective treatment instrument channels is relatively far apart.

[0146] Furthermore, for example, after the endoscope 40 is inserted into the body, the relationship between the position of the lens of the imager 42 and the protruding position of the medical manipulator 500 can be adjusted by driving the endoscope 40 to roll.

[0147] Various circumstances are taken into consideration when determining how to adjust the relationship between the position of the lens of the imager 42 and the protruding position of the medical manipulator 500. Examples of various circumstances include the user's experience, the positional relationship between the protruding position of the medical manipulator 500 and the treatment target, and the following circumstances.

[0148] For example, as shown in Fig. 30(A), if there is an inner wall shown in E105, the treatment tool 50 can reach the range shown in E106. Similarly, as shown in Fig. 30(B), if there is an inner wall shown in E115, the treatment tool 50 can reach the range shown in E116. Since the range shown in E116 is wider than the range shown in E106, if the direction in which the tip of the endoscope faces and the direction along the inner wall are not parallel, it may be convenient to adjust the positional relationship between the position of the lens of the imager 42 and the position from which the medical manipulator 500 protrudes.

[0149] 30, the case where the direction in which the endoscope tip faces and the direction along the inner wall are not parallel is described, but in this embodiment, when performing the first procedure (step S10), it is desirable that the direction in which the endoscope tip faces is parallel to the direction along the inner wall. However, it is not necessarily required that the direction in which the end of the overtube 46 faces and the direction along the inner wall are parallel.

[0150] 31(A), for example, suppose that the overtube 46 is positioned so that the angle between the direction along the inner wall of the lumen indicated by F0 and the direction in which the tip of the overtube 46 faces is angle RE1, and the overtube 46 cannot advance any further due to circumstances such as the structure of the lumen. In this case, if an endoscope 40 is used in which the cap 48 cannot be displaced independently of the overtube 46, for example, the third treatment tool 53 will only reach the location related to the line indicated by F2, and the overtube 46 will have to be advanced and retracted again, which is not convenient.

[0151] In this regard, the endoscope 40 of this embodiment can drive and control the endoscope insertion section so that the cap 48 can be displaced independently of the overtube 46. Therefore, as shown in F3 of FIG. As shown in Fig. 1, the endoscope bending portion can be bent so that the direction in which the tip of the endoscope faces follows the inner wall. In this case, the straight line indicated by F4 and the straight line indicated by F5 will have different directions. The straight line indicated by F4 is a straight line in the same direction as the tip of the overtube 46, and the straight line indicated by F5 is a straight line in the same direction as the tip of the endoscope.

[0152] The explanation will continue assuming that the direction of the endoscope tip is parallel to the inner wall, in other words, the X0 axis. For example, in the case of a medical manipulator 500 configured with a curved tip and a retractable base, the angle formed by the tip of the treatment tool 50 located at the tip of the medical manipulator 500 on an axis parallel to the axis of the medical manipulator 500 is the same regardless of the degree of retraction, but varies depending on the degree of curvature. Hereinafter, the tip of the treatment tool 50 will be simply referred to as the "treatment tool tip." Specifically, as conceptually shown in FIG. 32, for example, assume that the third medical manipulator 530 is retracted in a direction parallel to the X0 direction and is bent in the -Z0 direction at the position indicated by F01 or F02. In this case, the magnitudes of angles RE11 and RE12 in FIG. 32 are the same because the degree of retraction is different but the degree of curvature is the same. Similarly, the magnitudes of angles RE13 and RE14 in FIG. 32 are also the same. On the other hand, the magnitude of angle RE11 and the magnitude of angle RE13 in Fig. 32 are different because the degree of advancement and retreat is the same but the degree of curvature is different. The magnitude of angle RE12 and the magnitude of angle RE14 in Fig. 32 are also different.

[0153] Then, for example, as described above in FIG. 31(B), suppose that the orientation of the endoscope tip becomes parallel to the direction along the inner wall, and the third medical manipulator 530 is advanced in a direction parallel to the X0 direction. Furthermore, suppose that the third medical manipulator 530 is bent at the position shown in F03 or the position shown in F04, as shown in FIG. 33(A). In this case, as described above in FIG. 32, the magnitude of the angle RE15 in FIG. 33(A) is the same as the magnitude of the angle RE16. Therefore, if the magnitude of the angle RE15 is within an appropriate range for performing the first procedure (step S10), it is appropriate for the third treatment tool 53 to reach the entire treatment target shown in FIG. 33(A).

[0154] On the other hand, for example, as shown in Figure 33(B), suppose the third medical manipulator 530 is advanced in a state in which the direction of the endoscope tip is not parallel to the direction along the inner wall and the direction in which the third medical manipulator 530 is advanced is not parallel to the X0 direction. Furthermore, suppose the third medical manipulator 530 is bent at the position shown in F05 or the position shown in F06. In this case, the magnitude of angle RE17 in Figure 33(B) is greater than the magnitude of angle RE18. Therefore, for example, even if the magnitude of angle RE18 is within a range appropriate for performing the first procedure (step S10), the magnitude of angle RE17 may not be within a range appropriate for performing the first procedure (step S10).

[0155] In this way, when the endoscope 40 of this embodiment is used, the direction of the endoscope tip is parallel to the direction along the inner wall, thereby widening the range of treatment targets to which the treatment tool 50 can reach within an appropriate angle range. This reduces the number of times the position of the endoscope 40 needs to be moved when performing the first procedure (step S10). Note that although the third medical manipulator 530 and the third treatment tool 53 are exemplified in the description in Figures 32 and 33, the same applies to other medical manipulators 500 and other treatment tools 50.

[0156] Furthermore, it is not essential that the direction of the tip of the endoscope be parallel to the direction along the inner wall. In the situation shown in Figure 33(B) described above, if it is found that the magnitudes of the angles RE17 and RE18 are both within appropriate ranges for performing the first procedure (step S10), the direction of the tip of the endoscope does not have to be parallel to the direction along the inner wall.

[0157] In addition, there may be cases where the direction of the endoscope tip cannot be made parallel to the direction along the inner wall. For example, as shown in FIG. 34(A), when the direction of the tip of the overtube 46 is not parallel to the direction along the inner wall, Assume that the endoscope 40 is pointing upward relative to the direction of the distal end of the overtube 46. In this case, the line indicated by F6 forms an angle RE2 with respect to the direction along the inner wall. The line indicated by F6 is a line in the same direction as the direction in which the distal end of the overtube 46 faces. In this situation, for example, by bending the endoscope bending section downward by an angle RE3, as shown by F7 in FIG. 34(B), it is possible to only make an angle RE4 between the direction in which the distal end of the endoscope faces and the direction along the inner wall as shown by F9. Note that angle RE3 is the angle between the direction in which the distal end of the overtube 46 faces as shown by F8 and the direction in which the distal end of the endoscope faces as shown by F9. Furthermore, the relationship RE4 = RE3 - RE2 exists among the angles RE2, RE3, and RE4. Thus, the situation shown in FIG. 34(B) is not optimal for performing the first procedure (step S10). However, if the distal end of the treatment tool 50 can be directed in a manner suitable for performing the first procedure (step S10), the technique of this embodiment may be applicable, as will be described in detail later with reference to FIG. 39, etc.

[0158] Next, step S240 described above with reference to FIG. 16 will be conceptually explained using FIG. 35. The area indicated by F11 in FIG. 35 corresponds to the movable range of the tip of the endoscope indicated by E64 in FIG. 26. The area indicated by F12 in FIG. 35 corresponds to the first working area indicated by E92 in FIG. 29. Then, in step S240, the processor 100 assigns first working areas to the area indicated by F13 at predetermined intervals. Like the area indicated by F11, the area indicated by F13 corresponds to the movable range of the tip of the endoscope. This sets a first group of first working areas. Note that for ease of explanation, FIG. 35 illustrates multiple working areas so that they do not overlap with each other; however, by shortening the predetermined interval, the first working areas will overlap with each other. The predetermined interval can be determined appropriately by the user, taking into account the number of first working areas assigned to the movable range.

[0159] The orientation of each first working area varies depending on the position of the movable range of the tip of the endoscope. For example, the orientation of the first working area shown in F14, the orientation of the first working area shown in F15, and the orientation of the first working area shown in F16 are all different.

[0160] 36 is a flowchart illustrating step S250 in more detail. Processor 100 performs a process of determining whether or not there is an unselected work area (step S251). If there is an unselected work area (YES in step S251), processor 100 selects one work area (step S252). Then, processor 100 determines whether or not the selected first work area intersects with the processing target by a certain range or more (step S254-A). If processor 100 determines that the selected first work area does not intersect with the processing target by a certain range or more (NO in step S254-A), it excludes the selected first work area (step S258) and performs step S251 again.

[0161] On the other hand, if the processor 100 determines that the selected first working area intersects with the treatment target by a certain range or more (YES in step S254-A), it determines whether or not the direction of the treatment tool 50 is within the target range (step S256). If the processor 100 determines that the direction of the treatment tool 50 is within the target range (YES in step S256), it leaves the selected working area (step S257) and performs step S251 again. On the other hand, if the processor 100 determines that the direction in which the treatment tool tip is facing is not within the target range (NO in step S256), it excludes the selected working area (step S258) and performs step S251 again. If all working areas have been selected, the processor 100 determines NO in step S251 and ends the flow.

[0162] Step S254-A will be described in detail. For example, if the area indicated by F20 in FIG. 37 is the area to be processed, the first working areas indicated by F21, F22, F23, F24, and F25 are considered not to intersect with the area to be processed by a certain range or more. More specifically, for example, the processor 100 compares the position information of the area to be processed stored in step S140 with the position information of the first working area stored in step S231-B, and if the sets of overlapping position information are within a certain range, it determines NO in step S254. As a result, the first working areas indicated by F21, F22, F23, F24, and F25 are excluded in step S258. Also, for example, the processor 100 determines YES in step S254-A for the first working areas indicated by F26, F27, and F28 in FIG. 37. Note that Figure 37 is an example when viewed in the X0-Y0 plane, and if the Z0 direction is also taken into consideration, even if the answer is YES in step S254-A, it may be judged as NO in step S256 described below and may be excluded in the subsequent step S258.

[0163] Step S256 will be described in more detail. Whether the distal end of the treatment tool is within the target range is determined taking into consideration various circumstances. Specifically, for example, as shown in FIG. 38(A), when the direction in which the first working area indicated by F31 faces can be considered to be substantially parallel to the X0 axis, the magnitude of angle RE51 and the magnitude of angle RE52 are the same, as in the case described above with reference to FIG. 33(A). Then, if the magnitude of angle RE51 (= the magnitude of angle RE52) is within a range suitable for performing the first procedure (step S10), it is suitable to perform the first procedure (step S10) on the entire treatment target indicated by F32, and therefore processor 100 determines YES in step S256 for the first working area indicated by F31.

[0164] Furthermore, for example, as shown in Figure 38(B), when the direction in which the first working area indicated by F33 faces is different from the direction of the X0 axis, the magnitude of angle RE53 and the magnitude of angle RE54 will be different, as in the case described above with reference to Figure 33(B). Then, for example, processor 100 determines that the magnitude of angle RE54 is within a range suitable for performing the first procedure (step S10), but the magnitude of angle RE53 is not within a range suitable for performing the first procedure (step S10). In this case, it cannot be said that it is suitable to perform the first procedure (step S10) on the entire treatment target indicated by F34, and therefore processor 100 determines NO in step S256 for the first working area indicated by F33.

[0165] Note that whether step S256 returns YES or NO is not necessarily uniquely determined by the orientation of the first working area and the direction of the X0 axis. For example, as shown in FIG. 38(C), the orientation of the first working area indicated by F35 is the same as the orientation of the first working area indicated by F33, so the magnitudes of angles RE55 and RE56 are different. However, there may be cases where the magnitudes of angles RE55 and RE56 are both within a range appropriate for performing the first procedure (step S10), for example, because the size of the treatment target indicated by F36 is small. In this case, processor 100 determines YES in step S256 for the first working area indicated by F33.

[0166] 38(A) to 38(C) are examples in which it is determined whether the direction in which the tip of the third treatment tool 53 faces is suitable for the entire X0 direction component of the treatment target, but the processing of step S256 is not limited to this. For example, if the direction in which the tip of the treatment tool faces is suitable for a continuous part of the entire treatment target, it may be determined YES in step S256.

[0167] 39, when the direction of the first working area shown in F37 is significantly different from the direction of the X0 axis, it is not realistic to control the direction of the tip of the third treatment tool 53 to be within an appropriate range for the entire X0 direction component of the treatment target shown in F38. Therefore, for example, when the direction of the tip of the third treatment tool 53 is within a range appropriate for performing the first procedure (step S10) within the continuous range shown in F39, the processor 100 may determine YES in step S256 for the first working area shown in F37. In this case, when performing the first procedure using the first working area shown in F37, for example, the user should be careful to incise only the line related to the area shown in F39.

[0168] The examples in FIGS. 38 and 39 are examples in which step S256 is applied to the first procedure (step S10), but the same idea can also be applied to the case in which step S256 is applied to the second procedure (step S20).

[0169] In this way, by performing step S250 shown in detail in Fig. 36, a second working area group is set based on a first working area group set by mechanically assigning it to the range of motion of the tip of the endoscope. Hereinafter, when the method of this embodiment is applied to the first procedure (step S10), the second working area group set by step S250 will be specifically referred to as a "first second working area group." Note that, as will be described later, when the method of this embodiment is applied to the second procedure (step S20), the second working area group extracted by step S250 will be specifically referred to as a "second second working area group."

[0170] An example of the processing of step S260 will be described in more detail with reference to FIG. 40. The user determines whether or not he or she desires to select the working area (step S260-1). If the user does not desire to select the working area (NO in step S260-1), the user decides to operate the medical manipulator system 5 in the first mode (step S261). Thereafter, the processor 100 sets a third working area group (step S265). Details of step S265 will be described later. Note that, when step S261 is performed, steps S310, S320, and S330 described above with reference to FIG. 14 are determined by the processor 100. In other words, the first mode can also be called an automatic mode, and the processing from step S261 onwards, which will be described later, is performed automatically.

[0171] On the other hand, if the user desires to select the working area by the user (YES in step S260-1), the user decides to operate the medical manipulator system 5 in the second mode (step S262). For example, the processor 100 displays the second working area group set in step S250 on the touch panel 620. Then, the user performs an operation to select the first working area, etc., using the touch panel function. After completing the selection of the first working area, the user operates an operation unit (not shown), etc., to perform the processing from step S310 onwards. In other words, the second mode can also be called a manual mode.

[0172] In this embodiment, after the second work area group is set in step S250, the first mode or the second mode is selected in step S260, but this is not limited to this. For example, steps S250 and S265 may be performed after the first mode is selected. In this case, if the user selects the second mode, they individually select work areas that make up the third work area group from the first work area group.

[0173] In the following, the third working area group when the method of this embodiment is applied to the first procedure (step S10) will be referred to as the "first third working area group." Similarly, the third working area group when the method of this embodiment is applied to the second procedure (step S10) will be referred to as the "second third working area group."

[0174] Step S265 will be described in more detail using FIGS. 41, 42, 43, 44, and 45. FIGS. 41 and 42 are flowcharts illustrating a more detailed processing example of step S265. Step S265 involves setting a first third working area group by extracting only the working areas necessary for the procedure from the first second working area group set in step S250, and setting the order of the first working areas from the first third working area group. Hereinafter, the working area selected in performing step S265 will be referred to as the "selected working area." More specifically, the selected working area when applying the method of this embodiment to the first procedure (step S10) will be referred to as the "first selected working area." Similarly, the selected working area when applying the method of this embodiment to the second procedure (step S20) will be referred to as the "second selected working area."

[0175] Processor 100 calculates the length of the superimposed line for each first selection work area (step S265-1A). The length of the superimposed line refers to the length of the line where the boundary line of the processing target and the first selection work area overlap, and a specific example will be described later with reference to FIG. 44. Processor 100 also stores data in memory that associates each first selection work area with the calculated length of the superimposed line. Note that the length of the superimposed line may also be defined as the length of the line where the second selection work area (described later) and the boundary line of the processing target overlap, and details will be described later.

[0176] Then, processor 100 sets a first selected working area (step S265-2A). For example, processor 100 selects the longest superimposed line among the superimposed lines calculated in step S265-1A, and selects the first selected working area associated with the selected superimposed line as the first selected working area. Note that the first selected working area does not necessarily coincide with the first selected working area.

[0177] Then, processor 100 sets a first (K+1)th selected working area that includes the first end of the first Kth selected working area (step S265-3A). For example, if K=1, processor 100 sets a first second selected working area that includes the first end of the first first selected working area set in step S265-2A. More specifically, processor 100 sets the first selected working area that includes the first end of the first selected working area and has the longest superimposed line as the second selected working area. The first end refers to one end of the superimposed line. As will be described later, the other end of the superimposed line is referred to as the second end. Furthermore, step S265-2A may include a process in which processor 100 stores position coordinate data relating to the first end and position coordinate data relating to the second end in memory.

[0178] Processor 100 then determines whether the superimposed line associated with the first K-th selected working area includes the second end of the first selected working area (step S265-4A). If processor 100 determines that the superimposed line associated with the K-th selected working area does not include the second end of the first selected working area (NO in step S265-4A), it increments K by 1 and performs step S265-3 again. If processor 100 determines that the superimposed line associated with the K-th selected working area includes the second end of the first selected working area (YES in step S265-4A), it performs processing from step S265-6A onward, which will be described later. The range of K associated with steps S265-3A and S265-5A is 1≦K≦N. That is, steps S265-3A, S265-4A, and S265-5A result in N first selected working areas being selected.

[0179] If processor 100 determines that the superimposed line related to the Kth selected working area includes the second end of the first selected working area (YES in step S265-4A), it sets the first first working area (step S265-6A). For example, processor 100 performs a process to calculate the distance between the position coordinates of the tip of the endoscope related to the first selected working area and the position coordinates of the tip of the endoscope where it is actually located. Then, processor 100 sets the first selected working area related to the shortest distance among the calculated distances as the first first working area.

[0180] Then, the processor 100 sets the first Nth working area (step S265-7A). More specifically, the processor 100 sets the first second working area, ..., the first Nth working area. This sets the order of the first working areas.

[0181] For example, assume that the treatment target is located at the position indicated by F40 in FIG. 43 and the distal end of the endoscope is located at the position indicated by F41. The area indicated by F42 is the first working area calculated in step S230. Although not shown, assume that the first second working area group is set by performing the above-described steps S240 and S250, the user selects the first mode (NO in step S260-1, step S261), and step S265 is performed. Then, in step S265-1A, a first first selected working area is set as conceptually shown by F50 in FIG. 44. More specifically, the overlapping line indicated by F52 is determined to be the longest overlapping line, and the first working area indicated by F51 is set as the first first selected working area. The processor 100 may display a display similar to that shown by F50 on the display 610 or the touch panel 620. The same applies to the displays F60 and F70.

[0182] Thereafter, in step S265-3A, a first second selected working area is set, for example, as shown in F60 in Figure 44. Specifically, for example, in the first first selected working area shown in F61, a first end portion shown in G1 and a second end portion shown in G2 are set. Then, the first working area shown in F62 is set as the first second selected working area because it includes the first end portion shown in G1 and is the working area with the longest superimposed line. Note that the second selected working area shown in F62 does not include the second end portion shown in G2, so a NO determination is made in step S265-4A.

[0183] As described above, in the medical manipulator system 5 of this embodiment, the processor 100 sets the selected working area that overlaps the boundary line of the treatment object the longest as the first selected working area. The processor 100 also sets the selected working area that includes the first end, which is one end of the boundary line included in the first selected working area, and that overlaps the boundary line the longest as the second selected working area. This minimizes the number of selected working areas required to set the third group of working areas. This minimizes the number of times the endoscope 40 is moved during the procedure. This allows the user to perform the procedure efficiently.

[0184] Then, steps S265-3A, S265-4A, and S265-5A are repeatedly performed, thereby performing, for example, the processing shown in F70 of FIG. 44. As described above, the first selected working area shown in F71 includes the first end portion shown in G11 and the second end portion shown in G12, and the first selected working area shown in F72 includes the first end portion shown in G11. Then, a first selected working area shown in F73, a first selected working area shown in F74, a first selected working area shown in F75, and a first selected working area shown in F76 are further set. Furthermore, since the first selected working area shown in F76 includes the second end portion shown in G12, processor 100 determines YES in step S265-4A.

[0185] 44, steps S265-1A to S265-5A further select the first first selected work area through the first sixth selected work area from the first second work area group set in step S250, thereby setting the first third work area group. That is, K in steps S265-3A and S265-5A is 6.

[0186] For the above reasons, in the medical manipulator system 5 of this embodiment, when the processor 100 selects the Nth selected working area (first sixth selected working area) that includes the second end, which is the other end of the boundary line included in the first selected working area, the processor 100 sets the multiple selected working areas that include the first to Nth selected working areas as a third working area group. In this way, a condition for completing the selection of selected working areas necessary to configure the third working area group can be established.

[0187] In the first procedure (step S10), as shown in F70 of FIG. 44, the overlapping area of ​​the first selected working area to the first sixth selected working area does not need to include the entire area of ​​the treatment target. This is because in the first procedure (step S10), it is sufficient if the user can perform incision with the third treatment tool 53 along the boundary line of the treatment target. In other words, the first selected working area to the first sixth selected working area need to be arranged along the boundary line of the treatment target. That is, in the medical manipulator system 5 of this embodiment, the processor 100 selects, from the working areas included in the second working area group, a working area to be used for the procedure as a selected working area, and arranges the multiple selected working areas along the boundary line of the treatment target, thereby setting the third working area group. In this way, the third working area group can be appropriately set based on the second working area group.

[0188] Then, in step S265-6A, for example, processing as shown in FIG. 45 is performed. Specifically, processor 100 sets the first selected working area indicated by F81, which is closest to the position of the tip of the endoscope, as the first first working area among the first to first sixth selected working areas. More specifically, for example, the position coordinates of the tip of the endoscope corresponding to the selected working area indicated by F81 are associated as the position indicated by F91 in step S240. Although not shown, the same is true for the selected working areas indicated by F82, F83, F84, F85, and F86. Then, since the position indicated by F91 is closest to the position of the tip of the endoscope indicated by F90, processor 100 sets the first selected working area indicated by F81 as the first first working area, and sets the position indicated by F91 as the first first position.

[0189] Thereafter, in step S265-7A, the first second working area to the first Nth working area are set. The user may determine the rules for setting the first second working area as appropriate. For example, when setting the first second working area to the first Nth working area in counterclockwise order from the first first working area as viewed from the +Z0 direction, the processor 100 sets the first selected working area shown in F82 of FIG. 45 as the first second working area and also sets a first second position (not shown). Similarly, the processor 100 sets the first selected working area shown in F83 as the first third working area and also sets a first third position (not shown). Similarly, the processor 100 sets the first selected working area shown in F84 as the first fourth working area and also sets a first fourth position (not shown). Similarly, the processor 100 sets the first selected working area shown in F85 as the first fifth working area and also sets a first fifth position (not shown). Similarly, the processor 100 sets the first selected working area indicated by F86 as the first sixth working area, and also sets the first sixth position (not shown).

[0190] As described above, in the medical manipulator system 5 of this embodiment, the processor 100 calculates the overlap length between the boundary line of the treatment object and the working area, and sets a predetermined group of working areas (first third group of working areas) including the first working area based on the longest working area among the multiple working areas. By doing so, it is possible to reduce the number of first working areas constituting the predetermined group of working areas (first third group of working areas). This makes it possible to reduce the number of times the tip of the endoscope is moved in the first procedure (step S10). This makes it possible to improve the work efficiency of the first procedure (step S10).

[0191] The technique of this embodiment may be realized as a control method. That is, the control method of this embodiment calculates the overlap length between the boundary line of the processing target and the working area, and sets a predetermined group of working areas (first third group of working areas) including the first working area based on the working area with the longest length among the multiple working areas.

[0192] Furthermore, when setting first second working areas in clockwise order from the first first working area as viewed from the +Z0 direction, processor 100 sets the first selected working area shown in F86 of Fig. 45 as the first second working area. Similarly, processor 100 sets the first selected working area shown in F85 as the first third working area, the first selected working area shown in F84 as the first fourth working area, the first selected working area shown in F83 as the first fifth working area, and the first selected working area shown in F82 as the first sixth working area. Although not shown, the first first position, first second position, first third position, first fourth position, first fifth position, and first sixth position are also set in the same manner.

[0193] Alternatively, the order of the selected working areas may be set in descending order, for example, from the closest working area to the current position of the tip of the endoscope. In this case, for example, the processor 100 sets the first selected working area indicated by F82 as the first second working area. Similarly, the processor 100 sets the first selected working area indicated by F86 as the first third working area, the first selected working area indicated by F85 as the first fourth working area, the first selected working area indicated by F83 as the first fifth working area, and the first selected working area indicated by F84 as the first sixth working area. Although not shown, the first first position, first second position, first third position, first fourth position, first fifth position, and first sixth position are also set in a similar manner.

[0194] Thereafter, steps S310, S320, and S330 in FIG. 14 are performed, whereby the processor 100 controls the drive device 20 to automatically move the endoscope 40 from the first first position to the first sixth position.

[0195] In this way, the first first working area group set in step S240 is reduced to set as a first second working area group in step S250, and the first second working area group set in step S250 is further reduced to set as a first third working area group in step S265, and a first first working area is set from the first working areas included in the first third working area group. From the above, in the medical manipulator system 5 of this embodiment, the processor 100 sets the first working area group by calculating information about the working area for each of a plurality of positions in the movable range of the endoscope 40 (step S240), and sets a plurality of working areas that overlap with the treatment target area as a second working area group based on the first working area group (step S250). Furthermore, the processor 100 sets a plurality of working areas to be used in the procedure as a third working area group based on the second working area group, and sets the working area closest to the position of the endoscope 40 as the first working area based on the third working area group (step S265). In this way, the number of working areas can be reduced in stages. This allows for accurate selection of a working area suitable for efficiently performing the procedure.

[0196] 14 to the second procedure (step S20). In this case, the details of step S100 in Fig. 15, step S210 in Fig. 16, and step S220 in Fig. 16 are the same as those in the case where the method of this embodiment is applied to the first procedure (step S20), and therefore will not be described.

[0197] The flowchart in FIG. 46 shows a more detailed example of the process of step S230 when the technique of this embodiment is applied to the second procedure (step S20). The processor 100 sets a second spatial region based on the operating range of the second treatment tool 52 (step S232). Then, the processor 100 sets a third spatial region based on the operating range of the third treatment tool 53 (step S233). Then, the processor 100 calculates a second working region based on the second spatial region and the third spatial region (step S234). Note that step S233 in FIG. 46 is a process common to step S231-A in FIG. 27 in terms of the operating range of the third treatment tool 53. That is, the third spatial region in step S233 is the same as the first spatial region in step S231-A. Therefore, the outline of the third spatial region is as shown by H3 in FIG. 47.

[0198] Step S232 will be conceptually explained. As described above, the second medical manipulator 520 that drives the second treatment tool 52 is configured to be able to move forward and backward and bend using the same mechanism as the endoscope 40 and the third treatment tool 53. Therefore, the general shape of the second spatial region is as shown in H2, similar to the general shape shown in H3.

[0199] Step S234 will be conceptually explained. In the second procedure (step S20), the second treatment tool 52 grasps the treatment target, and the third treatment tool 53 performs an incision. Therefore, as shown in H12 in FIG. 48(A), it is desirable that the second treatment tool 52 be positioned in a region between the 12 o'clock direction and the 6 o'clock direction when viewed from the direction in which the tip of the endoscope faces the second spatial region. Note that the 12 o'clock direction here refers to the 12 o'clock direction when the second spatial region viewed from the direction in which the tip of the endoscope faces is considered to be the face of a 12-hour clock. Similarly, the 6 o'clock direction here refers to the 6 o'clock direction when the second spatial region viewed from the direction in which the tip of the endoscope faces is considered to be the face of a 12-hour clock. Furthermore, in the second procedure (step S20), the second treatment tool 52 is required to grasp the treatment target and pull it up in the +Z0 direction. Therefore, it is desirable that the second treatment tool 52 be able to move in the Z0 direction within the range indicated by H14 in Fig. 48(B), for example. The specific range indicated by H14 is determined appropriately depending on the case.

[0200] Furthermore, when viewed from the X0-Z0 plane, in the second procedure (step S20), it is desirable that the third treatment tool 53 move within the range indicated by H24 in FIG. 48(B) based on the direction along the X0 direction. The specific range indicated by H24 is determined appropriately depending on the case. Therefore, when the third spatial region indicated by H23 in FIG. 49 is viewed from the distal end of the endoscope, it is desirable that the third treatment tool 53 be positioned within the range indicated by the dotted lines indicated by H25 and H26. Note that the dotted lines indicated by H25 and H26 are lines perpendicular to the line indicated by the dashed line indicated by H27. The dashed line indicated by H27 is a line extending from 12 o'clock to 6 o'clock when the third spatial region viewed from the distal end of the endoscope is considered to be the face of a 12-hour clock. In other words, the dotted lines indicated by H25 and H26 are lines parallel to the direction extending from 3 o'clock to 9 o'clock when the third spatial region viewed from the distal end of the endoscope is considered to be the face of a 12-hour clock.

[0201] 48(B), it is considered ideal to orient the tip of the endoscope in a direction parallel to the X0 direction, as shown in the above-described Fig. 31(B), similar to the first procedure (step S10). However, similar to the first procedure (step S10), even if the direction of the tip of the endoscope cannot be made parallel to the X0 direction, it may be possible to perform treatment by bending the treatment tool 50, for example.

[0202] 47, 48, and 49, it is also considered that the second treatment tool and the third treatment tool 53 are visible within a predetermined range of the endoscopic image, as described above with reference to Figures 28(A) and 28(B). Note that the grasping forceps, which is the second treatment tool 52, only needs to be positioned so that the user can see it within a predetermined range of the endoscopic image when grasping the treatment target.

[0203] That is, the medical manipulator system 5 of this embodiment includes a plurality of medical manipulators 500 and treatment tools 50 corresponding to the respective medical manipulators 500. The processor 100 acquires information about a working area, which is a range that is within a predetermined range of the endoscopic image and is virtually set based on the range in which each treatment tool 50 is captured and the range of motion of each medical manipulator 500. In this way, the range of the second working area can be set more appropriately.

[0204] From the above, the second spatial region shown in H32 in Fig. 50 is calculated in step S232, but the second spatial region is reduced to the region shown in H42 in step S234. Similarly, the third spatial region shown in H33 in Fig. 50 is calculated in step S233, but the third spatial region is reduced to the region shown in H43 in step S234. Note that although Fig. 50 shows the X0-Y0 planar view, the second spatial region and the third spatial region are also reduced in the Z0 direction. Specifically, for example, although not shown, the second spatial region and the third spatial region are reduced in consideration of the relationship between the position of the lens of the imager 42 described above in Fig. 30 and the position where the medical manipulator 500 protrudes, the conditions described above in Figs. 48(B) and 49, and the like.

[0205] The second working area finally calculated in step S234 may be appropriately selected by the user. For example, if it is desired that the medical manipulator system 5 automatically determine all working positions as described below, the area shown by H44 in FIG. 50 may be set as the working area. The second working area shown by H44 is an area where both the reduced second spatial area shown by H42 and the reduced third spatial area shown by H43 overlap. Furthermore, for example, if the user desires the medical manipulator system 5 to determine a position suitable for using the second treatment tool 52 and to manually operate the third treatment tool 53, the reduced second spatial area shown by H42 may be set as the second working area. Furthermore, for example, if the user desires the medical manipulator system 5 to determine a position suitable for using the third treatment tool 53 and to manually operate the second treatment tool 52, the reduced second spatial area shown by H43 may be set as the second working area.

[0206] For ease of explanation, the direction of the line shown in H47 with respect to the second work area shown in H44 may be referred to as the "direction in which the second work area faces." Similarly, if the work area shown in H42 is the second work area, the direction of the line shown in H45 may be referred to as the "direction in which the second work area faces," and if the work area shown in H43 is the second work area, the direction of the line shown in H46 may be referred to as the "direction in which the second work area faces."

[0207] Thereafter, the processor performs step S240 in Fig. 16 and then step S250. Explanation of step S240 will be omitted for the same reasons as for step S100, etc. The flowchart in Fig. 51 is a more detailed example of the processing of step S250 when the technique of this embodiment is applied to the second procedure (step S20). Note that in the explanation of Fig. 51, explanations of processes common to the flowchart in Fig. 36 (steps S251, S252, S258, S259, etc.) will be omitted as appropriate.

[0208] When the determination in step S251 is YES, the processor 100 performs the above-mentioned step S252 to determine whether the selected second working area intersects with the processing target (step S254-B). That is, in step S254-B of FIG. 51, it does not matter to what extent the selected second working area intersects with the processing target.

[0209] If the processor 100 determines that the selected second working area does not intersect with the processing target (NO in step S254B), it excludes the selected working area (step S258) and performs step S251 again.

[0210] On the other hand, when the processor 100 determines that the selected working area intersects with the treatment target (YES in step S254-B), it determines whether the direction of the treatment tool tip is within the target range (step S256).

[0211] The purpose of step S256 in Fig. 51 is similar to the purpose of step S256 in Fig. 36. That is, although not shown, the determination is based on whether the third treatment tool 53 can reach the treatment target within the angle range shown by H24 in Fig. 48(B). Whether the direction in which the tip of the third treatment tool 53 faces falls within an appropriate angle range depends on the relationship between the X0 direction and the direction in which the second working area faces, the size of the treatment target, and the like, as in the case described above with Fig. 38. Furthermore, as in the case described above with Fig. 39, if there is an area in which it is appropriate to perform the second procedure (step S20) in a certain range or more, the process may be such that YES is returned in step S256 even if the tip of the third treatment tool 53 cannot be directed within an appropriate angle range for the entire treatment target.

[0212] If the direction of the treatment tool tip is within the target range (YES in step S256), the processor 100 performs step S257 and then performs step S251 again. On the other hand, if the direction in which the treatment tool tip is facing is not within the target range (NO in step S256), the processor 100 performs step S258 and then performs step S251 again. If the processor 100 determines NO in step S251, it ends the flow. This sets a second second working area group consisting of the second working areas obtained through step S257.

[0213] Thereafter, it is assumed that the user does not wish to select the working area by himself / herself and selects to operate the medical manipulator system 5 in the first mode (NO in step S260-1, step S261). An example of the processing in step S265 in this case will be described with reference to Figs. 52, 53, and 54.

[0214] 52, processor 100 sets a second first selected working area from the second group of second working areas set in step S250 (step S265-1B). For example, processor 100 selects a second working area that includes the boundary of the processing object and has a large overlapping area with the processing object, and sets it as the second first selected working area. Processor 100 then arranges the second selected working area according to a predetermined rule (step S265-2B).

[0215] Conceptually, for example, in FIG. 53, a treatment target is located at the position indicated by H60, and the tip of the endoscope is located at the position indicated by H61. The relationship between the position information of the treatment target indicated by H60 in FIG. 53 and the position information indicated by H61 is the same as the relationship between the position information of the treatment target indicated by F40 in FIG. 43 and the position information indicated by F41. In other words, the situation shown in FIG. 53 is a situation in which a user performs a first procedure (step S10) in the situation shown in FIG. 43, and then subsequently performs a second procedure (step S20). The treatment target indicated by H60 in FIG. 53 may have a slightly different shape, etc., compared to the treatment target indicated by F40 in FIG. 43 due to incision when the first procedure (step S10) is further performed. However, in this embodiment, the treatment target has the same three-dimensional shape information before and after the first procedure (step S10).

[0216] The area shown in H62 in Fig. 53 is the second work area calculated in step S230, and corresponds to the work area shown in H42 in Fig. 50. Note that the area shown in H62 in Fig. 53 may correspond to the second work area shown in H43 in Fig. 50, or may correspond to the second work area shown in H44 in Fig. 50.

[0217] In the situation shown in Figure 53, suppose that steps up to step S250 have been performed and a second set of second working areas has been set. Then, suppose that step S265-1B in Figure 52 selects the second working area shown in H71 in Figure 54 as the second selected first working area. The second working area shown in H71 includes the boundary of the treatment object and intersects with the treatment object over a wide area.

[0218] Thereafter, in step S265-2B of FIG. 52, a second selected working area shown in H72, a second selected working area shown in H73, a second selected working area shown in H74, a second selected working area shown in H75, and a second selected working area shown in H76 are selected and arranged, respectively. The second selected working areas shown in H72 to H76 are selected, for example, as follows: Based on the position coordinates of the tip of the endoscope corresponding to the second first selected working area shown in H71, positions are set and arranged in a grid pattern at a predetermined interval. Then, the second working area when the set position is the position of the tip of the endoscope is selected as the second selected working area and arranged together with the second first selected working area. The predetermined interval is set based on the size of the second working area, etc., so that the arranged second working areas overlap each other.

[0219] In the second procedure (step S20), ESD, the range of motion of the second treatment tool 52 and the third treatment tool 53 must cover the entire area of ​​the treatment target. Therefore, by automatically performing the method described above in steps S265-1B and S265-2B, the second selection working area can be appropriately selected. Note that in Figure 54, the second selection working areas are arranged in a grid pattern, but they may also be arranged in a staggered pattern, which can be determined appropriately by the user.

[0220] Returning to the flow of FIG. 52, the explanation will be continued. Thereafter, the processor 100 sets a second Nth working area (step S265-3B). More specifically, the processor 100 sets the order of the second Nth working area to the second Nth working area. The processor 100, for example, performs a process of selecting the position of the endoscope tip closest to the first first position (position shown in F91 of FIG. 45) corresponding to the first first working area (working area shown in F81 of FIG. 45) described above as the second first position, and a process of setting a second selected working area related to the selected second first position as the second first working area. Specifically, in the case of FIG. 54, for example, the processor 100 sets the second selected working area shown in H72 as the second first working area and sets the second first position (not shown). Note that if the first procedure (step S10) is omitted, the second working area related to the position of the endoscope tip closest to the current position of the endoscope tip may be set as the second first working area.

[0221] From the above, in the medical manipulator system 5 of this embodiment, the second first position is the position of the endoscope corresponding to the plurality of second working areas that is closest to the first first position. In this way, the start position of the second procedure (step S20) can be made closer to the start position of the first procedure (step S10). This can improve the work efficiency of the treatment.

[0222] Then, processor 100 sets a second second working area, a second third working area, a second fourth working area, a second fifth working area, and a second sixth working area. In this way, processor 100 sets a set of second working areas consisting of second first working area to second sixth working area as a second third working area group.

[0223] Since the second procedure (step S20), ESD, involves dissecting a lesion using the third treatment tool 53 from the proximal side as viewed from the distal end of the endoscope, it is considered convenient to set the order of the second working areas in order of proximity to the distal end of the endoscope. Therefore, in the situation shown in FIG. 54 , the processor 100 sets, for example, the second working area indicated by H71 as the second second working area. Similarly, the processor 100 sets the second working area indicated by H73 as the second third working area, the second working area indicated by H75 as the second fourth working area, the second working area indicated by H74 as the second fifth working area, and the second working area indicated by H76 as the second sixth working area. Although not shown, a second first position, a second second position, a second third position, a second fourth position, a second fifth position, and a second sixth position are also set. Thereafter, the processor 100 ends the flow of step S260, and sequentially moves the endoscope 40 to the second N positions corresponding to the respective second N working areas in the subsequent steps S310, S320, and S330.

[0224] Note that instead of steps S265-1B and S265-2B in FIG. 52, the second 1st selected working area to the second Nth selected working area may be set by the method described below. For example, although not shown in a flowchart, processor 100 calculates the length of the overlapping line and the size of the overlapping area for each second selected working area. The size of the overlapping area refers to the size of the area where the treatment target and the second working area overlap in the X0-Y0 planar view. Processor 100 also stores data in memory that associates the calculated length of the overlapping line and the size of the overlapping area for each second working area.

[0225] The processor 100 then sets a second first selected working area based on the data on the length of the superimposed line and the size of the superimposed area stored in memory. For example, the processor 100 extracts a predetermined number of data items with the largest sizes from the data on the size of the superimposed area. The processor 100 may then set the second working area associated with the data item with the longest superimposed line length as the second first selected working area. In this manner, the second first selected working area indicated by J11 is set, as conceptually shown at J10 in FIG. 55, for example. In other words, the superimposed line indicated by J12 is longer than the superimposed line associated with the other candidate second working area, and therefore the second working area indicated by J11 is selected as the second first selected working area.

[0226] The processor 100 may display a display similar to that of J10 on the display 610 or the touch panel 620. The same applies to the display of J20 and J30 in Fig. 55, and the display of J50 and J60 in Fig. 56 described later.

[0227] Thereafter, the processor 100 sets a second second selected working area as shown in J20. For example, by selecting a second first selected working area, the processor 100 sets a first end portion shown in G21 and a second end portion shown in G22. Then, for second working areas other than the second first selected working area, the processor 100 extracts a predetermined number of data items with the largest sizes from the data on the overlapping areas. The processor 100 then sets the second working area associated with the extracted data item that includes the first end portion shown in G21 and has the longest overlapping line as the second second selected working area. Second selected working areas from the second third selected working area onward are set in the same manner. As a result, in addition to the second first selected working area shown in J21, a second first selected working area shown in J22 is set. In other words, since the overlapping line shown in J23 is longer than the overlapping line associated with the other candidate second working area, the second working area shown in J22 is selected as the second second selected working area.

[0228] The processor 100 also determines whether the selected second second selected working area includes the second end shown in G22. In the example shown in J20 in Fig. 55, the second second selected working area shown in J22 does not include the second end shown in G22, so the processor 100 further performs processing to set a second third selected working area.

[0229] By repeating the above method, the situation shown at J30 in Fig. 55 is reached. That is, a second first selected working area is set at J31, a second second selected working area is set at J32, a second third selected working area is set at J33, a second fourth selected working area is set at J34, a second fifth selected working area is set at J35, a second sixth selected working area is set at J36, a second seventh selected working area is set at J37, and a second eighth selected working area is set at J38. The second eighth selected working area shown at J38 includes the second end.

[0230] Then, the processor 100 calculates the differential region when the situation shown at J30 in Fig. 55 occurs. The differential region is a region of the processing target region in the X0-Y0 plan view that does not overlap with any of the second selected working regions selected up to that point, such as the region shown at J40 in Fig. 56. That is, the processor 100 calculates the area of ​​the differential region shown at J40 in Fig. 56 and stores the calculated area data in memory. Note that the dotted line shape shown at J41 is an excerpt of the outline of the differential region shown at J40.

[0231] The processor 100 also calculates the length of the contour of the differential region. That is, the processor 100 obtains a set of points that form the contour of the differential region shown at J41 in Fig. 56, and stores data of the obtained set of points in memory.

[0232] Processor 100 then further sets a second selected working area based on the data on the area of ​​the differential region and the data on the contour length of the differential region. For example, processor 100 determines the size of the area that overlaps with the differential region for an unselected second working area. Processor 100 then extracts a predetermined number of data items with the largest area sizes that overlap with the differential region. Processor 100 then determines the contour length of the differential region included in the area that overlaps with the differential region for the second working area associated with each extracted data item. Processor 100 then further sets the second working area associated with the boundary line of the differential region with the longest contour length among the determined lengths of the contours of the differential regions as the second selected working area.

[0233] Specifically, for example, assume that the second work area shown in J51 is selected using the above-described method, as shown in J50 in Fig. 56. As a result, the second work area shown in J51 is set as the second ninth selected work area.

[0234] Then, by repeating the same method, a second ninth selected working area shown in J61, a second tenth selected working area shown in J62, and a second eleventh selected working area shown in J63 are set in the X0-Y0 plan view, as shown in J60 in Fig. 56. As a result, the entire treatment target area is superimposed on the set of second selected working areas that have been set.

[0235] By the above method, the second first selected working area to the second eleventh selected working area are selected from the second second working area group set in step S250 to perform the second procedure (step S20), and the second third working area group is set. In the examples of Figures 55 and 56, the position of endoscope 40 needs to be controlled at least 11 times to perform the second procedure (step S20).

[0236] The method for setting the second set of third working areas is not limited to the above. Many mathematical methods have been proposed for efficiently superimposing the area of ​​the treatment target on the second working area, and any of these mathematical methods may be appropriately adopted in step S265 of FIG.

[0237] Although the present embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible without substantially departing from the novel features and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of the medical manipulator system, processor, control method, etc. are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0238] 1...Surgical system, 5...Medical manipulator system, 10...Control device, 20...Driver, 21...First treatment tool drive device, 22...Second treatment tool drive device, 23...Third treatment tool drive device, 24...Endoscope drive device, 40...Endoscope, 42...Imager, 44...Treatment tool insertion port, 46...Overtube, 48...Cap, 50...Treatment tool, 51...First treatment tool, 52...Second treatment tool, 53...Third treatment tool, 60...Console, 70...Driving unit, 72...Source coil, 100...Processor, 220...Motor unit, 221...First bending operation drive section, 222...Second bending operation drive section, 223...Opening and closing operation drive section, 224...Rolling operation drive section, 225...Advancing and retreating operation drive section, 410...First sensor, 420...Second sensor, 422...Source coil, 424, 424A, 424B, 424 C, 424D...optical fiber, 428...Bragg grating section, 500...medical manipulator, 510...first medical manipulator, 516...injection needle, 520...second medical manipulator, 522...gripping section, 530...third medical manipulator, 610...display, 620...touch panel, 630...foot pedal, 631...first foot pedal, 632...second foot pedal, 633... Third foot pedal, 640...handle, 641...first handle, 642...second handle, 651...first part, 652...second part, 653...third part, RE1, RE2, RE3, RE4, RE11, RE12, RE13, RE14, RE15, RE16, RE17, RE18, RE51, RE52, RE53, RE54, RE55, RE56...angle, TA...operating table, t...distance

Claims

1. an endoscope including an imager for capturing an endoscopic image; a medical manipulator that protrudes beyond the distal end surface of the endoscope; a drive unit that controls the endoscope and the medical manipulator; a processor; Including, The processor: acquiring the endoscopic image showing the treatment target from the imager; Recognizing the region of the treatment target from the endoscopic image; Acquire three-dimensional shape information of the lumen, Acquire three-dimensional shape information of the endoscope; calculating information about a working area, which is a range in which the medical manipulator can work, for each of a plurality of positions in the movable range of the endoscope, thereby acquiring information about a plurality of working areas; A medical manipulator system characterized by controlling the drive of the endoscope to a first position corresponding to a first working area selected from the plurality of working areas based on the area to be treated, three-dimensional shape information of the lumen, and three-dimensional shape information of the endoscope.

2. 2. The medical manipulator system of claim 1, The processor: a medical manipulator system that acquires information about the working area, which is a range that is within a predetermined range of the endoscopic image and that is virtually set based on a range in which a treatment tool corresponding to the medical manipulator is captured and a movable range of the medical manipulator.

3. 2. The medical manipulator system of claim 1, a plurality of the medical manipulators and treatment tools corresponding to the medical manipulators, The processor: a medical manipulator system that acquires information about the working area, which is a range that is within a predetermined range of the endoscopic image and is virtually set based on the range in which each of the treatment tools is captured and the movable range of each of the medical manipulators.

4. 2. The medical manipulator system of claim 1, The processor: a medical manipulator system characterized in that information on a working area, which is a virtually set range, is acquired by superimposing a range in the endoscopic image that is within a given range suitable for arrangement of a treatment tool associated with the medical manipulator and in which the treatment tool associated with the medical manipulator is shown, on a movable range of the medical manipulator.

5. 2. The medical manipulator system of claim 1, The processor: Calculating an overlap length between the boundary line of the treatment object and the working area; A medical manipulator system, characterized in that a predetermined group of working areas including the first working area is set based on the working area with the longest length among the plurality of working areas.

6. 2. The medical manipulator system of claim 1, The processor: calculating information about the working area for each of a plurality of positions in the range of motion of the endoscope to set a first working area group; Based on the first working area group, a plurality of working areas overlapping the treatment target area are set as a second working area group; setting a plurality of the working areas to be used in the procedure as a third working area group based on the second working area group; A medical manipulator system, characterized in that a working area closest to a position of an endoscope is set as the first working area based on the third working area group.

7. 7. The medical manipulator system of claim 6, The processor: selecting the working area to be used for the procedure as a selected working area from among the working areas included in the second working area group; A medical manipulator system, characterized in that the third group of working areas is set by arranging a plurality of the selected working areas along the boundary line of the treatment object.

8. The medical manipulator system of claim 7, The processor: The selected working area having the longest overlapping length with the boundary line of the treatment object is set as a first selected working area; a medical manipulator system characterized in that the selected working area that includes a first end, which is one end of the boundary line included in the first selected working area, and that has the longest overlapping length with the boundary line is set as a second selected working area.

9. The medical manipulator system of claim 8, The processor: When an Nth selected working area including a second end portion that is the other end portion of the boundary line included in the first selected working area is selected, A medical manipulator system, characterized in that a plurality of the selected working areas including the first selected working area to the Nth selected working area are set as a third working area group.

10. 2. The medical manipulator system of claim 1, The processor: selecting the first working area and the second working area from the plurality of working areas; A medical manipulator system characterized in that the endoscope is moved to the first position corresponding to the selected first working area, and then the endoscope is moved to a second position corresponding to the selected second working area.

11. 2. The medical manipulator system of claim 1, The processor: In a first procedure, driving and controlling the position of the endoscope to a first first position corresponding to a first first working area selected from a plurality of first working areas; A medical manipulator system characterized in that, in a second procedure, the position of the endoscope is driven and controlled to a second first position corresponding to a second first working area selected from a plurality of second working areas.

12. The medical manipulator system of claim 11, The second first position is A medical manipulator system, characterized in that the first position is the closest position to the first position among the positions of the endoscope corresponding to the plurality of second working areas.

13. 2. The medical manipulator system of claim 1, A first sensor; A second sensor; Including, The processor: acquiring three-dimensional shape information of the lumen from the first sensor; a medical manipulator system, characterized in that three-dimensional shape information of the endoscope is obtained from the second sensor;

14. The medical manipulator system of claim 13, The first sensor is TOF (Time Of Flight) method or LIDAR (Light Detection and Ranging) method It is a distance sensor by the formula, The second sensor is A medical manipulator system characterized by a curved shape observation sensor that uses a magnetic field sensing method or a distortion sensing method.

15. An endoscope including an imager for capturing an endoscopic image, a medical manipulator protruding from a distal end surface of the endoscope, and a processor for controlling a drive device for controlling the endoscope and the medical manipulator, acquiring the endoscopic image showing the treatment target from the imager; Recognizing the region of the treatment target from the endoscopic image; Acquire three-dimensional shape information of the lumen, Acquire three-dimensional shape information of the endoscope; calculating information about a working area, which is a range in which the medical manipulator can work, for each of a plurality of positions in the movable range of the endoscope, thereby acquiring information about a plurality of working areas; A processor that controls the drive of the endoscope to a first position corresponding to a first working area selected from the plurality of working areas based on the area to be treated, three-dimensional shape information of the lumen, and three-dimensional shape information of the endoscope.

16. A control method for controlling an endoscope including an imager that captures an endoscopic image, a medical manipulator that protrudes beyond a distal end surface of the endoscope, and a drive device that controls the endoscope and the medical manipulator, comprising: acquiring the endoscopic image showing the treatment target from the imager; Recognizing the region of the treatment target from the endoscopic image; Acquire three-dimensional shape information of the lumen, Acquire three-dimensional shape information of the endoscope; calculating information about a working area, which is a range in which the medical manipulator can work, for each of a plurality of positions in the movable range of the endoscope, thereby acquiring information about a plurality of working areas; A control method characterized by driving and controlling the endoscope to a first position corresponding to a first working area selected from the plurality of working areas based on the area to be treated, three-dimensional shape information of the lumen, and three-dimensional shape information of the endoscope.

17. 17. The control method of claim 16, a control method for acquiring information on the working area, the working area being a range that is virtually set based on a range within a predetermined range of the endoscopic image in which a treatment tool corresponding to the medical manipulator is captured and a movable range of the medical manipulator.

18. 17. The control method of claim 16, Calculating an overlap length between the boundary line of the treatment object and the working area; A control method comprising: setting a predetermined group of work areas including the first work area based on the work area with the longest length among the plurality of work areas.

19. 17. The control method of claim 16, selecting the first working area and the second working area from the plurality of working areas; A control method characterized by moving the endoscope to the first position corresponding to the selected first working area, and then moving the endoscope to a second position corresponding to the selected second working area.

20. 17. The control method of claim 16, In a first procedure, driving and controlling the position of the endoscope to a first first position corresponding to a first first working area selected from a plurality of first working areas; A control method characterized by driving and controlling the position of the endoscope to a second first position corresponding to a second first working area selected from a plurality of second working areas in a second procedure.

Images