Endoscopic system and method of operating the endoscopy system

Abstract

An endoscope system (1) comprises: an endoscope (2); a movement device (3) that moves the endoscope (2); and a control device (4) that controls the endoscope (2) and the movement device (3). The control device (4): receives an image captured by the endoscope (2); calculates a first orientation of a first treatment tool in the image; determines whether the endoscope (2) is tracking the movement of the first treatment tool on the basis of the first orientation; when it is determined that the endoscope is tracking the movement of the first treatment tool, generates a command for the movement device (3) in order for the endoscope (2) to track the first treatment tool, and when it is determined that the endoscope is not tracking the movement of the first treatment tool, generates a command that is for the movement device (3) and that is for causing the endoscope (2) to stop tracking the first treatment tool.

Description

[Technical Field]

[0001] The present invention relates to an endoscopic system, and more particularly to an endoscopic system that allows the endoscope to follow a treatment instrument that it is tracking. [Background technology]

[0002] Conventionally, endoscopic systems are known that allow the endoscope to track a treatment instrument of interest, such as a treatment instrument held in the operator's right hand or a treatment instrument with a marker attached (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Patent No. 10028792 [Overview of the project] [Problems that the invention aims to solve]

[0004] In order to make an endoscope track a treatment instrument, it is necessary to accurately identify the treatment instrument to be tracked within the image. However, Patent Document 1 has the drawback of insufficient accuracy in identifying the treatment instrument to be tracked. For example, when multiple treatment instruments are present in the body, or when a specific treatment instrument of interest is identified by image recognition, misidentification of the treatment instrument to be tracked may occur. Furthermore, if the treatment instrument to be tracked is removed from the body and another treatment instrument is inserted, the other treatment instrument cannot be identified as the new target for tracking.

[0005] The present invention has been made in view of the above circumstances, and aims to provide an endoscopic system that can accurately determine the treatment instrument to be tracked. [Means for solving the problem]

[0006] One aspect of the present invention comprises an endoscope, a moving device for moving the endoscope, and a control device for controlling the endoscope and the moving device, wherein the control device receives an image captured by the endoscope. The first image is the image received at any point in time. The first position of the first treatment instrument in the image is calculated, The second posture of the second treatment instrument, which is the tracking target, is calculated in the second image, which is the image received a predetermined time before the first image, the difference in posture between the first posture and the second posture is calculated, and by comparing the difference in posture with a threshold, The system determines whether the endoscope follows the movement of the first treatment instrument, and if it determines that the endoscope follows the movement, it generates a command to the moving device for the endoscope to follow the first treatment instrument. accomplish It is an endoscopic system. [Effects of the Invention]

[0007] According to the present invention, the effect is that the treatment tool to be followed can be accurately identified. [Brief explanation of the drawing]

[0008] [Figure 1A] This is an external view of the overall configuration of an endoscope system according to one embodiment of the present invention. [Figure 1B] This diagram illustrates the movement of an endoscope and treatment instruments inserted into the body through a port in the body wall. [Figure 2A] Figure 1 is a block diagram showing the hardware configuration of the control unit of the endoscope system. [Figure 2B] Figure 1 is a block diagram showing the functions of the control system for the endoscope system. [Figure 3A] This is an example of an endoscopic image obtained during surgery. [Figure 3B] Here is another example of an endoscopic image taken during surgery. [Figure 4A] This diagram illustrates an example of how to set up the tracking target. [Figure 4B] This diagram illustrates another example of how to configure the target to follow. [Figure 5A] This diagram illustrates an example of a method for recognizing medical instruments. [Figure 5B] This diagram illustrates another example of a method for recognizing medical instruments. [Figure 6] This is a diagram illustrating examples of the position of medical instruments. [Figure 7] It is a flowchart of a following method according to the first embodiment. [Figure 8A] It is a flowchart of a modification example of the following method according to the first embodiment. [Figure 8B] It is a flowchart of the candidate narrowing routine of FIG. 8A. [Figure 9] It is an example of an endoscopic image during the replacement of the treatment tool that is the following target. [Figure 10] It is a flowchart of a following method according to the second embodiment. [Figure 11] It is a diagram for explaining an example of the position of the treatment tool. [Figure 12A] It is a diagram for explaining a method of determining a following target based on a pivot point. [Figure 12B] It is a diagram for explaining a method of determining a following target based on a pivot point. [Figure 13] It is a flowchart of a following method according to the third embodiment. [Figure 14A] It is a flowchart of a modification example of the following method according to the third embodiment. [Figure 14B] It is a flowchart of the candidate narrowing routine of FIG. 14A. [Figure 15] It is a flowchart of a modification example of the following method according to the third embodiment. [Figure 16] It is a diagram for explaining a method of estimating a pivot point from an endoscopic image. [Figure 17] It is a flowchart of a following method according to the fourth embodiment. [Figure 18] It is a diagram showing an example of the posture and position of the treatment tool superimposed on the endoscopic image.

Mode for Carrying Out the Invention

[0009] (First Embodiment) Hereinafter, an endoscopic system according to the first embodiment of the present invention will be described with reference to the drawings. As shown in Figures 1A and 1B, the endoscopic system 1 according to this embodiment is used in surgery that uses an endoscope 2 and one or more treatment instruments 6, such as laparoscopic surgery. Although only one treatment instrument 6 is shown in Figures 1A and 1B, two or more treatment instruments 6 may be used.

[0010] As shown in Figure 1B, multiple holes C are formed in the body wall B of patient A as ports for inserting an endoscope 2 and one or more treatment instruments 6 into the body. The endoscope 2 and one or more treatment instruments 6 are inserted into the body via trockers 7 inserted into each port C, and are supported by the body wall B so as to be able to swing around a predetermined pivot point P, which is the position of the port C, and their position and orientation within the body can be changed by swinging around the pivot point P.

[0011] The endoscopic system 1 comprises an endoscope 2, a moving device 3 that holds the endoscope 2 and moves the endoscope 2 inside the body, a control device 4 connected to the endoscope 2 and the moving device 3 and controlling the endoscope 2 and the moving device 3, and a display device 5 that displays endoscopic images. The endoscope 2 is, for example, a rigid endoscope and includes an imaging unit (not shown) which has an image sensor and acquires an endoscopic image. As shown in Figures 3A and 3B, the endoscope 2 acquires an endoscopic image D including one or more treatment instruments 61, 62, 63, 64 by the imaging unit and transmits the endoscopic image D to the control device 4.

[0012] The mobile device 3 comprises a robotic arm 3a having multiple joints 3b, and holds the proximal end of the endoscope 2 at the tip of the robotic arm 3a. The mobile device 3 can move the endoscope 2 and change its position and orientation by the movement of the joints 3b.

[0013] As shown in Figure 2A, the control device 4 comprises at least one processor 4a, such as a central processing unit, a storage unit 4b, an input interface 4c, an output interface 4d, and a network interface 4e. The endoscopic image D transmitted from the endoscope 2 is sequentially input to the control device 4 via the input interface 4c, and sequentially output to the display device 5 via the output interface 4d, where it is displayed. The operator operates the treatment instruments 61, 62, 63, and 64 inserted into the body while observing the endoscopic image D displayed on the display device 5, and performs treatment on the affected area inside the body using the treatment instruments 61, 62, 63, and 64. The display device 5 is any display, such as a liquid crystal display.

[0014] The memory unit 4b includes a volatile working memory such as RAM (random access memory) and a non-volatile recording medium such as ROM (read-only memory) or a hard disk. The non-volatile recording medium stores the program and data necessary to cause the processor 4a to execute processing. The functions of the control device 4 described later are realized when the program is executed by the processor 4a. Some functions of the control device 4 may be realized by dedicated logic circuits or the like.

[0015] As shown in Figures 3A and 3B, the type, arrangement, and number of instruments in the endoscopic image D change during surgery. For example, an energy device 61 may be replaced with forceps 63, or an instrument 62 may be removed and then another instrument 64 inserted from another port. The control device 4 sets a treatment instrument 61 in the endoscopic image D as the tracking target, and controls the moving device 3 based on the position of the tracking target 61 to make the endoscope 2 follow the tracking target 61, thereby keeping the tracking target 61 within the field of view F of the endoscope 2. For example, the control device 4 controls the moving device 3 based on the position of the tracking target 61 in the endoscopic image D so that the tip of the tracking target treatment instrument 61 is positioned in a predetermined central region within the endoscopic image D.

[0016] Specifically, as shown in Figure 2B, the control device 4 includes a target setting unit 41 for setting the treatment instrument 61 to be tracked, a recognition unit 42 for recognizing treatment instruments 61 and 62 in the current endoscope image D, a calculation unit 43 for calculating the current posture of each treatment instrument 61 and 62 in the current endoscope image D, a determination unit 44 for determining whether the endoscope 2 will track the movement of each treatment instrument 61 and 62 based on the current posture of each treatment instrument 61 and 62 and the past posture of the tracked target 61, a storage unit 45 for storing the current posture of the tracked target treatment instrument 61 that the endoscope 2 is determined to track, and a command generation unit 46 for generating commands to the moving device 3.

[0017] The target setting unit 41 either automatically sets one treatment instrument 61 as the target to follow, or sets one treatment instrument 61 as the target to follow based on the operator's operation. In automatic mode, as shown in Figure 4A, a treatment instrument that is present in the endoscopic image D and meets predetermined conditions is set as the target for tracking. In one example, the predetermined condition is the type of treatment instrument. For example, an energy device is a treatment instrument that the physician focuses on during the procedure, and usually only one is present in the endoscopic image D. The target setting unit 41 recognizes the type of each treatment instrument 61,62 in the endoscopic image D and sets the energy device 61, such as an electrosurgical unit, as the target to track.

[0018] In manual operation, as shown in Figure 4B, the target instrument 61 is set as the tracking target by moving it to a predetermined cursor E set within the endoscopic image D. Alternatively, the target instrument 61 is set as the tracking target by specifying the number assigned to each instrument 61, 62 or the number assigned to the port by the operator.

[0019] The recognition unit 42 recognizes all treatment instruments 61 and 62 within the current endoscopic image D, which is the latest endoscopic image acquired by the endoscope 2, by processing the current endoscopic image D. Known image recognition techniques using deep learning, such as semantic segmentation shown in Figure 5A or instance segmentation shown in Figure 5B, are used to recognize the treatment instruments 61 and 62. The calculation unit 43 calculates the current two-dimensional or three-dimensional posture of each of the treatment instruments 61, 62 recognized by the recognition unit 42.

[0020] As shown in Figure 6, an example of the two-dimensional orientation of the instruments 61 and 62 is either the longitudinal vector q of each instrument 61 and 62 on the image plane (XY plane) of the endoscopic image D, or the angle φ that the longitudinal axis G of each instrument 61 and 62 makes with a predetermined reference axis H in the endoscopic image D. One example of the three-dimensional orientation of the instruments 61 and 62 is the longitudinal vector q of each instrument 61 and 62 in the three-dimensional space of the endoscopic image D. Other examples of the three-dimensional orientation of the instruments 61 and 62 are the angle φ of the instruments 61 and 62 on the image plane, the inclination angle of the instruments 61 and 62 in the depth direction (Z direction) perpendicular to the image plane, and the angle that the longitudinal axis G makes with the X or Y axis (horizontal and vertical axes of the endoscopic image D). The vector q or angle is calculated from the endoscopic image D, which is a stereo image containing information on the three-dimensional position of the instruments 61 and 62, or is acquired by an optical or magnetic orientation sensor. Taking advantage of the fact that the thickness of the shafts 61a and 62a of the instruments 61 and 62 in the endoscopic image D differs depending on the depth direction, the vector or angle may also be calculated from the thickness of the shafts 61a and 62a in the two-dimensional endoscopic image D.

[0021] The judgment unit 44 acquires the past posture (second posture) of the tracking target (second treatment instrument) from the storage unit 45. The past posture is the posture of the tracking target in past endoscopic images D, and as will be described later, it is the posture stored in the storage unit 45 a predetermined time before the current posture. Next, the determination unit 44 determines whether or not the endoscope 2 will follow the movement of each treatment instrument 61, 62, based on the current position of each treatment instrument 61, 62 and the past position of the target to be followed.

[0022] Specifically, the judgment unit 44 compares the current position (first position) of the treatment instrument (first treatment instrument) 61 with the past position of the target to be tracked. If the current position of the treatment instrument 61 is the same as or nearly the same as the past position of the target to be tracked, the judgment unit 44 determines that the endoscope 2 will track the movement of the treatment instrument 61. If the current position of the treatment instrument 61 is not the same as or nearly the same as the past position of the target to be tracked, the judgment unit 44 determines that the endoscope 2 will not track the movement of the treatment instrument 61.

[0023] For example, the determination unit 44 calculates the deviation d, which is the amount of deviation between the current posture of the treatment instrument 61 and the past posture of the object being tracked. If the magnitude of the deviation d is less than or equal to a predetermined threshold, the determination unit 44 determines that the current posture of the treatment instrument 61 is the same as or approximately the same as the past posture of the object being tracked. If the magnitude of the deviation d is greater than the predetermined threshold, the determination unit 44 determines that the current posture of the treatment instrument 61 is not the same as or approximately the same as the past posture of the object being tracked. Similarly, the determination unit 44 determines whether the endoscope 2 follows the movement of the other treatment instrument (third treatment instrument) 62 based on whether the current position (third position) of the other treatment instrument (third treatment instrument) 62 is the same as or substantially the same as the past position of the target to be followed.

[0024] A treatment instrument whose current position is the same as or nearly the same as the past position of the target to be tracked is a candidate for tracking. If the current position of a single treatment instrument 61 is the same as or nearly the same as the past position of the target to be tracked, that is, if there is only one candidate, the determination unit 44 determines that the candidate treatment instrument 61 is the target to be tracked and determines that the endoscope 2 will track the movement of the treatment instrument 61. On the other hand, if the current positions of multiple treatment instruments 61, 62 are the same as or nearly the same as the past position of the target to be tracked, that is, if there are multiple candidates, the determination unit 44 determines that the treatment instrument 61 with the smallest deviation d is the target to be tracked from among the multiple candidates 61, 62, and ultimately determines that the endoscope 2 will track the movement of the treatment instrument 61.

[0025] Since multiple treatment instruments 61 and 62 are inserted into the body through different ports C, the positions of the multiple treatment instruments 61 and 62 are different from each other, and the position of each treatment instrument 61 and 62 can change within a limited range. That is, the position of the treatment instrument 61 to be tracked is different from the position of the other treatment instruments 62, and is constant or approximately constant as long as the position and position of the endoscope 2 do not change. Therefore, the treatment instrument 61 whose current position is the same as or approximately the same as its past position can be identified as the treatment instrument to be tracked.

[0026] If the endoscope 2 determines that it is not following the movement of all the instruments 61 and 62, for example, if the deviation d of all the instruments 61 and 62 is greater than a threshold, the determination unit 44 determines that there are no instruments corresponding to the tracking target in the endoscope image D and notifies the target setting unit 41 of this determination. The target setting unit 41 responds to the notification from the determination unit 44 and sets the tracking target again.

[0027] In one example, the deviation d is calculated from the following formula (1). q t is a vector representing the current posture, and q t-i This is a vector representing the past posture of the target being followed. In equation (1), each term may be weighted.

number

[0028] The recognition unit 42 may also recognize the type of each treatment tool 61, 62, and the determination unit 44 may determine the target to follow based on the type of treatment tool 61, 62 in addition to the current posture. In this case, the storage unit 45 may further store the type of target to follow.

[0029] The memory unit 45 consists of memory unit 4b. The memory unit 45 stores the posture of the target to be tracked when the target to be tracked is set by the target setting unit 41. This posture is calculated, for example, by the calculation unit 43 from the endoscopic image D when the target to be tracked is set. Subsequently, the memory unit 45 stores the current posture of the treatment instrument 61 that has been determined by the judgment unit 44 to be tracked, from among the current postures calculated by the calculation unit 43. The postures stored in the memory unit 45 are updated by replacing the previously stored posture with the newly calculated current posture. The current posture stored in the memory unit 45 is used as a past posture in the determination of the target to be tracked, which is subsequently performed by the judgment unit 44.

[0030] The command generation unit 46 generates a command to the mobile device 3 based on the determination result of the determination unit 44 and transmits the command to the mobile device 3. Specifically, if it is determined that the endoscope 2 will follow the movement of the treatment instrument 61, the command generation unit 46 generates a command for the endoscope 2 to follow the treatment instrument 61. On the other hand, if it is determined that the endoscope 2 will not follow the movement of the treatment instrument 61, the command generation unit 46 generates a command for the endoscope 2 to stop following the treatment instrument 61.

[0031] Next, we will explain the operation of Endoscopic System 1. Endoscope 2 and one or more treatment instruments 61, 62 are inserted into the body via trockers 7 on different ports C, and an endoscopic image D including one or more treatment instruments 61, 62 is acquired by endoscope 2. The operator performs the procedure using one or more treatment instruments 61, 62 while observing the endoscopic image D displayed on the display device 5.

[0032] During treatment with the treatment instruments 61 and 62, as shown in Figure 7, the control device 4 executes a tracking method to make the endoscope 2 follow the treatment instrument 61, which is moving within the body. The tracking method includes the steps of: setting the treatment instrument 61 to be tracked (step S1); calculating the current position of each treatment instrument 61, 62 in the endoscopic image D (steps S2-S4); determining the target to track based on the current position (steps S5-S8); storing the current position of the target to track (step S9); and generating commands for the mobile device 3 based on the determination result of the target to track (steps S10, S11).

[0033] First, the target setting unit 41 sets a treatment instrument 61 within the endoscopic image D as the tracking target (step S1). After setting the tracking target, the control device 4 starts controlling the moving device 3, causing the endoscope 2 to start tracking the tracking target 61. For example, the control device 4 calculates the two-dimensional or three-dimensional position of the tracking target 61 based on the endoscopic image D, and moves the endoscope 2 using the moving device 3 so that the tracking target 61 is positioned in the central region of the endoscopic image D.

[0034] The control device 4, in parallel with controlling the mobile device 3, executes processes S2 to S11 to determine and track the target to be followed 61. First, the recognition unit 42 receives the current endoscopic image D from the endoscope 2 (step S2) and recognizes all the treatment instruments 61 and 62 in the current endoscopic image D (step S3). Next, the calculation unit 43 calculates the current position of each of the treatment instruments 61 and 62 in the current endoscopic image D (step S4).

[0035] Next, the determination unit 44 obtains the past posture of the target to be followed 61 from the storage unit 45 (step S5). The past posture obtained at this time is the posture from a predetermined time prior to the current posture, which was stored in the storage unit 45 in the previous step S9. Next, the determination unit 44 determines whether the endoscope 2 follows the movement of each treatment instrument 61, 62, that is, whether each treatment instrument 61, 62 is a target for tracking, based on the current position of each treatment instrument 61, 62 and the past position of the tracking target 61 (step S6). For example, the determination unit 44 calculates the deviation d of the current position of each treatment instrument 61, 62 from its past position and determines whether the deviation d of each treatment instrument 61, 62 is below a predetermined threshold and is the minimum.

[0036] If there is a treatment instrument 61 that the endoscope 2 is determined to follow (YES in step S7), the determination unit 44 decides that treatment instrument 61 to be the target of tracking (step S8). Next, the memory unit 45 stores the current position of the treatment instrument 61 that has been decided as the target of tracking (step S9). The command generation unit 46 also generates a command for the endoscope 2 to follow the treatment instrument 61 (step S10) and transmits the command to the moving device 3.

[0037] On the other hand, if there is no treatment instrument that matches the tracking target (NO in step S7), the command generation unit 46 generates a command for the endoscope 2 to stop tracking the treatment instrument 61 (step S11) and transmits the command to the moving device 3. Subsequently, the target setting unit 41 sets one treatment instrument in the endoscope image D as the tracking target again (step S1), and steps S2 to S11 are executed again. Steps S1 to S11 are repeated until tracking is completed in step S12.

[0038] Thus, according to this embodiment, the current position of each treatment instrument 61, 62 in the current endoscopic image D is calculated, and it is determined based on the current position whether each treatment instrument 61, 62 is a target for tracking. Since the position of each treatment instrument 61, 62 is restricted by the port C and the trocker 7, the position of the treatment instrument 61 that is a target for tracking in the endoscopic image D does not usually change abruptly. Also, the positions of multiple treatment instruments 61, 62 in the endoscopic image D are all different from each other. Therefore, it is possible to accurately determine whether each treatment instrument 61, 62 is a target for tracking based on its current position. Furthermore, the deviation d of the current posture relative to the past posture of the target being tracked is calculated, and the target being tracked is determined based on this deviation d. This allows the target being tracked to be determined with only simple calculations.

[0039] In this embodiment, the determination unit 44 tracks and determines only the object to be tracked based on its posture, but as shown in Figures 8A and 8B, it may also determine to track other treatment devices based on their posture. In other words, if the endoscopic image D contains multiple treatment instruments 61 and 62, in step S1, the target setting unit 41 sets one treatment instrument 61 as the tracking target and the other treatment instruments 62 as the tracking targets. In step S6', the decision unit 44 extracts candidates for the tracking target from among the multiple treatment instruments 61 and 62 based on the current position of each treatment instrument 61 and 62 and the past position of the tracking target, for example, based on the deviation d. Subsequently, the decision unit 44 narrows down the candidates for the tracking target to one by tracking the tracking target 62 (step S13).

[0040] Specifically, as shown in Figure 8B, the determination unit 44 selects a tracking target j (step S131) ​​and obtains the past posture of the tracking target j from the storage unit 45 (step S132). Next, the determination unit 44 calculates the deviation d of the current posture of the tracking target j from the past posture in the same manner as in steps S6 and S6', and extracts treatment devices whose deviation d is below a predetermined threshold as candidates for tracking target j (step S133).

[0041] If there is only one candidate ("only one" in step S134), the decision unit 44 determines the candidate to be the target j to track (step S135) and updates the past posture of the target j stored in the memory unit 45 to its current posture (step S136). Next, if there is one or more candidates ("only one" or "multiple" in step S134), the decision unit 44 determines whether the target j to track is included in the list of candidates to follow (step S137), and if it is included in the list of candidates to follow, it removes the target j to track from the list of candidates to follow (step S138). Subsequently, the determination unit 44 selects another target j+1 (step S140) and repeats steps S132 to S138 to perform the processing in steps S132 to S138 for all targets (step S139). This narrows down the candidates for the targets to follow.

[0042] As shown in Figure 8A, if, after narrowing down the candidates in step S13, only one candidate remains to be followed (step S14), the decision unit 44 determines that the remaining candidate will be followed (step S8). On the other hand, if there are zero or more remaining candidates (step S14), the determination unit 44 determines that there are no targets to follow or that it is impossible to determine a target to follow, and the target setting unit 41 sets a target to follow again (step S1).

[0043] Thus, according to this modified method, by tracking all the treatment instruments 61 and 62 contained within the endoscopic image D, it is possible to eliminate those that are likely to be other treatment instruments 62 from the list of candidates to track, thereby narrowing down the list of candidates to track. This suppresses misrecognition of the tracked object and further improves the accuracy of the recognition of the tracked object. If the tracking target is misidentified, Endoscope 2 will continue to track the wrong instrument. Therefore, it is important to prevent misidentification of the tracking target. If, after narrowing down the candidates, there are zero or multiple candidates for the tracking target, Endoscope 2 can be prevented from continuing to track the wrong instrument by resetting the tracking target.

[0044] In the above embodiment, when it is determined that the endoscope 2 is following the movements of multiple treatment instruments 61, 62, the determination unit 44 determines the target to follow based on the deviation d, and the command generation unit 46 generates a command for the endoscope 2 to follow the target. However, instead, the command generation unit 46 may generate a command to stop the endoscope 2 from following all treatment instruments 61, 62. In this case, the endoscope 2 determines whether to follow the movement of each treatment instrument (first treatment instrument, third treatment instrument) 61, 62 based on the current position (first position, third position) of each treatment instrument 61, 62. Then, the determination unit 44 generates a first result determining whether the endoscope 2 follows the movement of treatment instrument 61, and a second result determining whether the endoscope 2 follows the movement of treatment instrument (third treatment instrument) 62.

[0045] If the first result indicates that the endoscope 2 should follow the instrument 61, and the second result indicates that the endoscope 2 should not follow the instrument, the command generation unit 46 generates a command for the endoscope 2 to follow the instrument 61. If the first result indicates that the endoscope 2 should not follow the instrument, and the second result indicates that the endoscope 2 should follow the instrument 62, the command generation unit 46 generates a command for the endoscope 2 to follow the instrument 62. In this way, when there is only one candidate for the object to be followed, the control device 4 controls the moving device 3 to make the endoscope 2 follow the object to be followed.

[0046] If both the first and second results indicate that tracking is to be performed, or if both the first and second results indicate that tracking is not to be performed, the command generation unit 46 generates a command for the endoscope 2 to stop tracking the treatment instruments 61 and 62. In this way, if there are multiple candidates for the tracking target, or if there are no candidates, the control device 4 controls the moving device 3 to stop the endoscope 2 from tracking the tracking target.

[0047] In the above embodiment, the control device 4 may automatically return the target to its original position after determining that there is no target to follow. For example, as shown in Figure 9, if the tracking instrument 61 disappears from the endoscopic image D due to instrument replacement or other reasons, or if the deviation d of the posture of the tracking instrument 61 temporarily exceeds a threshold for any reason, the control device 4 determines in step S7 that there is no tracking instrument, and loses track of the tracking instrument. In this case, after step S7, the control device 4 repeats steps S2 to S7 using the next endoscopic image D. The control device 4 repeats steps S2 to S7 until it finds a tracking instrument 63 whose deviation d is below the threshold, and restores the tracking instrument by setting the tracking instrument 63 whose deviation d is below the threshold as the tracking instrument. As a result, a tracking instrument 63 whose current posture is the same as or nearly the same as the posture of the tracking instrument 61 immediately before it was determined that there was no tracking instrument is automatically restored as the tracking instrument.

[0048] For example, when replacing the tracking instrument 61, the tracking instrument disappears from the endoscopic image D when the instrument 61 is removed from the body, and then reappears in the endoscopic image D when another instrument 63 is inserted into the body. Unless the endoscope 2 is moved during the replacement of instruments 61 and 63, the posture of the tracking instrument does not change before and after it disappears. Therefore, even after losing sight of the tracking instrument, the control device 4 can recognize the tracking instrument again based on the current posture of the instrument. This eliminates the need to reset the tracking instrument when replacing instruments, etc.

[0049] In the above embodiment, the storage unit 45 updates the posture of the object to be tracked each time the object to be tracked is determined. Alternatively, the storage unit 45 may store the posture of the object to be tracked at multiple time points by adding the current posture to past postures. In this case, the determination unit 44 may use multiple past postures of the target at multiple points in the past to determine which target to follow. For example, the determination unit 44 may calculate the deviation d of the current posture for each of the multiple past postures and determine which target to follow based on the multiple deviations d.

[0050] (Second Embodiment) Next, an endoscope system according to a second embodiment of the present invention will be described. The endoscope system 1 according to this embodiment differs from the first embodiment in that it determines the target to track based on the current position in addition to the current posture of the treatment instruments 61 and 62. In this embodiment, the configurations that differ from the first embodiment will be described, and the same reference numerals will be used for the configurations common to the first embodiment, and their description will be omitted. Similar to the first embodiment, the endoscope system 1 according to this embodiment comprises an endoscope 2, a mobile device 3, a control device 4, and a display device 5.

[0051] As shown in Figure 10, the calculation unit 43 calculates the current position of each treatment instrument 61, 62 in addition to the current posture from the current endoscopic image D (step S14). The positions of the treatment tools 61 and 62 are, for example, the coordinates of at least one feature point of the area of each treatment tool 61 and 62 recognized by the recognition unit 42. Examples of feature points are, as shown in FIG. 11, the tip a or the base end b of the treatment tool 61, the tip c of the shaft 61a, the center of gravity g of the treatment tool 61, or the corners e and f of the rectangle surrounding the treatment tool area.

[0052] The determination unit 44 acquires the past posture and past position (second position) of the tracking target from the storage unit 45 (step S15). The past posture and position acquired at this time are the posture and position of the tracking target in the past endoscopic image D, and as will be described later, they are the posture and position stored in the storage unit 45 in step S19 a predetermined time before the current posture and current position. Next, the determination unit 44 determines whether the endoscope 2 follows the movement of each of the treatment tools 61 and 62 based on the current posture and current position of the treatment tools 61 and 62.

[0053] Specifically, the determination unit 44 compares the current posture and current position (first position) of the treatment tool 61 with the past posture and past position of the tracking target 61, respectively. When the current posture is the same as or substantially the same as the past posture of the tracking target, and the current position is the same as or substantially the same as the past position of the tracking target, the determination unit 44 determines that the endoscope 2 follows the movement of the treatment tool 61, that is, determines that the treatment tool 61 is the tracking target (step S16). On the other hand, when the current posture is not the same as or substantially the same as the past posture of the tracking target, and / or the current position is not the same as or substantially the same as the past position of the tracking target, the determination unit 44 determines that the endoscope 2 does not follow the movement of the treatment tool 61 (step S16). For the other treatment tool 62, the determination unit 44 makes the same determination.

[0054] For example, the determination unit 44 calculates the deviation d' from the following formula (2), and determines the treatment tool for which the deviation d' is less than or equal to a predetermined threshold value and is the minimum as the tracking target. The deviation d' is the deviation of the current position p t-i from the past position p t and the deviation of the current posture q t-i from the past posture qt It is the deviation of and the sum of .

number

[0055] The memory unit 45 stores the current posture and current position of the treatment instrument 61 that the judgment unit 44 has determined to be the target of tracking (step S19). According to this embodiment, the target to be tracked is determined based on the current posture and position of the treatment tools 61 and 62. This further improves the accuracy of target recognition.

[0056] In this embodiment as well, the control device 4 may automatically restore the tracking target after determining that there is no tracking target. In this case, after step S7, the control device 4 repeats steps S2 to S7 until a treatment instrument with a deviation d' below the threshold is found, and restores the tracking target by setting the treatment instrument with a deviation d' below the threshold as the tracking target. As a result, a treatment instrument whose current posture and position are the same as or approximately the same as the posture and position of the tracking target immediately before it was determined that there was no tracking target is restored as the tracking target. For example, as long as the endoscope is not moved during instrument replacement, the posture and position of the tracked object do not change before and after the tracked object disappears from view. Therefore, even after losing sight of the tracked object, the control device 4 can recognize the tracked object again based on the posture and position of the instrument.

[0057] (Third embodiment) Next, an endoscope system according to a third embodiment of the present invention will be described. The endoscope system 1 according to this embodiment differs from the first embodiment in that it determines whether the endoscope 2 follows each treatment instrument 61, 62 based on the current position of each treatment instrument 61, 62 and the past position of the target being followed, as well as the pivot points P1, P2 of each treatment instrument 61, 62. In this embodiment, configurations different from the first and second embodiments will be described, and configurations common to the first and second embodiments will be denoted by the same reference numerals and their description will be omitted. Similar to the first embodiment, the endoscope system 1 according to this embodiment comprises an endoscope 2, a mobile device 3, a control device 4, and a display device 5.

[0058] In this embodiment, the positions of the pivot points P1 and P2 of each of the treatment instruments 61 and 62 are set in the control device 4 and stored in the storage unit 45. For example, before inserting the endoscope 2 into the body, any operator, such as the surgeon, can teach the control device 4 the position of pivot point P1 by pressing a setting button (not shown) with the tip of the endoscope 2 positioned at pivot point P1. The control device 4 calculates the position of the tip of the endoscope 2 in the world coordinate system from the angles of each joint 3b of the robot arm 3a when the setting button is pressed, and sets the calculated position of the tip of the endoscope 2 to the position of pivot point P1. The world coordinate system is a coordinate system fixed to the base end of the robot arm 3a. Next, the control device 4 projects each pivot point P1 and P2 onto the image plane of the endoscope image D by coordinate transformation and calculates the positions of each pivot point P1 and P2 in the image coordinate system of the endoscope image D.

[0059] Similar to the first embodiment, the determination unit 44 determines whether the endoscope 2 will follow the movement of each treatment instrument 61, 62 based on the current position of each treatment instrument 61, 62 and the past position of the target to be followed, thereby determining the target to follow. If, for example, the deviation d of all treatment instruments 61, 62 is greater than a threshold and the determination unit 44 determines that there are no treatment instruments corresponding to the target to follow in the endoscope image D, the determination unit 44 then obtains pivot points P1, P2 from the storage unit 45 and determines whether the endoscope 2 will follow each treatment instrument 61, 62 based on the positions of the pivot points (first pivot points) P1, P2 of each treatment instrument 61, 62.

[0060] Specifically, as shown in Figures 12A and 12B, the determination unit 44 calculates a first vector V1 representing the orientation of each treatment instrument 61, 62 in the current endoscopic image D, and a second vector V2 connecting the pivot point (second pivot point) P1 of the tracking target and the treatment instrument. The first vector V1 is, for example, a vector connecting the base end b and the center of gravity g of each treatment instrument 61, 62 in the endoscopic image D. The second vector V2 is, for example, a vector connecting the pivot point P1 of the tracking target on the image plane of the endoscopic image D and the center of gravity g of each treatment instrument 61, 62.

[0061] In the case of the treatment tool 61 that is being followed, the first vector V1 is parallel to the second vector V2, and in the case of the treatment tool 62 that is not being followed, the first vector V1 forms an angle with respect to the second vector V2. The determination unit 44 calculates the angle θ between the first vector V1 and the second vector V2 for the treatment instrument 61. If the angle θ is less than or equal to a predetermined threshold, the determination unit 44 determines that the endoscope 2 follows the movement of the treatment instrument 61, and if the angle θ is greater than the predetermined threshold, the determination unit 44 determines that the endoscope 2 does not follow the movement of the treatment instrument 61. Similarly, the determination unit 44 calculates the angle θ between the first vector V1 and the second vector V2 for the other treatment instrument 62. The determination unit 44 determines that the endoscope 2 follows the movement of the treatment instrument 62 if the angle θ is less than or equal to a predetermined threshold, and determines that the endoscope 2 does not follow the movement of the treatment instrument 62 if the angle θ is greater than the predetermined threshold.

[0062] A treatment tool 61 whose angle θ is below a predetermined threshold is a candidate for tracking. Therefore, the determination unit 44 extracts a candidate for tracking from among the multiple treatment tools 61 and 62 based on the above determination. If there are no treatment instruments whose angle θ is below a predetermined threshold, the determination unit 44 determines that there are no treatment instruments corresponding to the tracking target in the endoscopic image D, and notifies the target setting unit 41 of this determination.

[0063] Next, the judgment unit 44 calculates a third vector V3 connecting each pivot point P2 of each treatment instrument 62 other than the target of tracking to the treatment instrument 62 for each treatment instrument 61, 62 in the current endoscopic image D, and calculates the angle θ' between the first vector V1 and the third vector V3. Then, it excludes treatment instruments from the candidates whose angle θ' is below a threshold, and determines that the remaining candidates are targets for tracking.

[0064] As shown in Figure 12A, when the pivot point P1 of the tracking target 61 is far from the pivot point P2 of the other treatment tool 62, the angle θ of the other treatment tool 62 will be greater than a predetermined threshold, so the other treatment tool 62 will not be mistakenly recognized as the tracking target. On the other hand, as shown in Figure 12B, when the pivot point P2 of the other treatment tool 62 is close to the pivot point P1 of the tracking target 61, the angle θ of the other treatment tool 62 will also be below a predetermined threshold, and there is a possibility that the other treatment tool 62 will be mistakenly recognized as the tracking target. As described above, by excluding treatment devices where both angles θ and θ' are below the threshold from the list of candidates to track, misidentification of the tracked object can be prevented.

[0065] Next, we will explain the operation of Endoscopic System 1. As shown in Figure 13, the positions of pivot points P1 and P2 in the image coordinate system of each treatment instrument 61 and 62 are set in the control device 4 (step S21), and then steps S1 to S7 are executed in the same manner as in the first embodiment. If it is determined that there is no treatment instrument that corresponds to the tracking target (NO in step S7), the determination unit 44 obtains the pivot points P1 and P2 from the storage unit 45 and determines the tracking target based on the pivot points P1 and P2 (step S22).

[0066] Specifically, the decision unit 44 calculates a first vector V1 representing the current posture of the treatment tool 61 and a second vector V2 from the pivot point P1 to the treatment tool 61. It then calculates the angle θ between the two vectors V1 and V2, and if the angle θ is less than or equal to a threshold, it selects the treatment tool 61 as a candidate to follow. Similarly, the decision unit 44 calculates the angle θ for the other treatment tools 62, and if the angle θ is less than or equal to a threshold, it selects the treatment tool 62 as a candidate to follow. Next, for each treatment tool selected as a candidate, the decision unit 44 calculates a third vector V3 connecting each pivot point P2 of the treatment tools 62 other than the target to follow to the treatment tool, and calculates the angle θ' between the first vector V1 and the third vector V3. The decision unit 44 then excludes treatment tools whose angle θ' is less than or equal to a threshold from the candidates, determines the remaining candidates to be followed (YES in step S23), and decides the remaining candidates to be followed (step S8). If it is determined in step S23 that there is no treatment device that corresponds to the target to follow, the control device 4 will automatically return the target to follow by repeating steps S2 to S8, S22, and S23 until a target to follow is found.

[0067] Thus, according to this embodiment, if the tracking target cannot be determined based on the current posture of the treatment tools 61 and 62, the tracking target is subsequently determined based on the pivot points P1 and P2. This reduces the likelihood of the control device 4 losing track of the tracking target. In this embodiment, the determination unit 44 may, in step S22, not narrow down the candidates based on the angle θ', and may determine that the treatment instrument whose angle θ is less than or equal to a predetermined threshold and is the minimum is the target for tracking. In this case, in step S21, only the position of the pivot point P1 of the target for tracking may be set.

[0068] In this embodiment, the determination unit 44 determines the target to follow based on the current and past attitudes, but instead, the determination unit 44 may determine the target to follow based on the current attitude and pivot points P1 and P2. Alternatively, the determination unit 44 may narrow down the candidates for the target to follow based on pivot points P1 and P2. In other words, as shown in Figure 14A, in step S22', the determination unit 44 selects a treatment instrument whose angle θ is less than or equal to a predetermined threshold as a candidate for tracking, in the same manner as in step S22.

[0069] If there are multiple candidates for the tracked object (multiple in step S7'), the decision unit 44 narrows down the candidates (step S24). Specifically, as shown in Figure 14B, in step S24, the decision unit 44 selects one treatment instrument from among the candidates as the tracked object j (step S131), and extracts candidates for the tracked object j based on the posture and pivot point of the tracked object j (step S141). Specifically, the decision unit 44 calculates a first vector representing the posture of the tracked object j and a second vector connecting the pivot point of the tracked object j to each treatment instrument, and selects a treatment instrument as a candidate for the tracked object j if the angle θ between the first vector and the second vector is less than or equal to a predetermined threshold.

[0070] If there is one or more candidates, the determination unit 44 determines whether the candidate for tracking target j is included in the list of candidates to follow (step S137), and if it is included in the list of candidates to follow, it removes the candidate for tracking target j from the list of candidates to follow (step S138). The determination unit 44 selects each candidate treatment tool to be followed as the tracking target j in order, and executes the processes in steps S132 to S138 (steps S139, S140). This narrows down the candidates for tracking targets. If the number of candidates for tracking becomes zero before all of the candidate treatment tools for tracking are selected as tracking target j (NO in step S142), the determination unit 44 may terminate the candidate narrowing down at that point.

[0071] In this embodiment, the control device 4 may determine whether the endoscope 2 follows the movement of each treatment instrument 61, 62 based on the current posture, current position and pivot points P1, P2 of each treatment instrument 61, 62 and the past posture and pivot point P1 of the target to be tracked. That is, as shown in Figure 15, the control device 4 may perform steps S14 to S16 of the second embodiment instead of steps S4 to S6. Furthermore, in this embodiment, the positions of pivot points P1 and P2 are set prior to tracking the target; however, as shown in Figure 15, the determination unit 44 may estimate the positions of pivot points P1 and P2 from the current endoscopic image D.

[0072] In the modified example shown in Figure 15, after the target to be tracked is determined (step S8), the current positions of the pivot points P1 and P2 of each treatment instrument 61 and 62 are estimated using the current endoscopic image D (step S31), and the positions of the pivot points P1 and P2 are stored in the memory unit 45 (step S32). Then, after it is determined in step S7 that there is no treatment instrument to be tracked, the past positions of the pivot points P1 and P2 already stored in the memory unit 45 are obtained (step S33), and the target to be tracked is determined using the past positions of the pivot points P1 and P2 (step S22). According to this modified example, the step of setting the position of the pivot point in step S21 can be made unnecessary.

[0073] In step S31, the determination unit 44 uses the position of the treatment instrument 61 in the endoscopic image D, the depth information of the treatment instrument 61, and the position and orientation of the endoscope 2 to estimate the position of the pivot point P1 of the treatment instrument 61 in the current endoscopic image D. Similarly, the determination unit 44 also estimates the pivot point P2 of the other treatment instrument 62 in the current endoscopic image D.

[0074] Specifically, as shown in Figure 16, the decision unit 44 calculates the two-dimensional positions of the tip a and base b of the treatment instrument 61 in the image coordinate system from the current endoscopic image D. The decision unit 44 also acquires the depth-direction positions of the tip a and base b of the treatment instrument 61 using a three-dimensional measurement means such as a three-dimensional camera. Then, using the depth-direction positions of the tip a and base b, the decision unit 44 converts the two-dimensional positions of the tip a and base b in the image coordinate system into three-dimensional positions of the tip and base in the camera coordinate system. Next, the decision unit 44 calculates the position and orientation of the tip of the endoscope 2 in the world coordinate system from the angles of each joint 3b of the robot arm 3a, and uses the position and orientation of the tip of the endoscope 2 in the world coordinate system to convert the three-dimensional positions of the tip a and base b in the camera coordinate system into positions of the tip a and base b in the world coordinate system. The determination unit 44 then calculates the longitudinal axis G of the treatment tool 61 connecting the tip a and base b in the world coordinate system, retrieves past longitudinal axis Gs from the storage unit 45, and calculates the intersection of the two longitudinal axes Gs as the pivot point P1. The position of the calculated pivot point P1 is stored in the storage unit 45.

[0075] (Fourth Embodiment) Next, an endoscope system according to the fourth embodiment of the present invention will be described. The endoscope system 1 according to this embodiment differs from the first embodiment in that it does not determine the target to follow based on the deviation d of the posture of the treatment instruments 61 and 62, but rather determines whether the endoscope 2 follows the movement of each treatment instrument 61 and 62 based on the current posture and pivot points P1 and P2 of the treatment instruments 61 and 62. In this embodiment, configurations different from the first to third embodiments will be described, and configurations common to the first to third embodiments will be denoted by the same reference numerals and their description will be omitted. Similar to the first embodiment, the endoscope system 1 according to this embodiment comprises an endoscope 2, a mobile device 3, a control device 4, and a display device 5.

[0076] As shown in Figure 17, following step S4, the determination unit 44 determines the target to follow based on the positions of the pivot points P1 and P2 (step S22). Step S22 is as described in the third embodiment. As described above, according to this embodiment, a first vector V1 representing the current posture of each treatment instrument 61, 62 and a second vector V2 connecting the pivot point P1 of the tracking target and each treatment instrument 61, 62 are calculated, and whether or not each treatment instrument 61, 62 is the tracking target is determined based on the angle θ between vectors V1 and V2. Normally, the posture of the tracking target treatment instrument 61 and the pivot point P1 do not change rapidly. Also, the postures of multiple treatment instruments 61, 62 in the endoscopic image D are different from each other. Therefore, it is possible to accurately determine whether or not each treatment instrument 61, 62 is the tracking target based on the angle θ. In this embodiment as well, in step S22, instead of narrowing down the candidates based on the angle θ', the treatment instrument whose angle θ is below a predetermined threshold and is the minimum value may be determined to be the target for tracking.

[0077] In each of the above embodiments, the posture and position calculated by the calculation unit 43 may be superimposed on the endoscopic image D. For example, as shown in Figure 18, a line I representing the longitudinal axis of the treatment tool 61 as the posture of the treatment tool 61 to be tracked, and a dot J representing the base end b as the position of the treatment tool 61, may be superimposed on the treatment tool 61 to be tracked. With this configuration, if the tracking instrument 61 disappears from the endoscopic image D, the posture I and position J of the instrument 61 immediately before its disappearance are continuously displayed, making it easy to identify the tracking instrument based on its posture I and position J when it returns. [Explanation of Symbols]

[0078] 1 Endoscopy System 2 Endoscope 3. Mobile device 4. Control device 5 Display device 6.63 Treatment tools 61. Treatment tools (first treatment tool, second treatment tool) 62. Treatment tools (third treatment tool) P1 Pivot points (first pivot point, second pivot point) P2 Pivot Point

Claims

1. Endoscope and, A moving device for moving the endoscope, The system comprises a control device for controlling the endoscope and the mobile device, The control device is The endoscope receives the image it captures. The first posture of the first treatment instrument is calculated in the first image, which is the image received at any given time. The second posture of the second treatment instrument, which is the tracking target, is calculated in the second image, which is the image received a predetermined time before the first image. The difference in posture between the first posture and the second posture is calculated. By comparing the difference in posture with a threshold, it is determined whether or not the endoscope follows the movement of the first treatment instrument. An endoscope system that, when it determines that it should follow, generates a command to the moving device for the endoscope to follow the first treatment instrument.

2. The endoscope system according to claim 1, wherein if the control device determines that it is not following, the endoscope generates a command to the moving device to stop the endoscope from following the first treatment instrument.

3. The endoscopic system according to claim 1, wherein the aforementioned arbitrary time is the present.

4. The endoscopic system according to claim 1, wherein the second treatment instrument is the first treatment instrument.

5. The control device is The first position of the first treatment instrument is calculated, The second position of the second treatment instrument is calculated, The difference between the first position and the second position is calculated, The endoscope system according to claim 1, which determines whether or not the endoscope follows the movement of the first treatment instrument by comparing the difference in posture with the threshold and the difference in position with a further threshold.

6. The endoscopic system according to claim 5, wherein the second treatment instrument is the first treatment instrument.

7. The endoscopic system according to claim 1, wherein the first posture is the angle between the reference axis in the image and the first treatment instrument.

8. The endoscopic system according to claim 1, wherein the first posture is a longitudinal vector of the first treatment instrument in the image.

9. Endoscope and, A moving device for moving the endoscope, The system comprises a control device for controlling the endoscope and the mobile device, The control device is The endoscope receives the image it captures. The posture of the treatment instrument in the aforementioned image is calculated, The position of the pivot point of the aforementioned treatment device is obtained, The position of the base end of the aforementioned treatment instrument is calculated, A first vector is calculated that is parallel to a first line passing through the position of the base end of the first treatment instrument and an arbitrary position on the first treatment instrument. A second vector is calculated that is parallel to the second line passing through the pivot point of the second treatment tool, which is the target of tracking, and the aforementioned arbitrary position. When the angle between the first vector and the second vector is less than or equal to a predetermined threshold, the endoscope follows the movement of the first treatment instrument. An endoscope system that, when it determines that it should follow, generates a command to the moving device for the endoscope to follow the first treatment instrument.

10. A device for moving the endoscope, A method for operating an endoscope system comprising a control device for controlling the endoscope and the mobile device, The control device performs the steps of calculating the first posture of the first treatment instrument in the first image received from the endoscope at any given time, The control device performs the steps of calculating the second posture of the second treatment instrument, which is the tracking target, in the second image received from the endoscope a predetermined time before the first image, The control device performs the steps of calculating the difference in posture between the first posture and the second posture, The control device determines whether the endoscope follows the movement of the first treatment instrument by comparing the difference in posture with a threshold, A method for operating an endoscope system, comprising the step of, if the control device determines to follow, generating a command to a moving device for the endoscope to follow the first treatment instrument.

Images