Image processing device, image processing method, and image processing program

Abstract

The present invention provides an image processing device, an image processing method, and an image processing program that can quickly and accurately determine the tip position and direction of a treatment instrument, even when obstacles such as trockers positioned close to the instrument appear in the endoscopic image. [Solution] An image processing device 10 for processing endoscopic images of a body cavity taken by an endoscope 31 is provided, comprising a processor 1, the processor 1 detects a treatment instrument region in which a long treatment instrument 37 exists from the endoscopic image, sets a region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument 37 as a candidate treatment instrument base region S in the endoscopic image, sets a region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis as a candidate background region B, calculates the similarity between the color information of the candidate treatment instrument base region S and the color information of the candidate background region B, and estimates information regarding the orientation of the treatment instrument 37 based on the calculated similarity.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program. [Background technology]

[0002] Conventionally, in endoscopic surgery, a technique is known to track the tip of a surgical instrument by detecting its tip position and direction (see, for example, Patent Document 1). The tip detection device described in Patent Document 1 identifies the side of the image edge, which includes all four sides of the endoscopic image, that has the largest contact area with the segmentation mask indicating the region of the surgical instrument on the endoscopic image, as the root of the surgical instrument. Based on the identified root of the surgical instrument, the tip position and direction of the surgical instrument are then detected. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-30681 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the tip detection device described in Patent Document 1 may have issues where the segmentation mask does not come into contact with the edge of the image due to obstacles such as trockers appearing in the endoscopic image, which affects the detection of the tip of the treatment instrument. For example, when a treatment instrument is inserted into the abdominal cavity, the portion of the instrument inserted into the trocker may not be recognized as part of the segmentation mask. In such cases, there is a problem in that the tip position and direction of the treatment instrument cannot be accurately detected.

[0005] The present invention has been made in view of the above circumstances, and aims to provide an image processing device, an image processing method, and an image processing program that can quickly and accurately determine the tip position and direction of a treatment instrument even when an obstacle such as a trocker placed in close proximity to the treatment instrument is visible in the endoscopic image. [Means for solving the problem]

[0006] To achieve the above objective, the present invention provides the following means. A first aspect of the present invention is an image processing apparatus for processing endoscopic images of a body cavity taken by an endoscope, comprising a processor, the processor detecting a treatment instrument region in which a long treatment instrument exists from the endoscopic image, setting a region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument as a candidate treatment instrument base region, and setting a region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis as a candidate background region, calculating the similarity between the color information of the candidate treatment instrument base region and the color information of the candidate background region, and estimating information regarding the orientation of the treatment instrument based on the calculated similarity.

[0007] A second aspect of the present invention is an image processing method for processing an endoscopic image of a body cavity taken by an endoscope, comprising: detecting a treatment instrument region in which a long treatment instrument exists from the endoscopic image; setting a region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument as a candidate treatment instrument base region in the endoscopic image, and setting a region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis as a candidate background region; calculating the similarity between the color information of the candidate treatment instrument base region and the color information of the candidate background region; and estimating information regarding the orientation of the treatment instrument based on the calculated similarity.

[0008] A third aspect of the present invention is an image processing program that causes a computer to process an endoscopic image of a body cavity taken by an endoscope, the program causing the computer to perform the following actions: detect a treatment instrument region in which a long treatment instrument exists from the endoscopic image; set a region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument as a candidate treatment instrument base region in the endoscopic image, and set a region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis as a candidate background region; calculate the similarity between the color information of the candidate treatment instrument base region and the color information of the candidate background region; and estimate information regarding the orientation of the treatment instrument based on the calculated similarity. [Effects of the Invention]

[0009] The image processing apparatus, image processing method, and image processing program according to the present invention have the effect of quickly and accurately determining the tip position and direction of a treatment instrument, even when an obstacle such as a trocker placed in close proximity to the treatment instrument is visible in the endoscopic image. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing an endoscope system according to the first embodiment of the present invention. [Figure 2] Figure 1 shows an example of an endoscopic image obtained using an endoscope. [Figure 3] This figure shows an example of a treatment instrument mask image, obtained by extracting only the treatment instrument area from an endoscopic image in Figure 2. [Figure 4] This figure shows an example of a background mask image from which the halation portion has been removed from the treatment device mask image in Figure 3. [Figure 5] This figure shows an example of a rectangular region identified in the background mask image of Figure 4. [Figure 6] This figure shows an example of a candidate background region selected from the image in Figure 5. [Figure 7] Figure 3 shows an example of a rectangular region identified in the image of the treatment device mask. [Figure 8] It is a diagram showing an example of a treatment tool proximal candidate region selected from the image of FIG. 7. [Figure 9] It is a diagram explaining the relationship between the rectangular region and the treatment tool proximal candidate region. [Figure 10] It is a diagram showing an example of an endoscopic image divided into a plurality of small regions. [Figure 11] It is a diagram showing a small region corresponding to the background candidate region in the endoscopic image of FIG. 10. [Figure 12] It is a diagram showing an example of a treatment tool proximal candidate region on an endoscopic image. [Figure 13] It is a graph showing the relationship between the cosine similarity when the treatment tool proximal candidate region is determined as the tip of the treatment tool and the cosine similarity when it is determined as the base end of the treatment tool. [Figure 14] It is a diagram showing an example of a treatment tool mask image explaining a method for detecting the tip position of a treatment tool. [Figure 15] It is a diagram showing an example of a distance measurement candidate point on a treatment tool mask. [Figure 16] It is a diagram showing an example of the tip position of a treatment tool. [Figure 17] It is a flowchart explaining the image processing method according to the first embodiment of the present invention. [Figure 18] It is a diagram showing an example of an endoscopic image in which a treatment tool is reflected. [Figure 19] It is a diagram showing a treatment tool mask image obtained by extracting only the treatment tool region from the endoscopic image of FIG. 18. [Figure 20] It is a diagram showing an example of a background candidate region set in the treatment tool mask image of FIG. 20. [Figure 21] It is a diagram showing an example of a treatment tool proximal candidate region set in the treatment tool mask image of FIG. 20. [Figure 22] It is a diagram showing an example of a treatment tool proximal candidate region determined as the base end of the treatment tool. [Figure 23] It is a diagram showing an example of a treatment tool proximal candidate region determined as the tip of the treatment tool. [Figure 24]This figure shows an example of a situation where neither the tip nor the base of the instrument is in contact with the edge of the endoscopic image. [Figure 25] This figure shows another example where neither the tip nor the base of the instrument is in contact with the edge of the endoscopic image. [Figure 26] This figure shows an example of a situation where the proximal end of the treatment instrument is in contact with the edge of the endoscopic image. [Figure 27] This figure shows an example of a situation where both the tip and proximal end of the treatment instrument are in contact with the edge of the endoscopic image. [Figure 28] This figure shows an example of a situation where the tip of the treatment instrument is in contact with the edge of the endoscopic image. [Figure 29] This figure shows another example where the tip of the instrument is in contact with the edge of the endoscopic image. [Figure 30] This is a flowchart illustrating an image processing method according to a modified example of the first embodiment of the present invention. [Figure 31] This figure shows an example of an endoscopic image where the tip of the instrument is in contact with the edge. [Figure 32] This figure shows a treatment instrument mask image, extracted from the endoscopic image in Figure 31, showing only the treatment instrument area. [Figure 33] Figure 32 shows an example of a rectangular region identified in the image of the treatment device mask. [Figure 34] This figure shows an example of a background candidate region to be set in the treatment device mask image in Figure 33. [Figure 35] Figure 33 shows an example of a candidate proximal end region for the treatment device in the treatment device mask image. [Figure 36] This figure shows an example of a candidate area for the proximal end of a treatment instrument, which is determined to be the tip of the treatment instrument. [Figure 37] This is a schematic diagram showing an endoscope system according to the second embodiment. [Figure 38] This diagram illustrates the positional relationship between the tip of the treatment instrument and the endoscope. [Figure 39] This figure shows an example of an endoscopic image in which a Trocca is visible. [Figure 40]This is a flowchart illustrating the control method of the endoscope system according to the second embodiment. [Figure 41] This figure shows an example of an endoscopic image used to illustrate a method for detecting troccas using image processing. [Figure 42] This figure shows an example of a depth map generated based on the endoscopic image in Figure 41. [Figure 43] This figure shows an example of a binarized image generated from the depth map in Figure 42. [Figure 44] This graph shows an example of an RGB histogram of the region being visualized by the endoscopic trocker. [Figure 45] This graph shows an example of an RGB histogram for a region other than the area being visualized by the endoscopic trocker. [Figure 46] This is a flowchart illustrating the control method of the endoscope system according to the first modified example of the second embodiment. [Figure 47] This figure shows an example of a judgment result displayed on an endoscopic image. [Figure 48] This figure shows another example of a judgment result displayed on an endoscopic image. [Figure 49] This figure shows the endoscope inserted to a position where the endoscopic trocker is no longer visible in the endoscopic image. [Figure 50] This is a flowchart illustrating the control method for an endoscope system according to a second modified example of the second embodiment. [Figure 51] This is a flowchart illustrating a control method for an endoscope system according to yet another modification of the second embodiment. [Figure 52] This is a schematic diagram showing an endoscope system according to the third embodiment. [Figure 53] This diagram illustrates how to bring the endoscope closer to the treatment instrument from an overhead view. [Figure 54] This diagram illustrates the operation of the endoscope to keep the tip of the treatment instrument centered in the endoscopic image. [Figure 55] This diagram illustrates how to set a high speed for the forward and backward movement of the endoscope when there is a large distance between the tip of the treatment instrument and the endoscope. [Figure 56] This diagram illustrates how the speed of the endoscope's movement in the forward and backward directions is reduced when the distance between the tip of the treatment instrument and the endoscope decreases. [Figure 57] This graph illustrates the relationship between the distance from the endoscope to the object and the speed of the endoscope. [Figure 58] This diagram illustrates how the planar movement speed of the endoscope is changed according to the distance between the tip of the treatment instrument and the endoscope. [Modes for carrying out the invention]

[0011] [First Embodiment] An image processing apparatus, an image processing method, and an image processing program according to the first embodiment of the present invention will be described below with reference to the drawings. The image processing device 10 according to this embodiment is applied to an endoscope system 100, as shown in Figure 1. The endoscopic system 100 is used in surgery, such as laparoscopic surgery, in which an endoscope 31 and a treatment instrument 37 are inserted into the body of a patient, and the treatment instrument 37 is used to treat a target area such as a diseased area while the treatment instrument 37 is observed by the endoscope 31.

[0012] The treatment instrument 37 is a long member, having a tip 37a at one end along its longitudinal axis shaped according to the purpose or use of the treatment, and a base end (not shown) at the other end which is a handle or operating part for the operator to grasp. The treatment instrument 37 is inserted into the subject, for example, into the abdominal cavity, via a treatment instrument trocker 39 which is fixed in a penetrating state to the surface of the patient's body, with the tip 37a inserted. The treatment instrument trocker 39 is a cylindrical instrument that is inserted into the subject through a hole formed in the body wall, and is pivotable with the position of the hole in the body wall as a fulcrum.

[0013] The endoscopic system 100 includes an endoscope 31 for photographing the inside of a body cavity, a video system 33 for generating endoscopic images from image information acquired by the endoscope 31, a monitor 35 such as a liquid crystal display for displaying the endoscopic images generated by the video system 33, and an image processing device 10 for processing the endoscopic images.

[0014] The monitor 35 receives endoscopic images generated by the video system 33 sequentially, so that the endoscopic images of the body cavity taken by the endoscope 31 are displayed in real time.

[0015] The image processing device 10 is comprised of any computer, such as a personal computer. The image processing device 10 includes at least one processor 1, such as a central processing unit, a storage unit 3, a memory 5, and a user interface 7.

[0016] The storage unit 3 is a computer-readable non-temporary storage medium, such as a known magnetic disk, optical disk, or flash memory. The storage unit 3 stores an image processing program that causes the processor 1 to execute the image processing method according to this embodiment.

[0017] Memory 5 is RAM (random It consists of volatile memory devices such as access memory and is used as a workspace for processor 1. User interface 7 has input devices such as a mouse, keyboard, and touch panel, and accepts user operation of these input devices.

[0018] The processor 1 includes, as functional units, a treatment tool area detection unit 11, a treatment tool mask generation unit 13, and a treatment tool direction detection unit 15. The instrument region detection unit 11 uses deep learning, such as a CNN (Convolutional Neural Network), to detect the instrument region where the instrument 37 is located in the endoscopic image shown in Figure 2, which is sent from the video system 33. The endoscopic image may be represented in either RGB (Red, Green, Blue) format or HSV (Hue, Saturation, Brightness) format for the color information of each pixel. The detected instrument region information is sent to the instrument mask generation unit 13 along with the endoscopic image. Note that the instrument region does not include the region where the instrument trocker 39 is located.

[0019] The instrument mask generation unit 13 generates an instrument mask image, as shown in Figure 3, which extracts only the instrument region, based on the endoscopic image and instrument region information sent from the instrument region detection unit 11. The instrument mask image is an image in which the areas other than the instrument region on the endoscopic image are filled with a black or white background, while only the instrument region is displayed in a different color from the background. The instrument region portion on the instrument mask image is called instrument mask A. The instrument mask image is sent to the instrument direction detection unit 15.

[0020] The treatment tool direction detection unit 15 includes a background candidate region determination unit 17, a treatment tool base end candidate region determination unit 19, a first color information extraction unit 21, a second color information extraction unit 23, a similarity calculation unit 25, and an appearance direction determination unit 27.

[0021] The background candidate region determination unit 17 generates a background mask image as shown in Figure 4 by removing the treatment tool mask A and the halation portion from the treatment tool mask image sent from the treatment tool mask generation unit 13. Removal of the halation portion is not mandatory, and it may be left in place.

[0022] The background candidate region determination unit 17 identifies the smallest rectangular region F containing the treatment instrument region by rectangular fitting in the generated background mask image or treatment instrument mask image, as shown in Figure 5. The background candidate region determination unit 17 also sets the region adjacent to the rectangular region F in a direction perpendicular to the longitudinal axis of the treatment instrument 37 as the background candidate region B, as shown in Figure 6. The method for selecting the background candidate region B is, for example, to detect the short side of the rectangular region F, and then to enlarge the rectangular region F in a direction parallel to the short side, as shown in Figure 5. Then, as shown in Figure 6, the enlarged portion of the rectangular region F is selected as the background candidate region B.

[0023] The proximal end candidate region determination unit 19 identifies the smallest rectangular region F containing the treatment instrument region by rectangular fitting in the treatment instrument mask image sent from the treatment instrument mask generation unit 13, as shown in Figure 7. Then, as shown in Figure 8, it sets the region adjacent to the rectangular region F in the longitudinal axis direction of the treatment instrument 37 as the proximal end candidate region S. The method for selecting the proximal end candidate region S is, for example, to detect the long side of the rectangular region F, and then calculate a straight line G that passes through the center of the rectangular region F and is parallel to the long side of the rectangular region F, as shown in Figure 8. Then, the proximal end candidate region S is set on the extension of the calculated straight line G and near the short side of the rectangular region F.

[0024] The candidate proximal region S of the treatment instrument should preferably have the following relationship with respect to the rectangular region F, as shown in Figure 9. Specifically, the size H1 in the direction of the straight line G in the candidate proximal region S should preferably be 1 to 3 times the size H2 in the direction perpendicular to the straight line G in the rectangular region F. Furthermore, the size H3 in the direction perpendicular to the straight line G in the candidate proximal region S should preferably be the same as or approximately the same size as the size H2 in the direction perpendicular to the straight line G in the rectangular region F. In addition, the distance H4 between the candidate proximal region S and the rectangular region F in the direction of the straight line G should preferably be 0, meaning that the candidate proximal region S and the rectangular region are tangent to each other in the direction of the straight line G.

[0025] As shown in Figure 10, the first color information extraction unit 21 divides the endoscopic image displayed on the monitor 35 into multiple sub-regions and then generates an HSV histogram for each sub-region. Then, as shown in Figure 11, the first color information extraction unit 21 generates an HSV histogram for each sub-region corresponding to the background candidate region B. It is desirable that each sub-region be within the range of 1 / 20 to 1 / 30 of the endoscopic image size.

[0026] As shown in Figure 12, the second color information extraction unit 23 generates an HSV histogram for each candidate treatment instrument base end region S of the endoscopic image displayed on the monitor 35.

[0027] The similarity calculation unit 25 calculates the cosine similarity between the HSV histogram of each sub-region of the background candidate region B and the HSV histogram of the treatment tool base end candidate region S using the following formula.

number

[0028] Using the above formula, for example, the cosine similarity C shown in Figure 13 can be obtained. j The following is calculated. In Figure 13, similarity C1 is the cosine similarity of the HSV histogram of one treatment tool base-end candidate region S to the HSV histogram of the background candidate region B shown in Figure 12, and similarity C2 is the cosine similarity of the HSV histogram of the other treatment tool base-end candidate region S to the HSV histogram of the background candidate region B shown in Figure 12.

[0029] The direction determination unit 27 estimates information regarding the orientation of the treatment tool 37 based on the calculated cosine similarity. For example, the direction determination unit 27 determines that a candidate treatment tool base end region S whose cosine similarity is equal to or greater than a predetermined similarity threshold is the tip 37a of the treatment tool 37. It also determines that a candidate treatment tool base end region S whose cosine similarity is less than the predetermined similarity threshold is the base end of the treatment tool 37.

[0030] In other words, since the color of the instrument trocker 39 is usually different from the color of the inner wall of the body cavity which is the background, the instrument proximal end candidate region S, in which the difference in color information with the background candidate region B is small, is considered to be the background region on the tip 37a side of the instrument 37. On the other hand, the instrument proximal end candidate region S, in which the difference in color information with the background candidate region B is large, is considered to be the region of the instrument trocker 39 into which the proximal end of the instrument 37 is inserted.

[0031] The direction determination unit 27 also estimates the direction of the treatment tool 37 based on the positional relationship between the tip 37a and the base of the treatment tool 37. The direction of the treatment tool 37 is, for example, the direction in which the tip 37a points relative to the base of the treatment tool 37. In Figure 1, the direction of the treatment tool 37 is indicated by arrow α. Arrow α is not actually displayed in the image, but is illustrated in the image for explanatory purposes. The same applies to Figures 18 and 24-26.

[0032] An example of a method for detecting the position of the tip 37a of the treatment instrument 37 on an endoscopic image is described below. First, after detecting the treatment instrument area on the endoscopic image, a treatment instrument mask image is created as shown in Figure 14. Next, the centroid position I of the treatment instrument mask A in the treatment instrument mask image is calculated. Then, the principal axis vector M connecting the calculated centroid position I and the proximal end position L of the treatment instrument mask A is calculated. Next, as shown in Figure 15, the tip of the treatment instrument mask A is obtained, and the obtained tip is designated as the distance measurement candidate point N. As shown in Figure 15, if the tip of the treatment instrument mask A is forked with equal lengths, the tip of either one of the forks can be designated as the distance measurement candidate point N.

[0033] Next, as shown in Figure 16, a perpendicular line Q is drawn that passes through the candidate distance measurement point N and is perpendicular to the principal axis vector M. The intersection point of the principal axis vector M and the perpendicular line Q is defined as position R of the tip 37a of the treatment tool 37. In the examples shown in Figures 14 to 16, a treatment tool 37 with a bifurcated tip is used as an example for explanation, but the same method can be applied to a treatment tool 37 with a single tip 37a. In this case, the candidate distance measurement point N may be located on the principal axis vector M.

[0034] Next, we will describe the image processing method executed by the image processing device 10 and the image processing program for executing the image processing method. The image processing method according to this embodiment includes, as shown in the flowchart of Figure 17, a step SA2 for detecting a treatment instrument region from an endoscope image, a step SA4 for setting a background candidate region B and a step SA5 for setting a treatment instrument proximal end candidate region S in the endoscope image, a step SA9 for calculating the similarity between the color information of the treatment instrument proximal end candidate region S and the color information of the background candidate region B, and steps SA11 and SA12 for estimating information regarding the orientation of the treatment instrument 37 based on the calculated similarity. In step SA11, the treatment instrument proximal end candidate region S whose calculated similarity is equal to or greater than a predetermined threshold is determined to be the tip 37a of the treatment instrument 37. In step SA12, the orientation of the treatment instrument 37 is calculated based on the tip 37a of the treatment instrument 37.

[0035] The image processing program according to this embodiment causes the processor 1 to execute steps SA1 to SA12. By the processor 1 executing the processing according to the image processing program, the above-mentioned functions of the treatment instrument area detection unit 11, the treatment instrument mask generation unit 13, and the treatment instrument direction detection unit 15 are realized. Hereinafter, each of the processes performed by these functional units will be described as processes performed by the processor 1.

[0036] First, when the treatment instrument 37 is photographed by the endoscope 31 in the body cavity, the processor 1 receives an endoscopic image showing the treatment instrument 37 as shown in FIG. 18 (step SA1). Next, after the treatment instrument area is detected on the endoscopic image by the processor 1, a treatment instrument mask image with the treatment instrument area extracted is generated as shown in FIG. 19 (step SA2).

[0037] Subsequently, after the rectangular area F including the treatment instrument area is specified by rectangular fitting in the treatment instrument mask image by the processor 1 as shown in FIG. 20 (step SA3), a background candidate area B adjacent to the rectangular area F in the direction orthogonal to its longitudinal axis is set (step SA4). Also, as shown in FIG. 21, a treatment instrument proximal end candidate area S adjacent to the rectangular area F in its longitudinal axis direction is set (step SA5).

[0038] Subsequently, after the endoscopic image is divided into a plurality of small areas by the processor 1, an HSV histogram for each small area is generated (step SA6). Next, an HSV histogram B for each small area corresponding to the background candidate area B i is generated (step SA7), and an HSV histogram S for the treatment instrument proximal end candidate area S j is generated (step SA8).

[0039] Subsequently, based on the following formula, the cosine similarity C i between the HSV histogram B for each small area of the background candidate area B and the HSV histogram S for the treatment instrument proximal end candidate area S j is calculated (step SA9). Then, by comparing the calculated cosine similarity C j with a predetermined similarity threshold j , it is determined whether the color of each treatment instrument proximal end candidate area S is similar to the color of the background candidate area B (step SA10). j

Equation

[0040] ​ Next, the processor 1 determines that any candidate treatment tool base end region S with a cosine similarity smaller than a predetermined similarity threshold, for example, a candidate treatment tool base end region S with color information that is not similar to the color information of the background candidate region B as shown in Figure 22, is the base end of the treatment tool 37. On the other hand, any candidate treatment tool base end region S with a cosine similarity greater than or equal to a predetermined similarity threshold, for example, a candidate treatment tool base end region S with color information that is the same as or similar to the color information of the background candidate region B as shown in Figure 23, is determined to be the tip 37a of the treatment tool 37 (step SA11). Then, the direction of the treatment tool 37 is estimated based on the positional relationship between the determined tip 37a and the base end of the treatment tool 37 (step SA12).

[0041] As described above, according to the image processing apparatus 10, image processing method, and image processing program of this embodiment, it is determined whether the candidate area S of the treatment tool base end is a background area in front of the treatment tool 37 or a part of the treatment tool trocker 39 inserted into the base end of the treatment tool 37, based on the similarity between the color information of the candidate area S of the treatment tool base end and the color information of the candidate area B of the background.

[0042] Therefore, even in situations where obstacles such as the instrument trocker 39 positioned close to the instrument 37 are visible in the endoscopic image, the tip position and direction of the instrument 37 can be quickly and accurately determined by the information on the orientation of the instrument 37 estimated by the processor 1.

[0043] This embodiment can be applied when neither of the longitudinal axes of the treatment instrument 37 is in contact with the edge of the endoscopic image, as shown in Figures 24 and 25. Alternatively, the procedure may be modified as shown in the flowchart of Figure 30 to apply when, for example, at least one of the longitudinal axes of the treatment instrument 37 is in contact with the edge of the endoscopic image, as shown in Figures 26 to 29.

[0044] In this case, after the processor 1 generates a treatment instrument mask image from the endoscopic image as shown in Figures 31 and 32 (step SA2), it detects the number of locations where the treatment instrument mask A is in contact with the edge of the endoscopic image (step SB2) before performing rectangular fitting.

[0045] If there are no contact points as shown in Figures 24 and 25, or if there is only one contact point as shown in Figures 26, 28, and 29 (Step SB3 "NO"), then Processor 1 performs rectangular fitting, for example, as shown in Figure 33 (Step SA3), and then sets a candidate background region B as shown in Figure 34 (Step SA4). Then, based on rectangular information corresponding to the number of places where the instrument mask A is in contact with the edge of the endoscopic image, a candidate proximal region S of the instrument is set (Step SB5).

[0046] For example, if there is contact at only one point, the processor 1 sets the candidate proximal end region S of the instrument only near the short side of the rectangular region F on the extension of the calculated straight line G, and on the side where the instrument mask A is not in contact with the edge of the endoscopic image, as shown in Figure 35.

[0047] In other words, if one side of the treatment instrument mask A in the longitudinal axis direction is in contact with the edge of the endoscopic image, there will be only one candidate treatment instrument proximal end region S. In this case, if the similarity between the color information of the candidate treatment instrument proximal end region S and the color information of the candidate background region B is greater than or equal to a predetermined similarity threshold, the processor 1 determines that the candidate treatment instrument proximal end region S is the tip 37a of the treatment instrument 37, as shown in Figure 36 (step SA11). On the other hand, if the similarity is less than the predetermined similarity threshold, the candidate treatment instrument proximal end region S is determined to be the proximal end of the treatment instrument 37.

[0048] If the treatment device mask A is not in contact with any edge of the endoscopic image, follow the flowchart in Figure 17. On the other hand, if the treatment instrument mask A is in contact with the edge of the endoscopic image in two places, that is, if both ends of the treatment instrument mask A along the longitudinal axis are in contact with the edge of the endoscopic image as shown in Figure 27 (step SB3 "YES"), it is difficult to determine which end along the longitudinal axis is the tip 37a and which is the proximal end, so the process is terminated.

[0049] According to this modified example, even when one end of the treatment instrument 37 in the longitudinal axis direction is in contact with the edge of the endoscopic image, the tip position and direction of the treatment instrument 37 can be quickly and accurately determined.

[0050] [Second Embodiment] Next, the endoscope system and its control method according to the second embodiment will be described below with reference to the drawings. The endoscope system 200 and its control method according to this embodiment differ from the first embodiment in that, as shown in Figure 37, the operation of the endoscope 31 is restricted when the endoscope trocker 32 for inserting the endoscope 31 into the body of the patient W, who is the subject of the study, is visible at the edge of the endoscope image. In describing this embodiment, parts that have the same configuration as the endoscope system 100 according to the first embodiment described above are denoted by the same reference numerals and their descriptions are omitted.

[0051] The endoscopic system 200 includes an endoscope 31 inserted into the patient W's body, a moving device 41 for moving the endoscope 31, a control device 45 for controlling the moving device 41, and a monitor 35. The endoscope 31 is inserted into the abdominal cavity via an endoscope trocker 32, which is fixed in a penetrating state to the surface of the patient W's body. The endoscope trocker 32 is a tubular instrument that is inserted into the body through a hole formed in the body wall, and is pivotable with respect to the endoscope pivot point T at the location of the hole in the body wall.

[0052] The mobile device 41 includes a scope holder 43, such as a robotic arm having multiple joints, and holds the base end of the endoscope 31 at the tip of the scope holder 43. In one example, the scope holder 43 has three degrees of freedom of motion: linear movement forward and backward along the Z axis, rotation around the X axis (pitch), and rotation around the Y axis (yaw), and preferably further has a degree of freedom of motion of rotation around the Z axis (roll). The Z axis is an axis on the same straight line as the optical axis and longitudinal axis of the endoscope 31, and the X axis and Y axis are axes that are perpendicular to the optical axis of the endoscope 31 and extend in directions corresponding to the lateral and vertical directions of the endoscope image, respectively.

[0053] The control device 45 includes a processor 51, a storage unit 3, a memory 5, a user interface 7, and a robot control controller 57 that controls the scope holder 43 according to instructions from the processor 51. The storage unit 3 stores a control program and the like that causes the processor 51 to execute the control method according to this embodiment.

[0054] The processor 51 detects the tip 37a of the treatment instrument 37 in the endoscopic image in order to make the endoscopic image track the tip 37a of the treatment instrument 37. The processor 51 includes a trocker detection unit 53 and a control command value generation unit 55 as functional units. The processor 51 realizes the functions of the trocker detection unit 53 and the control command value generation unit 55 by executing a program read from the storage unit 3 to the memory 5.

[0055] After receiving the endoscopic image acquired by the endoscope 31, the trocker detection unit 53 uses image processing and / or deep learning to determine whether or not an endoscopic trocker 32 is visible at the edge of the endoscopic image. The deep learning determination is made by detecting the endoscopic trocker 32 itself from the endoscopic image.

[0056] The control command value generation unit 55 generates command values ​​for the instrument tracking operation that controls the scope holder 43, based on the control program. If the trocker detection unit 53 determines that the endoscope trocker 32 is visible at the edge of the endoscope image, the control command value generation unit 55 adjusts the generated instrument tracking operation command value to a value that prevents the endoscope 31 from retracting in its longitudinal axis direction.

[0057] Specifically, as shown in Figures 38 and 39, the control command value generation unit 55 generates command values ​​for the movement speed in the XY direction within the range where the tip 37a of the treatment instrument 37 is contained within the MC of the endoscope image, and also generates command values ​​for the movement speed away in the Z-axis direction within the range where the tip 37a of the treatment instrument 37 is contained within the target distance. The MC of the endoscope image is the region where the treatment instrument tracking operation is stopped. If the tip 37a of the treatment instrument 37 is contained within the MC during the treatment instrument tracking operation, the tracking of the treatment instrument 37 by the endoscope 31 is stopped.

[0058] On the other hand, in the case of tracking operation of the treatment instrument 37 while the endoscopic trocker 32 is visible, command values ​​for the movement speed in the XY direction are generated within the range in which the tip 37a of the treatment instrument 37 is contained within the MC of the endoscopic image, and the movement speed in the Z axis direction is adjusted to a command value of 0, thereby preventing the endoscopic trocker 32 from appearing in the endoscopic image any further. The generated or adjusted command values ​​for the instrument tracking motion are sent to the robot control controller 57.

[0059] The robot control controller 57 operates the scope holder 43 based on the command values ​​for the instrument tracking operation sent from the processor 51.

[0060] The operation of the endoscopic system 200 configured in this way will be explained with reference to the flowchart in Figure 40. To observe the inside of a patient's body cavity using the endoscopic system 200 according to this embodiment, the endoscope 31, mounted on the scope holder 43, is inserted into the body cavity via the endoscope trocker 32. When the inside of the body cavity is captured by the endoscope 31, the endoscopic image is received by the processor 51 (step SC1).

[0061] Next, the processor 51 generates command values ​​for the instrument tracking operation according to the control program (step SC2). The processor 51 also determines whether or not the endoscopic trocker 32 is visible at the edge of the endoscopic image sent from the endoscope 31 (step SC3).

[0062] If the processor 51 determines that the endoscope trocker 32 is not visible (step SC4 "NO"), the generated command value for the instrument tracking movement is sent directly to the robot control controller 57 (step SC6). On the other hand, if the processor 51 determines that the endoscope trocker 32 is visible (step SC4 "YES"), the generated command value for the instrument tracking movement is adjusted to a command value that does not move the endoscope 31 backward in its longitudinal axis direction (step SC5), and then the adjusted command value is sent to the robot control controller 57 (step SC6).

[0063] Next, the robot control controller 57 operates the scope holder 43 based on the command value for instrument tracking sent from the processor 51 (step SC7). As a result, the endoscope 31 moves together with the scope holder 43, and the endoscope 31 acquires an endoscopic image that tracks the tip 37a of the instrument 37. Steps SC1 to SC7 are repeated until the process is completed (step SC8).

[0064] In this case, with conventional endoscopic systems, when performing laparoscopic surgery using the scope holder 43, the endoscope 31 may unintentionally slip out of the body cavity depending on the operation commands to the scope holder 43, or the endoscopic trocker 32 may appear at the edge of the endoscopic image. If the endoscopic trocker 32 is prominently displayed in the endoscopic image, surgery becomes difficult. International Publication No. 2023 / 145265 describes controlling the operation of the endoscope 31 based on the area of ​​the endoscopic trocker in the endoscopic image, but the specific method for detecting the endoscopic trocker is not disclosed.

[0065] In contrast, according to the endoscope system 200 and its control method according to this embodiment, as shown in Figure 37, when the endoscope trocker 32 is visible at the edge of the endoscope image, the movement of the endoscope 31 to retract in the longitudinal axis direction is suppressed. This prevents the endoscope 31 from coming out of the body cavity or the endoscope trocker 32 from appearing prominently in the endoscope image.

[0066] In this embodiment, the command value for instrument tracking operation generated by the processor 51 was used as an example to illustrate the operation input for the scope holder 43. However, commands from the user via the UI (User Interface) may also be used. In that case, even if a command is input via the UI, if the processor 51 determines that the endoscope trocker 32 is visible at the edge of the endoscope image, it may adjust the command to not retract the endoscope 31 in its longitudinal axis direction, similar to the case of the instrument tracking operation command value.

[0067] In this embodiment, the determination of whether or not the endoscopic trocker 32 is visible at the edge of the endoscopic image is performed by detecting the endoscopic trocker 32 using deep learning. However, this may be replaced by detecting the endoscopic trocker 32 using image processing.

[0068] As a method for detecting the endoscopic trocker 32 using image processing, for example, distance information and edge quantity may be used. This method detects the endoscopic trocker 32 based on the premise that, while an endoscopic trocker made of a material such as metal or plastic is less likely to develop edges at relatively close distances due to its smooth surface, organs are more prone to developing edges.

[0069] In this case, the processor 51 first uses, for example, the depth estimation AI model Midas to distinguish between the background region of organs, etc., and the region where the endoscope trocker 32 is thought to be located, to generate a depth map as shown in Figure 42 from the endoscope image as shown in Figure 41. Next, using Otsu's binarization, the processor generates a binarized image as shown in Figure 43, which divides the depth map into distances farther than a predetermined threshold and distances closer to it.

[0070] Furthermore, the processor 51 generates edge images from the endoscopic image by edge detection, for example using the Canny method, in order to distinguish between the background region of organs, etc., and the region where the endoscopic trocker 32 is thought to be located. Next, using Otsu's binarization, a binarized image is generated by dividing the edge image into areas where the edges are stronger and weaker than a predetermined threshold.

[0071] Then, the processor 51 determines the edge quantity E in the region close to the endoscope 31 based on the binarized depth map image and the binarized edge image. near And, edge amount E in the region far from the endoscope 31. far The calculated edge amount E is then calculated. near and edge amount E far The presence or absence of the endoscopic trocker 32 is determined by comparing it with the following. For example, if r in the following equation is greater than or equal to a predetermined certain value, it is determined that the endoscopic trocker 32 is present, while if it is less than the predetermined certain value, it is determined that the endoscopic trocker 32 is not present. r=E near / E farr

[0072] Furthermore, as a method for detecting the endoscopic trocker 32 using image processing, color information may be used instead of edge quantity. In this case, the processor 51 first generates a depth map from the endoscopic image using Midas, and then generates a binarized image by dividing the depth map into far and near distances using Otsu's binarization. It also generates an RGB histogram from the endoscopic image. Then, based on the generated binarized image and RGB histogram, it calculates the peak P of the RGB histogram in the near distance region. near and the peak P of the RGB histogram in the far distance region far Detects.

[0073] Next, processor 51, peak P near The number of peaks is peak P far If the number of peaks is greater than a certain threshold, it is determined that the endoscopic trocker 32 is visible in the endoscopic image. Conversely, if the number is less than a certain threshold, it is determined that the endoscopic trocker 32 is not visible in the endoscopic image.

[0074] In endoscopic images, the area where the endoscopic trocker 32 is visible is blurred when the endoscopic trocker 32 is transparent, causing background organs and other elements to appear blurred. Therefore, when examining the RGB histogram of the area where the endoscopic trocker 32 is visible, i.e., the area at a close distance, multiple peaks in the histogram can be seen, as shown in Figure 44, for example. On the other hand, in endoscopic images, the area outside the area where the endoscopic trocker 32 is visible is not blurred, or only slightly blurred. Therefore, when examining the RGB histogram of the area outside the area where the endoscopic trocker 32 is visible, i.e., the area at a far distance, the histogram tends to cluster at one point, as shown in Figure 45, for example. In Figures 44 and 45, the horizontal axis represents the class interval, and the vertical axis represents the frequency. The class interval on the horizontal axis refers to the interval in which the data is divided. In this embodiment, multiple intervals are created by dividing the range of RGB color representation from 0 to 255 into specific ranges. The frequency on the vertical axis indicates how many data points correspond to each class interval. According to this modified example, the endoscopic trocker 32 can be easily detected based on the number of peaks in the RGB histogram.

[0075] This embodiment can be modified as follows. As a first modification, as shown in the flowchart of Figure 46, when the processor 51 determines that the endoscopic trocker 32 is visible in the endoscopic image (step SC4 "YES"), the determination result indicating that the endoscopic trocker 32 is visible may be displayed on the sub-monitor 59 (step SD5).

[0076] The result of the determination may be displayed by flashing the area around the endoscope trocker 32 on the sub-monitor 59 screen, as shown in Figure 47. Alternatively, as shown in Figure 48, a mark U, such as a square (□) or a circle (○), may be displayed in one of the four corners of the sub-monitor 59 screen to indicate that the endoscope trocker 32 is being displayed. The mark U may be made to flash. In addition, the user may be notified that the endoscope trocker 32 is being displayed by sounding a warning sound or the like.

[0077] As a second modification, the command value for the endoscope tracking motion may be adjusted based on the distance between the tip 31a of the endoscope 31 and the endoscope pivot point T, which is the pivot point for the oscillation of the endoscope 31. In this case, first, if the processor 51 determines that the endoscopic trocker 32 is visible in the endoscopic image as shown in Figure 37, it records the distance between the tip 31a of the endoscope 31 and the endoscopic pivot point T while the endoscope 31 is inserted to a position where the endoscopic trocker 32 is no longer visible as shown in Figure 49.

[0078] Furthermore, by operating the scope holder 43 according to the command value for instrument tracking, if the distance between the tip 31a of the endoscope 31 and the endoscope pivot point T becomes shorter than the recorded distance, the processor 51 may adjust the command value to prevent the endoscope 31 from retracting in its longitudinal axis direction.

[0079] When observing using this modified method, as shown in the flowchart of Figure 50, the user first pre-sets an arbitrary distance between the tip 31a of the endoscope 31 and the endoscope pivot point T as the withdrawal limit (step SE1).

[0080] Next, if the processor 51 determines that the endoscope trocker 32 is visible in the image (step SC4 "YES"), a command value for instrument tracking operation is generated, in which the endoscope 31 is advanced in the Z-axis direction while tracking the tip 37a of the instrument 37 (step SE5). Then, the pre-set withdrawal limit is updated based on the distance between the tip 31a of the endoscope 31 and the endoscope pivot point T at the point when the endoscope trocker 32 is no longer visible in the endoscopic image (step SE6). Finally, the generated command value is transmitted to the robot control controller 57 (step SC6).

[0081] If the processor 51 determines that the endoscope trocker 32 is not visible (step SC4 "NO"), it determines whether the pre-set withdrawal limit has been reached (step SE7). If it is determined that the limit has not been reached, the command value for the instrument tracking movement generated in step SE5 is sent directly to the robot control controller 57 (step SC6). On the other hand, if it is determined that the withdrawal limit has been reached (step SE7 "YES"), the processor 51 adjusts the command value for the instrument tracking movement to a command value that does not move the endoscope 31 backward in its longitudinal axis direction (step SC5), and then the adjusted command value is sent to the robot control controller 57 (step SC6).

[0082] As described above, this modified version not only prevents the endoscope 31 from slipping out of the body cavity, but also makes it possible to achieve a field of view of the endoscope 31 in which the endoscope trocker 32 is not visible in the endoscopic image. In this modified example, as shown in the flowchart of Figure 51, if the processor 51 determines that the endoscopic trocker 32 is visible in the endoscopic image (step SC4 "YES"), the determination result indicating that the endoscopic trocker 32 is visible may be displayed on the sub-monitor 59 (step SD5).

[0083] [Third Embodiment] Next, the endoscopic system and its control method according to the third embodiment will be described below with reference to the drawings. As shown in Figure 52, the endoscope system 300 according to this embodiment differs from the first and second embodiments in that it changes the operating speed of the endoscope 31 according to the distance between the tip 37a of the treatment instrument 37 and the endoscope 31. In describing this embodiment, parts that have the same configuration as the endoscope system 100 according to the first embodiment and the endoscope system 200 according to the second embodiment described above are denoted by the same reference numerals and their descriptions are omitted.

[0084] The endoscopic system 300 includes an endoscope 31 inserted into the patient W's body, a moving device 41 for moving the endoscope 31 within the body, a control device 60 for controlling the moving device 41, a video system 33, and a monitor 35. The control device 60 comprises a processor 61, a storage unit 3, a memory 5, and a user interface 7. The storage unit 3 stores a control program that causes the processor 61 to execute the endoscope control method according to this embodiment.

[0085] The processor 61 includes, as a functional unit, a distance calculation unit 63 that processes endoscopic images sent from the video system 33 to calculate the distance from the endoscope 31 to the tip 37a of the treatment instrument 37; an operating speed control unit 65 that controls the operating speed of the endoscope 31 in the forward and backward direction along the longitudinal axis and the operating speed in the planar direction intersecting the forward and backward direction, respectively, based on the distance calculated by the distance calculation unit 63; and a position and orientation storage unit 67 that stores the position and orientation of the endoscope 31.

[0086] As shown in Figure 53, when the endoscope 31 is brought closer to the treatment instrument 37 from an overhead view to return to a distance suitable for detailed treatment, the movement speed of the endoscope 31 in the forward and backward direction is set as follows. For example, as shown in Figure 54, if the distance between the endoscope 31 and the tip 37a of the treatment instrument 37 is greater than a predetermined distance reference value, the movement speed of the endoscope 31 in the forward and backward direction is set to be large. On the other hand, as shown in Figure 55, for example, if the distance between the endoscope 31 and the tip 37a of the treatment instrument 37 becomes closer than a predetermined distance reference value, the movement speed of the endoscope 31 in the forward and backward direction is gradually reduced.

[0087] Figure 56 shows the relationship between the distance from the endoscope 31 to the object and the endoscope speed. When the distance between the tip 37a of the treatment instrument 37 and the endoscope 31 is, for example, 140 mm or more, the forward and backward movement speed of the endoscope 31 is set to a large value. On the other hand, when the distance between the tip 37a of the treatment instrument 37 and the endoscope 31 is less than 140 mm, the forward and backward movement speed of the endoscope 31 is gradually reduced. In Figure 56, the solid line shows an example of a speed command value (mm / s) with a speed gradient, and the dashed line shows an example of a speed command value (mm / s) with a constant speed. The hatched area in Figure 56 shows a tracking suppression region similar to conventional speed adjustment, and the non-hatted area shows a speed adjustment region in which the speed of the endoscope 31 is changed according to the distance.

[0088] Furthermore, as shown in Figure 57, the operating speed control unit 65 sets the operating speed in the direction intersecting the longitudinal axis of the endoscope 31, i.e., the planar direction, as follows when the endoscope 31 is made to follow the tip 37a of the treatment instrument 37, thereby keeping the tip 37a of the treatment instrument 37 in the center of the endoscopic image. For example, as shown in the upper part of Figure 58, when the endoscope 31 is closer to the tip 37a of the treatment instrument 37 compared to the reference position shown in Figure 57, i.e., when the insertion amount of the endoscope 31 is greater than a predetermined threshold, the operating speed of the endoscope 31 in the planar direction is increased. On the other hand, as shown in the lower part of Figure 58, when the endoscope 31 is farther from the tip 37a of the treatment instrument 37 compared to the reference position shown in Figure 57, i.e., when the insertion amount of the endoscope 31 is less than or equal to a predetermined threshold, the operating speed of the endoscope 31 in the planar direction is decreased. The insertion amount of the endoscope 31 is, for example, the distance from the tip 31a of the endoscope 31 to the endoscope trocker 32.

[0089] The planar movement speed of the endoscope 31 is adjusted, for example, according to the following formula. The amount of movement of an object on the screen at a reference distance L0 and reference velocity V0 is P0 [pix / s].

number

[0090] The amount of movement of an object on the screen, P1 [pix / s], given distance L1 and velocity V1.

number

[0091] Adjust V1 [rad / s] so that P0 = P1.

number

[0092] In this case, for example, in the endoscopic system and control method described in U.S. Patent No. 1,0028792, the endoscope was moved forward and backward at a constant speed when returning from a wide-angle view for observing the inside of the body to a view for performing detailed procedures. If the speed of the endoscope's forward and backward movement is too slow, it takes a long time to return to the view, causing stress. On the other hand, if the speed of the endoscope's forward and backward movement is too fast, the tip 31a of the endoscope may collide with the treatment instrument 37 or organs due to excessive force.

[0093] In contrast, according to the endoscope system 300 and its control method according to this embodiment, when the endoscope 31 is sufficiently far from the tip 37a of the treatment instrument 37, the forward and backward movement speed of the endoscope 31 can be set to be large, allowing for a quick return from the overhead view to the view for performing detailed procedures. Furthermore, when the endoscope 31 approaches the tip 37a of the treatment instrument 37 to a certain extent, the forward and backward movement speed of the endoscope 31 can be gradually reduced, preventing the tip 31a of the endoscope 31 from colliding with the treatment instrument 37 or organs.

[0094] Furthermore, in conventional endoscope systems and their control methods, the endoscope 31 was operated at a constant speed in the planar direction while tracking the tip 37a of the treatment instrument 37 in the center of the endoscopic image, while maintaining a constant distance between the tip 37a of the treatment instrument 37 and the endoscope 31. If the distance between the endoscope 31 and the tip 37a of the treatment instrument 37 is changed according to the user's preference while tracking the tip 37a of the treatment instrument 37, the apparent operating speed of the treatment instrument 37 on the plane of the endoscopic image changes from before the distance change, making it difficult to perform the procedure.

[0095] In contrast, according to the endoscope system 300 and its control method according to this embodiment, when the endoscope 31 is close to the tip 37a of the treatment instrument 37, the operating speed in the direction intersecting the forward and backward movement of the endoscope 31 is increased, while when the endoscope 31 is far from the tip 37a of the treatment instrument 37, the operating speed in the direction intersecting the forward and backward movement of the endoscope 31 is decreased. This makes it possible to maintain a constant apparent planar movement of the tip 37a of the treatment instrument 37 on the endoscopic image, regardless of changes in the distance between the tip 37a of the treatment instrument 37 and the endoscope 31.

[0096] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design changes and the like are also included within the scope of the gist of the present invention. For example, the present invention is not limited to being applied to the above embodiments and modified examples, but may also be applied to embodiments that appropriately combine these embodiments and modified examples, and is not particularly limited. [Explanation of Symbols]

[0097] 1 processor 10 Image Processing Device 37. Treatment tools 37a tip B Background candidate area S Proximal end candidate region of the treatment device α Direction of the treatment instrument

Claims

1. An image processing device for processing endoscopic images of a body cavity taken by an endoscope, Equipped with a processor, The processor, From the aforementioned endoscopic image, the treatment instrument region where a long treatment instrument is present is detected. In the endoscopic image, the region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument is set as the treatment instrument base end candidate region, and the region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis is set as the background candidate region. The similarity between the color information of the candidate base region of the treatment tool and the color information of the candidate background region is calculated. An image processing device that estimates information regarding the orientation of the treatment tool based on the calculated similarity.

2. The image processing apparatus according to claim 1, wherein the processor determines that the candidate region of the treatment tool base end having a similarity of a predetermined threshold or higher is the tip of the treatment tool.

3. The image processing apparatus according to claim 2, wherein the processor calculates the direction of the treatment tool based on the tip of the treatment tool.

4. The image processing apparatus according to claim 1, wherein the processor identifies a rectangular region with the smallest area including the treatment tool region by rectangular fitting, and sets the enlarged portion obtained by expanding the rectangular region in a direction parallel to the short side of the rectangular region as the background candidate region.

5. The image processing apparatus according to claim 1, wherein the processor identifies a rectangular region with the smallest area including the treatment tool region by rectangular fitting, and sets the candidate treatment tool base end region on the extension of a straight line passing through the center of the rectangular region and parallel to the long side of the rectangular region.

6. The image processing apparatus according to claim 2, wherein the processor generates a histogram for each of the subregions into which the endoscopic image is divided, calculates the cosine similarity between the histogram of each subregion corresponding to the background candidate region and the histogram of the treatment instrument base end candidate region as the similarity, and determines whether the similarity between the color information of the treatment instrument base end candidate region and the color information of the background candidate region is greater than or equal to a predetermined threshold based on the cosine similarity.

7. The image processing apparatus according to claim 6, wherein each of the aforementioned small regions is in the range of 1 / 20 to 1 / 30 of the size of the endoscopic image.

8. An image processing method for processing endoscopic images of a body cavity taken by an endoscope, To detect the instrument region where a long instrument is present from the aforementioned endoscopic image, In the endoscopic image, the region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument is set as the treatment instrument proximal end candidate region, and the region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis is set as the background candidate region. To calculate the similarity between the color information of the candidate base region of the treatment tool and the color information of the candidate background region. An image processing method that includes estimating information regarding the orientation of the treatment tool based on the calculated similarity.

9. The image processing method according to claim 8, which includes determining the proximal end candidate region of the treatment tool, in which the similarity is equal to or greater than a predetermined threshold, as the tip of the treatment tool.

10. The image processing method according to claim 9, which includes calculating the direction of a treatment tool based on the tip of the treatment tool.

11. By rectangular fitting, the smallest rectangular area including the treatment tool area is identified. The image processing method according to claim 8, further comprising setting the enlarged portion obtained by expanding the rectangular region in a direction parallel to the shorter side of the rectangular region as the background candidate region.

12. By rectangular fitting, the smallest rectangular area including the treatment tool area is identified. The image processing method according to claim 8, which includes setting the candidate base end region of the treatment tool on the extension of a straight line passing through the center of the rectangular region and parallel to the long side of the rectangular region.

13. This includes generating a histogram for each of the subregions into which the endoscopic image is divided, As the similarity, the cosine similarity between the histogram of each sub-region corresponding to the background candidate region and the histogram of the treatment tool base end candidate region is calculated. The image processing method according to claim 9, which determines whether the similarity is equal to or greater than a predetermined threshold based on the cosine similarity.

14. The image processing method according to claim 13, wherein the endoscopic image is divided into each of the small regions, each having a size ranging from 1 / 20 to 1 / 30 of the aforementioned endoscopic image.

15. An image processing program that causes a computer to process endoscopic images of a body cavity taken by an endoscope, To detect the instrument region where a long instrument is present from the aforementioned endoscopic image, In the endoscopic image, the region adjacent to the treatment instrument region in the longitudinal axis direction of the treatment instrument is set as the treatment instrument proximal end candidate region, and the region adjacent to the treatment instrument region in a direction perpendicular to the longitudinal axis is set as the background candidate region. To calculate the similarity between the color information of the candidate base region of the treatment tool and the color information of the candidate background region. An image processing program that causes the computer to estimate information regarding the orientation of the treatment tool based on the calculated similarity.

16. The image processing program according to claim 15, which causes the computer to further determine that the candidate area of ​​the prosthetic end of the prosthetic tool whose similarity is equal to or greater than a predetermined threshold is the tip of the prosthetic tool.

17. The image processing program according to claim 16, further causing the computer to calculate the direction of the treatment tool based on the tip of the treatment tool.

18. By rectangular fitting, the smallest rectangular area including the treatment tool area is identified. The image processing program according to claim 15, which causes the computer to further cause the computer to set the enlarged portion obtained by expanding the rectangular region in a direction parallel to the shorter side of the rectangular region as the background candidate region.

19. By rectangular fitting, the smallest rectangular area including the treatment tool area is identified. The image processing program according to claim 15, which further causes the computer to set the candidate base end region of the treatment tool on the extension of a straight line passing through the center of the rectangular region and parallel to the long side of the rectangular region.

20. The endoscopic image is divided into multiple subregions, and a histogram is generated for each subregion. As the similarity, the cosine similarity is calculated between the histogram of each sub-region corresponding to the background candidate region and the histogram of the treatment tool base end candidate region. The image processing program according to claim 16, which causes the computer to further perform a determination of whether the similarity is greater than or equal to a predetermined threshold based on the cosine similarity.

21. The image processing program according to claim 20, which causes the computer to further divide the endoscopic image into each of the small regions, each having a size ranging from 1 / 20 to 1 / 30 of the aforementioned endoscopic image.

Images