Image processing device, endoscope system, image processing method and program

Abstract

To provide an image processing device, etc. for displaying information on whether measurement of a distance between points designated by tips, etc. of a robot tool used in surgery is stable.SOLUTION: An image processing device 10 includes a processor 100 for performing display processing for a display DP. The processor 100 calculates three-dimensional position information on a first treatment instrument 31 and a predetermined part in an endoscopic image using an endoscopic image of a subject acquired by an endoscope 20 (step S100). The processor 100 measures a distance between a first measurement point 41 on a tip side of the first treatment instrument 31 and a second measurement point 42 related to the predetermined part on the basis of the three-dimensional position information on the first treatment instrument 31 and the predetermined part (step S200), and performs display processing for indicating whether the distance measurement is in a stable state for a display DP (second display DP2) (step S300).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an endoscope system, an image processing method, and a program. [Background technology]

[0002] Conventionally, there is known an image processing device that displays on a display an image acquired from an endoscope during a surgical operation, etc. Patent Document 1 discloses a method for displaying a measurement value obtained by measuring the distance between points designated by the tip of a robot tool used in a surgical operation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2010 / 0317965 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent document 1 discloses a method for displaying the measurement value obtained by measuring the distance between points specified by the tip of a robotic tool used in surgical operations, but does not propose a method for displaying information regarding whether the distance measurement is stable. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to an image processing device that includes a processor that performs display processing on a display, and that uses an endoscopic image of a subject acquired by an endoscope to calculate three-dimensional position information of a first treatment tool and a specified part in the endoscopic image, measures a distance between a first measurement point on the tip side of the first treatment tool and a second measurement point related to the specified part based on the three-dimensional position information of the first treatment tool and the specified part, and performs display processing on the display that indicates whether the distance measurement is stable or not.

[0006] Another aspect of the present disclosure relates to an endoscope system including the image processing device described above and an endoscope.

[0007] Another aspect of the present disclosure relates to an image processing method that causes a computer to perform the following processes: displaying an endoscopic image of a subject acquired by an endoscope on a display; calculating three-dimensional position information of a first treatment tool and a specified part in the endoscopic image using the endoscopic image of the subject; measuring the distance between a first measurement point on the tip side of the first treatment tool and a second measurement point related to the specified part based on the three-dimensional position information of the first treatment tool and the specified part; and displaying an indication of whether the distance measurement is stable.

[0008] Another aspect of the present disclosure relates to a program that causes a computer to execute the following processes: displaying an endoscopic image of a subject acquired by an endoscope on a display; calculating three-dimensional position information of a first treatment tool and a specified part in the endoscopic image using the endoscopic image of the subject; measuring the distance between a first measurement point on the tip side of the first treatment tool and a second measurement point related to the specified part based on the three-dimensional position information of the first treatment tool and the specified part; and displaying an indication of whether the distance measurement is stable. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing device included in an endoscope system. [Figure 2] 1 is a diagram illustrating the endoscope system in more detail. [Figure 3] FIG. 2 is another diagram illustrating the endoscope system in more detail. [Figure 4] 10 is a flowchart illustrating an example of processing according to the method of the present embodiment. [Figure 5] 10 is a flowchart illustrating an example of a position specification calculation process. [Figure 6] 10 is a flowchart illustrating a processing example of a first measurement point designation process. [Figure 7]10 is a flowchart illustrating an example of a second measurement point designation process. [Figure 8] 10 is a flowchart illustrating an example of measurement processing. [Figure 9] 10 is a flowchart illustrating a processing example of treatment tool stability determination processing. [Figure 10] FIG. 10 is a diagram illustrating an example of a screen when the treatment tool is not stable. [Figure 11] FIG. 10 is a diagram illustrating an example of a screen when the treatment tool is stable. [Figure 12] 10A and 10B are diagrams illustrating examples of aspects showing the stability of a treatment tool. [Figure 13] FIG. 10 is a diagram illustrating an example of displaying measured distances as time-series data. [Figure 14] FIG. 10 is a diagram illustrating an example of a screen when measuring distance using two treatment tools. [Figure 15] 10A and 10B are diagrams illustrating examples of icons indicating periods during which distance measurement is stable. [Figure 16] FIG. 10 is a diagram illustrating an example of a display mode of a measured distance. [Figure 17] FIG. 10 is a diagram illustrating an example of a screen including an image showing a range in which the tip of the first treatment tool is stable. [Figure 18] FIG. 10 is another diagram illustrating an example of a screen including an image showing the range in which the tip of the first treatment tool is stable. [Figure 19] FIG. 10 is a diagram for explaining an example of a screen including information on the tip position of the first treatment tool in a past frame. [Figure 20] 10 is a flowchart illustrating another example of the measurement process. [Figure 21] 10 is a flowchart illustrating an example of image stability determination processing. [Figure 22] 10A and 10B are diagrams illustrating an example of a screen including a display indicating image stability. [Figure 23] FIG. 10 is another diagram illustrating an example of a screen including a display showing image stability. [Figure 24] 10 is a flowchart illustrating another example of the image stability determination process. [Figure 25] FIG. 10 is a diagram illustrating an example of a screen including an image prompting rotation of the scope. [Figure 26]10 is a flowchart illustrating another example of the measurement process. [Figure 27] 10 is a flowchart illustrating another example of the measurement process. [Figure 28] 10 is a flowchart illustrating an example of Z coordinate correction processing. [Figure 29] FIG. 10A is a diagram illustrating a region of interest, and FIG. 10B is a diagram illustrating a method for calculating three-dimensional position information of a first measurement point after correction. [Figure 30] 10A and 10B are diagrams illustrating an example of an image including a first measurement point designated by another example of the first measurement point designation process. [Figure 31] 10 is a flowchart illustrating another example of the first measurement point designation process. [Figure 32] FIG. 4 is a diagram for explaining the designation of the first measurement point in more detail. [Figure 33] 10 is a flowchart illustrating another example of the first measurement point designation process. [Figure 34] FIG. 10 is another diagram illustrating the designation of the first measurement point in more detail. [Figure 35] 10 is a flowchart illustrating another example of the position specification calculation process. [Figure 36] FIG. 4 is a diagram illustrating tracking of a first measurement point and a second measurement point. [Figure 37] FIG. 10 is another diagram illustrating tracking of the first measurement point and the second measurement point. [Figure 38] 10A and 10B are diagrams illustrating examples of textures applied to the first treatment tool. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below. Note that the embodiments described below do not unduly limit the scope of the claims, and not all of the configurations described in the embodiments are necessarily essential components.

[0011] FIG. 1 is a block diagram illustrating an example configuration of an endoscope system 1 according to this embodiment. The endoscope system 1 according to this embodiment includes an image processing device 10 and an endoscope 20. The image processing device 10 includes a processor 100. The processor 100 according to this embodiment is configured with the following hardware: The hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware may be configured with one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices may be, for example, ICs, etc. The one or more circuit elements may be, for example, resistors, capacitors, etc.

[0012] Furthermore, for example, the image processing device 10 of this embodiment may be configured to include a memory (not shown in FIG. 1 ) and a processor 100 that operates based on information stored in the memory. This allows the processor 100 to function as a position designation calculation unit 110, a 3D construction unit 112, a measurement point selection unit 114, a distance calculation unit 120, a measurement execution determination unit 122, and the like, which will be described later. For convenience of explanation, the subject of the processes related to the method of this embodiment described below will be unified as the processor 100. The information stored in the memory may include, for example, programs and various data. The processor 100 may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or the like. The memory may be a volatile memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a non-volatile memory such as a read-only memory (ROM), a register, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the functions of each unit are realized as processing when the instructions are executed by the processor 100. The instructions here may be instructions from an instruction set that constitutes a program, or may be instructions that instruct the hardware circuitry of the processor 100 to operate. Memory is also called a storage device.

[0013] The above-described program can be stored in a non-transitory information storage medium, which is a computer-readable medium, such as an optical disk, a memory card, a hard disk drive, or a non-volatile memory.

[0014] The method of this embodiment can be applied to, for example, the procedure in endoscopic surgery shown in Fig. 2. In this case, it can be said that the endoscope system 1 functions, more specifically, as an endoscopic surgery system. Note that Fig. 2 is a conceptual diagram and does not strictly depict endoscopic surgery, for example, trocars are not shown.

[0015] In Fig. 2, the image processing device 10 is connected to an endoscope 20 and also to a display DP. In the endoscopic surgery shown in Fig. 2, multiple holes are drilled in the body wall of a subject, and the endoscope 20 is inserted into the body cavity through one of the holes, and treatment instruments are inserted into the body cavity through the other holes. In this case, the endoscope 20 is a rigid endoscope in which most of the insertion section is rigid. Note that Fig. 2 illustrates a state in which a first treatment instrument 31 and a second treatment instrument 32 are inserted into the body cavity as treatment instruments, but the number of treatment instruments is not limited to two.

[0016] Rigid endoscopes are well known, and detailed illustrations are omitted. However, the endoscope 20, which functions as a rigid endoscope, includes an imager at its tip. The image processing device 10 receives image signals from an imager (not shown) at the tip of the endoscope 20 via a cable (not shown). In the following description, the endoscope 20 may also be referred to as a "scope." The image processing device 10 generates a display image based on the received image signal and displays it on a display DP, as shown in A1 of FIG. 2 . In other words, the processor 100 controls the display DP. In this embodiment, an image captured by the imager at the tip of the endoscope 20 is referred to as an endoscopic image. Hereinafter, an endoscopic image may simply be referred to as an "image." The endoscopic system 1 configured in this manner displays an endoscopic image acquired by the endoscope 20 on the display DP, allowing a user to observe biological tissue in a body cavity while performing treatment on the biological tissue with a treatment tool. In this embodiment, the "user" refers to anyone involved in the treatment, including, for example, a surgeon who handles the treatment tool and a scopist who operates the endoscope 20. In recent endoscopic surgical procedures, surgical robot systems have been adopted in which the user operates a console to control a robot arm and perform the procedure, but this is not necessarily the case in all facilities, and the endoscope 20, first treatment tool 31, second treatment tool 32, etc. may be operated by the user's hands.

[0017] 2 shows only one display DP, the number of displays DP that can be connected to the image processing device 10 is not limited to one. For example, as shown in FIG. 3, the image processing device 10 may be connected to a first display DP1 and a second display DP2. In this case, the processor 100, not shown in FIG. 2, output A first display DP1 is connected via an interface 11, and a second display DP2 is connected via an interface 12. outputThe first display DP1 may be connected to a second display DP2 via an interface 12. In this case, the processor 100 that displays images on the first display DP1 and the processor 100 that displays images on the second display DP2 may be the same or different. In the following description, the first display DP1 and the second display DP2 may be referred to separately.

[0018] The display DP of this embodiment may also be a display device having a first display mode that displays a stereoscopic image. A stereoscopic image is an image that gives a user a sense of three-dimensionality. Specifically, a parallax barrier display device may be used, but a lenticular display device may also be used, and various other techniques may be employed. A parallax barrier display device uses a parallax barrier to display a left-eye image in the user's left eye and a right-eye image in the user's right eye, thereby displaying a stereoscopic image. The display DP of this embodiment may also be a display device having a second display mode that displays an endoscopic image in a two-dimensional manner, or may be configured to be switchable between the first and second display modes. For example, switching between the first and second display modes can be achieved by enabling and disabling the parallax barrier function.

[0019] The display DP of this embodiment may further include a touch panel function. In other words, the display DP of this embodiment may further include a pointing device for pointing to an arbitrary position on the screen. The detection method is not particularly limited and may include capacitive, resistive, ultrasonic, infrared, electromagnetic induction, or the like, and may be determined appropriately by the user. For example, although not shown, the display DP includes a touch panel control circuit that detects a user's touch operation. The touch panel control circuit detects the user's touch operation and outputs coordinate data on the display DP identified by the user's touch operation to the processor 100. The user's touch operation may be, for example, an operation in which the user's finger touches the surface of the display DP, but may also include an operation in which the user slides the finger while the finger is in contact with the surface of the display DP, an operation in which the user's finger is lifted from the surface of the display DP, or the like. The user's touch operation is not limited to an operation performed with the user's finger, but may also be an operation performed with a touch pen or the like held by the user. A touch pen is a tool that realizes touch operations similar to those of a user's finger, and when using a touch panel function that uses a capacitance detection method, for example, the touch pen is configured to include a conductor at the tip.

[0020] Furthermore, for example, different images may be displayed on the first display DP1 and the second display DP2. Specifically, for example, during a procedure, an image that the user mainly observes may be displayed on the first display DP1, and predetermined information may be displayed on the second display DP2. The predetermined information may be information about the distance between two predetermined measurement points specified by a predetermined method, and details of this information will be described later. The two predetermined measurement points are, for example, a first measurement point 41 and a second measurement point 42, which will be described later. The method of this embodiment relates to specifying two measurement points, measuring the distance between the two measurement points, and determining whether measurement is possible.

[0021] In this embodiment, as shown in FIG. 10 (described later), an example screen to which the method of this embodiment is applied is displayed on the second display DP2. However, this is merely an example and does not limit other display methods. For example, a screen similar to the screen shown in FIG. 10 may be displayed on a portion of the first display DP1. The same applies to the example screens shown in FIGS. 11, 14, 17, 18, 19, 22, 23, 25, 30, 36, and 37 (described later). Furthermore, although these example screens are illustrated in a two-dimensional manner for convenience, this does not prevent them from being displayed as stereoscopic images when the method of this embodiment is actually applied.

[0022] In addition, in this embodiment, as described later, as methods for obtaining position information of two predetermined measurement points, a first method of performing calculation based on the tip position of the treatment tool displayed on the second display DP2 and a second method of performing calculation based on the position of the subject corresponding to a position on the second display DP2 specified by the user are exemplified. In this embodiment, for simplicity of explanation, when the position information of two predetermined measurement points is obtained using both the first method and the second method, the measurement point whose position information is obtained using the first method is referred to as the first measurement point 41, and the measurement point whose position information is obtained using the second method is referred to as the second measurement point 42. Note that, as described later with reference to FIG. 14, the position information of the first measurement point 41 and the second measurement point 42 can also be obtained using only the first method.

[0023] The processor 100 included in the image processing device 10 of this embodiment functions as the position designation calculation unit 110 shown in Fig. 1 to calculate position information of two specified predetermined measurement points. The position designation calculation unit 110 also includes a 3D construction unit 112. In other words, the position information of the two specified measurement points is three-dimensional position information, and the processor 100 can calculate the three-dimensional position information of the two specified measurement points by functioning as the 3D construction unit 112, as will be described later.

[0024] 1, the processor 100 calculates the distance based on the position information calculated by the position designation calculation unit 110. Specifically, the processor 100 calculates the distance between the first measurement point 41 and the second measurement point 42 based on the three-dimensional position information of the first measurement point 41 and the three-dimensional position information of the second measurement point 42. Hereinafter, the distance between the first measurement point 41 and the second measurement point 42 may be simply referred to as the "distance." The distance calculation unit 120 also includes a measurement execution determination unit 122. That is, the processor 100 functions as the measurement execution determination unit 122 and determines whether the distance measurement is stable enough to perform the distance measurement.

[0025] In this embodiment, the term "stable distance measurement" specifically refers to, for example, small variations in the position information of the first measurement point 41 and the second measurement point 42, and the processor 100 determines whether the distance measurement is stable by, for example, a treatment tool stability determination process (step S250) described later. Furthermore, "stable distance measurement" may also refer to small variations in the measured distances, which will be described later with reference to FIG. 13. Furthermore, "stable distance measurement" may also include a stable endoscopic image, which will be described later with reference to FIG. 21, etc.

[0026] The processor 100 then functions as a video processor and displays the measured distance information on the display DP. More specifically, in the example of FIG. 3, the processor 100 performs the process of iconizing the measured distance information and the process of output The processor 100 then performs processing to display the information on the second display DP2 via the interface 12. The processor 100 can also display information other than the measured distance information, the details of which will be described later with reference to FIG.

[0027] The imager of this embodiment is, for example, a three-dimensional camera, more specifically, a stereo camera. The stereo camera is composed of a base camera and a reference camera. In this embodiment, an image captured by the base camera is called a base image, an image captured by the reference camera is called a reference image, and a set of a base image and a reference image is called a stereo image. The camera parameters of the base camera and the camera parameters of the reference camera are adjusted to be equal, and the position where the base camera captures the image and reference The only difference is the position where the camera captures the image. The positional relationship between the base camera and the reference camera is such that the optical axes of the base camera and the reference camera are parallel to each other, and the image plane of the base camera is reference The cameras are designed so that their imaging surfaces are on the same plane and aligned horizontally. However, since slight deviations in the relative positions may occur in actual imaging, it may be possible to correct the image by taking into account such deviations.

[0028] In a stereo camera, a base camera is placed on one of the left and right sides, and a reference camera is placed on the other, thereby capturing an image for the left eye and an image for the right eye, and three-dimensional position information of the subject can be calculated based on the image for the left eye and the image for the right eye. In other words, the processor 100 calculates three-dimensional position information based on the image for the left eye and the image for the right eye, which are endoscopic images acquired by the endoscope 20. In the following description, the left side will be the base camera and the right side will be the reference camera. In other words, in the following description, the image for the left eye will be the base image and the image for the right eye will be the reference image.

[0029] Although illustration is omitted because the method of obtaining three-dimensional position information using a stereo camera is well known, for example, if a desired position in three-dimensional space of an imaged object is position P, the two-dimensional coordinates of position P1 on the base image corresponding to position P and the two-dimensional coordinates of position P2 on the reference image corresponding to position P will not match, resulting in a deviation. This deviation is called parallax, and since the three-dimensional coordinates of position P are unknown, the parallax is also unknown.

[0030] The processor 100 functions as a 3D construction unit 112. When a stereo image is acquired from the endoscope 20, the processor 100 searches the reference image for a pixel corresponding to the pixel at position P1 in the base image, i.e., the pixel at point P2. The processor 100 calculates the disparity of point P based on the two-dimensional coordinates of the pixel at position P1 and the two-dimensional coordinates of the pixel at point P2. The method for calculating the disparity is called stereo matching. After determining the disparity of point P, the processor 100 calculates the three-dimensional coordinates of point P using the principle of triangulation based on the position information of the center of the base camera, the position information of the center of the reference camera, the position information of point P1, and the position information of point P2. Note that the higher the accuracy of the search for the pixel at position P2, the higher the accuracy of the parallax calculation, and the more accurately the three-dimensional coordinates of position P can be determined. By repeating the above calculation for each pixel in the base image, depth information of the subject and the three-dimensional position information of the subject can be obtained.

[0031] Various methods have been proposed as algorithms related to stereo matching. In this embodiment, for example, a method for calculating disparity based on semi-global block matching can be adopted, but this does not preclude the use of other algorithms. Furthermore, to implement these algorithms in the image processing device 10, programming using a predetermined open source library can be performed, and a program based on this programming can be stored in the above-mentioned non-transitory information storage medium.

[0032] For ease of explanation, FIG. 3 illustrates three mutually orthogonal axes: an X axis, a Y axis, and a Z axis. Hereinafter, the terms "X direction," "Y direction," and "Z direction" may be used. The "X direction" is a direction along the X axis and parallel to the horizontal direction of the second display DP2. The "Y direction" is a direction along the Y axis and parallel to the vertical direction of the second display DP2. The "Z direction" is a direction along the Z axis and parallel to the depth direction of the second display DP2. The "X direction" may also be referred to as the "horizontal direction." When a stereo camera is used as the imager in this embodiment, the base camera and the reference camera are arranged horizontally as described above, and therefore the X direction can also be referred to as the "parallax direction."

[0033] Below, examples of processing related to the method of this embodiment and examples of screens when this processing example is applied are shown. Note that these examples of screens are shown assuming the above-mentioned endoscopic surgery, but the method of this embodiment is not limited to endoscopic surgery using the endoscope 20 as a rigid endoscope. For example, the endoscopic system 1 of this embodiment may be configured to include the endoscope 20 as a flexible endoscope and the image processing device 10. This makes it possible to use the method of this embodiment for the purpose of measuring the distance from the entrance through which the endoscope 20 is inserted to a desired position, for example, in an endoscopic examination or the like.

[0034] 4 is a flowchart illustrating an example of processing according to the method of this embodiment. The processor 100 performs processing to determine whether measurement has started (step S10). If the processor 100 determines that measurement has started (YES in step S10), it performs processing from the position specification calculation processing (step S100) onwards, and if it determines that measurement has not started (NO in step S10), it performs step S10 again.

[0035] The process of step S10 can be implemented by various methods. For example, the user may press a predetermined button included in the endoscope 20, a button included in the treatment tool, a button on an operation panel (not shown) of the image processing device 10, or a foot pedal (not shown) connected to the image processing device 10. Alternatively, the image processing device 10 may include a voice command device so that the processor 100 determines YES in step S10 based on a predetermined voice. Alternatively, a memory (not shown) may include a gesture recognition program so that the processor 100 determines YES in step S10 based on the gesture recognition program when the user operates the treatment tool so that the tip of the treatment tool performs a predetermined gesture. Alternatively, the processor 100 may determine YES in step S10 when one treatment tool is detected from the endoscopic image and the user touches the touch panel of the second display DP2, as described below. Alternatively, the processor 100 may determine YES in step S10 when multiple treatment tools are detected from the endoscopic image.

[0036] Thereafter, the processor 100 performs a position designation calculation process (step S100), a measurement process (step S200), and a display update process (step S300), and then performs a process of determining whether or not to end the measurement (step S400). If the processor 100 determines not to end the measurement (NO in step S400), it performs the position designation calculation process (step S100) again, and if it determines to end the measurement (NO in step S400), it performs the process of determining whether or not to end the measurement (NO in step S400). YES), and the flow ends. The position specification calculation process (step S100) and the measurement process (step S200) will be described in detail later, but the position specification calculation process (step S100) calculates three-dimensional position information of the first measurement point 41 and the second measurement point 42. The processor 100 then functions as the distance calculation unit 120, and measures the distance between the first measurement point 41 and the second measurement point 42 by the measurement process (step S200). Then, by step S300, the image data newly generated or updated by the position specification calculation process (step S100) and the measurement process (step S200) is displayed on the second display DP2.

[0037] Until an event occurs that results in YES in step S400, the processor 100 performs position specification calculation processing (step S100), measurement processing (step S200), and display update processing (step S300) at the timing when one frame period of endoscopic images is acquired from the endoscope 20.

[0038] That is, in this embodiment, the processor 100 continues measuring distance until an event occurs that results in YES in step S400. The event that results in YES in step S400 is, for example, the user pressing a button to end a program related to measurement, or the passage of a certain amount of time from the time that YES was determined in step S10, etc.

[0039] The position designation calculation process (step S100) will be described in more detail using the flowchart in Fig. 5. The position designation calculation process (step S100) in Fig. 5 is a process for determining the first measurement point 41 by the first method described above and the second measurement point 42 by the second method described above. In Fig. 5, the processor 100 performs the first measurement point designation process (step S110) and the second measurement point designation process (step S120), and then performs the three-dimensional position information calculation process (step S130) and ends the flow.

[0040] The first measurement point designation process (step S110) will be described in more detail using the flowchart of FIG. 6. The processor 100 performs a process of recognizing each treatment tool in the image (step S112), and then a process of recognizing the tip of each treatment tool (step S114). For example, because treatment tools are made of metal, the treatment tools have a higher brightness value than tissue among the objects. Therefore, the processor 100 converts the endoscopic image into a brightness image, detects pixels having a brightness value equal to or greater than a predetermined threshold, groups the pixels by contour detection, and detects groups with a predetermined number of pixels or more as treatment tools, thereby achieving step S112. Furthermore, the processor 100 can achieve step S114 by regarding the tip of the pixel group detected in step S112 as the tip of the treatment tool. Note that steps S112 and S114 may be achieved by other methods. For example, treatment tools may be recognized using a trained model that has been machine-learned to enable image recognition of the presence or absence of a treatment tool in an endoscopic image. In this case, the trained model is, for example, a convolutional neural network (CNN). Furthermore, since the Z-direction position of the treatment tool in the endoscopic image is located closer to the viewer than the Z-direction position of the tissue, the image of the portion related to the treatment tool can be considered a foreground image, and the image of the portion other than the treatment tool can be considered a background image. Therefore, step S112 may be a process of extracting the region related to the treatment tool from the endoscopic image as the foreground image, and acquiring a treatment tool mask image, which is an image in which the region other than the region related to the foreground image is masked. The method of automatically generating a foreground mask image when a region related to the foreground image is specified can be realized by storing a program using a known software library in the non-transitory information storage medium described above, for example.

[0041] The processor 100 then performs a process (step S116) of designating the selected distal end portion as the first measurement point 41. If multiple treatment tools are displayed on the second display DP2, the processor 100 determines in step S114 that the number of recognized distal ends of the treatment tools is equal to the number of treatment tools displayed on the second display DP2. The processor 100 then functions as the measurement point selector 114 and selects the distal end portion of the desired treatment tool as the first measurement point 41 in step S116. Step S116 may be, for example, a notification process that prompts the user to select the distal end portion of the desired treatment tool, or a process in which the processor 100 selects the distal end portion of a predetermined treatment tool. The predetermined treatment tool is, for example, a treatment tool whose distal end coordinates are closest to the center coordinates of the second display DP2. Note that, for example, if the first measurement point designation process (step S110) is performed when only one treatment tool is displayed on the second display DP2, step S116 may be omitted.

[0042] FIG. 7 is a flowchart illustrating an example of the second measurement point designation process (step S120). The processor 100 designates the second measurement point 42 based on the portion designated by the user (step S122). For example, as described above, if the second display DP2 functions as a touch panel, the user touches a desired position on the subject displayed on the second display DP2. The touch panel control circuit of the second display DP2 then outputs position information data based on the touched position to the processor 100. The processor 100 then performs a process of converting the position information data received from the touch panel control circuit into position information on the endoscopic image. In other words, the second measurement point 42 is the position of the subject corresponding to the position on the second display DP2 designated by the user.

[0043] Returning to FIG. 5 , the three-dimensional position information calculation process (step S130) will be described. As described above, the processor 100 performs three-dimensional construction of the subject and calculates the depth direction in accordance with the stereo matching algorithm, thereby calculating three-dimensional position information of the first measurement point 41 and the second measurement point 42. More specifically, for example, the processor 100 performs preprocessing such as resizing on the stereo images, and then performs a process of rectifying the stereo images. The process of rectifying the stereo images is performed using pre-determined internal parameters and external parameters. The internal parameters include, for example, distortion due to the lenses of the base camera and the reference camera, the focal length of the lenses, and the pixel pitch. The external parameters include the position and rotation amount of the reference camera relative to the base camera. Furthermore, when the above-described treatment tool mask image is acquired, the processor 100 may perform preprocessing for stereo matching on the treatment tool mask image because the treatment tool mask image is deformed due to the above-described stereo image rectification process.

[0044] Then, the processor 100 performs stereo matching using an algorithm such as semi-global block matching, thereby calculating depth information based on the parallax and determining three-dimensional position information of the tip portion of the first treatment tool 31 selected in step S116.

[0045] The measurement process (step S200) will be described in more detail using the flowchart of Fig. 8. The processor 100 functions as the measurement execution determination unit 122 and performs treatment tool stability determination process (step S250). Thereafter, the processor 100 performs a process (step S270) to measure the distance between the first measurement point 41 and the second measurement point 42. The treatment tool stability determination process (step S250) will be described in detail later. Step S270 is performed in Fig. 5Based on the three-dimensional position information of the first measurement point 41 and the second measurement point 42 calculated in step S130, the processor 100 calculates the distance between the first measurement point 41 and the second measurement point 42. After performing step S270, the processor 100 performs a process of generating display data (step S290). The display data in step S290 includes display data in which information related to treatment tool stability obtained by the treatment tool stability determination process (step S250) described below is iconized, display data in which information on the distance obtained in step S270 is iconized, etc.

[0046] The treatment tool stability determination process (step S250) will be described in more detail using the flowchart of FIG. 9. The processor 100 performs a process (step S252) to determine whether data from the previous frame is available. If data from the previous frame is available (YES in step S252), the processor 100 performs a process (step S254) to determine whether the movement amount of the first measurement point 41 is within a first predetermined range. The movement amount of the first measurement point 41 refers to the magnitude of the distance between the three-dimensional position coordinates of the first measurement point 41 in the previous frame and the three-dimensional position coordinates of the first measurement point 41 in the current frame. Because the first treatment tool 31 and the endoscope 20 are operated by the user's hands, an endoscopic image may be captured that appears to show the tip of the first treatment tool 31 moving. Therefore, the first predetermined range is set as an allowable range for displacement of the tip position of the first treatment tool 31, and step S254 monitors whether the displacement occurring with frame update is acceptable. The first predetermined range may be an allowable range for changes in the tilt of the first treatment tool 31 relative to the horizontal direction.

[0047] On the other hand, if there is no data of the previous frame (NO in step S252), the processor 100 performs a process of incrementing the first count value by 1 (step S256). The details of the first count value will be described later. If the result in step S252 is NO, in other words, it is the first frame of the endoscopic image captured by the imager.

[0048] When the processor 100 determines that the movement amount of the first measurement point 41 is within the first predetermined range (YES in step S254), the processor 100 performs a process of adding 1 to the first count value (step S256).

[0049] If the processor 100 determines YES again in step S254, it further increments the first count value by 1. That is, if the position of the tip of the first treatment tool 31 is stable, the first count value is cumulatively incremented as the number of imaging frames increases. A first predetermined value may be further set as an upper limit of the first count value. This allows the user to determine that, when the first count value is equal to or greater than the first predetermined value, the tip position of the first treatment tool 31 is stable enough to determine that the accuracy of the measured distance between the first measurement point 41 and the second measurement point 42 is sufficiently high. If the processor 100 determines YES in step S254 when the first count value has reached the first predetermined value, it may perform processing in step S256 to maintain the first count value at the first predetermined value. More specifically, for example, if it is desired that the tip of the first treatment tool 31 be stable over a period obtained by multiplying one frame period by 30 frames, the first predetermined value may be set to 30.

[0050] In this embodiment, the period obtained by multiplying the period of one frame by the first predetermined value is called the first predetermined period. In other words, if the tip position of the first treatment tool 31 is stable for the first predetermined period, the user can determine that the accuracy of the measured distance between the first measurement point 41 and the second measurement point 42 is sufficiently high. In other words, the first count value has the technical significance of being an index of the period during which the tip position of the first treatment tool 31 continues to be stable.

[0051] Although not shown in a flowchart or the like, for example, the processor 100 may further perform a process of calculating the ratio between the first count value and the first predetermined value and storing the ratio in the treatment tool stability determination process (step S250).

[0052] On the other hand, when the processor 100 determines that the amount of movement of the first measurement point 41 is not within the first predetermined range (NO in step S254), it performs processing to set the first count value to 0 (step S258). The fact that the amount of movement of the first measurement point 41 is not within the first predetermined range means that the position of the tip of the first treatment tool 31 is not stable, and since it is considered that a certain period of time will be required until the position of the tip of the first treatment tool 31 stabilizes again, the first count value is set to 0.

[0053] Note that the processing when a NO determination is made in step S254 is not limited to this, and may be changed by the user as appropriate. For example, the processor 100 may subtract a predetermined number from the first count value in step S258, or may perform step S259, which will be described later, without adding or subtracting from the first count value. For example, if the tip of the first treatment tool 31 is stable but the NO determination is made in step S254 due to the occurrence of temporary noise, it is considered that it is not necessary to set the first count value to 0.

[0054] After performing step S256 or step S258, the processor 100 performs a process (step S259) of determining the stability of the first treatment tool 31 according to the first count value. For example, the processor 100 indicates the ratio of the accumulated first count value at the time of step S259 to a target value, and determines whether the position of the tip of the first treatment tool 31 is stable.

[0055] 10 shows an example of a screen of the second display DP2 to which the method of this embodiment is applied. The processor 100 displays the screen shown in A10, the icon shown in A13, the icon shown in A14, and the icon shown in A15 on the second display DP2 based on an endoscopic image captured by the imager of the endoscope 20.

[0056] On the screen shown in A10 of Fig. 10, the first treatment tool 31 is recognized by the first measurement point designation process (step S110), and the tip of the first treatment tool 31 is displayed as the first measurement point 41. Note that, for convenience, Fig. 10 displays only the first treatment tool 31 associated with the first measurement point 41 selected in step S116, but treatment tools other than the first treatment tool 31 may also be displayed on the screen of A10. The same applies to screen examples shown in Figs. 11, 14, 17, 18, 19, 22, 23, 25, 30, 36, and 37, which will be described later.

[0057] The icon shown in A13 indicates the ratio of the period during which the treatment tool is determined to be stable to the period during which it is desirable for the treatment tool to be stable. In other words, the ratio of the first count value accumulated in step S256 to the first predetermined value is displayed by a graph icon. Note that the form of the icon shown in A13 is not limited to this, and details will be described later.

[0058] The icon shown in A14 is a character icon relating to the stability of distance measurement, in other words, a character icon relating to the stability of the treatment tool. The icon shown in A15 is a symbol icon simply indicating the degree of stability. Note that the form of the icon shown in A15 is not limited to this, and details will be described later with reference to FIG. 12.

[0059] 10, the icon A13 indicates that the ratio of the first count value to the first predetermined value is not sufficient to determine that the treatment tool is stable. The icons A14 and A15 simply indicate that the treatment tool is not stable.

[0060] Meanwhile, the user specifies the second measurement point 42 using the touch panel function of the second display DP2. Then, based on the measurement results measured in step S270 of Fig. 8, an icon of distance information shown in A11 and an arrow icon shown in A12 are displayed. That is, in step S290 of Fig. 8, the processor 100 generates icons shown in A11, A12, A13, A14, and A15, an icon indicating the first measurement point 41, and an icon indicating the second measurement point 42. Then, in step S300, the processor 100 displays these icons together with the endoscopic image on the second display DP2.

[0061] In this embodiment, the distance information icon shown in A11 is configured not to include unit information, but it may be configured to display a specific unit such as "100 mm," or it may be possible to switch between a display mode including units and a display mode not including units. Also, for example, a table relating to units of length and their conversion may be stored in a memory (not shown), and when the user selects a unit of length, the distance information icon shown in A11 may change to an icon based on a number corresponding to the selected unit.

[0062] For ease of explanation, icons indicating the first measurement point 41 and the second measurement point 42 are superimposed on the screen shown in A10 and displayed on the second display DP2, but this is not a mandatory display and can be determined by the user as appropriate. In Fig. 10, the distal end position of the first treatment tool 31 and the display position of the icon indicating the first measurement point 41 are shifted for convenience. The same applies to Figs. 11, 14, 17, 18, 19, 22, 23, 25, 29(A), 36, and 37, which will be described later.

[0063] In addition, the arrow icon shown in A12 in Fig. 10 does not necessarily have to be displayed, and can be determined appropriately by the user. The same applies to Figs. 11, 14, 22, 23, 36, and 37, which will be described later.

[0064] The dotted frame shown in A16 is a frame indicating that the first treatment tool 31 has been recognized by image processing, and may be displayed on the second display DP2 to supplementarily indicate that the first treatment tool 31 has been recognized, but does not necessarily have to be displayed. Hereinafter, explanations and illustrations of the dotted frame may be omitted as appropriate.

[0065] 11 shows another example of a screen of the second display DP2 to which the method of this embodiment is applied. As in FIG. 10, the second display DP2 displays the screen shown in A20, and also displays the graph shown in A23. icon 8, a character icon shown in A24 is displayed. The character icon shown in A24 includes a character icon shown in A25. Also, as in the case of FIG. 10, the first treatment tool 31 is recognized by the first measurement point designation process (step S110) described above, and the tip of the first treatment tool 31 is displayed as the first measurement point 41. Meanwhile, the user touches the second display DP2 serving as a touch panel, thereby designating the second measurement point 42. Then, based on the measurement result measured in step S270 of FIG. 8, a distance information icon shown in A21 and an arrow icon shown in A22 are displayed.

[0066] Fig. 10 is an example of a screen when the stability of the treatment tool is insufficient, while Fig. 11 is an example of a screen when the stability of the treatment tool is sufficient. Therefore, the various icons displayed on the second display DP2 are also different. For example, the icon shown in A23 indicates that the ratio of the first count value to the first predetermined value is sufficient for the period during which it can be determined that the treatment tool is stable. The icons shown in A24 and A25 simply indicate that the treatment tool is stable. Furthermore, the display form of the distance information icon shown in A11 of Fig. 10 and the distance information icon shown in A21 of Fig. 11 are also different, and details will be described later in Fig. 16.

[0067] The symbol icon indicating the stability of the treatment tool is not limited to the modes shown in A15 of Fig. 10 and A25 of Fig. 11, and various modifications are possible. For example, as shown in Fig. 12, treatment tool stability mode patterns may be stored in a memory (not shown) according to the stability of the treatment tool, and a mode pattern may be selected according to the desired stability of the treatment tool. The stability of the treatment tool in Fig. 12 is, for example, the ratio of the first count value to the first predetermined value.

[0068] For example, if the first predetermined value is set to 30, the stability of the treatment tool will be 100% if the processor 100 judges YES in step S254 30 times in a row, and if the processor 100 judges YES in step S254 18 times in a row, the stability of the treatment tool will be 60%.

[0069] For example, if a certain degree of error in the measured distance is acceptable, it may be acceptable to use the distance measurement value when the stability of the treatment tool exceeds 60%. In this case, for example, pattern P-A1 in FIG. 12 may be used to distinguish between cases where the stability of the treatment tool is 100%, cases where the stability is 80% or more but less than 100%, cases where the stability is 60% or more but less than 80%, and cases where the stability is 0% or more but less than 60%. Note that "80% to 100%" in FIG. 12 means "80% or more but less than 100%," and the same applies to other notations. Furthermore, for example, when it is not necessary to distinguish whether the stability of the treatment tool is 100%, pattern P-A2 in FIG. 12 may be used. Pattern P-A2 differs from pattern P-A1 in that the same display mode is used for cases where the stability of the treatment tool is 100% and cases where the stability is 80% or more but less than 100%.

[0070] Furthermore, for example, if it is desired to grasp the measured distance as accurately as possible, it is desirable that the stability of the treatment tool reaches 100%. In this case, for example, a behavior pattern such as pattern P-A3 in FIG. 12 may be used. Pattern P-A3 is a behavior pattern consisting of two behaviors: a behavior when the stability of the treatment tool is 100% and a behavior when it is not. Furthermore, although pattern P-A3 indicates the stability of the treatment tool in the form of a symbol, the stability of the treatment tool may also be indicated in the form of letters, as shown in pattern P-A4, for example. In this way, the user can easily grasp the stability of the treatment tool.

[0071] Although not shown in the flowchart, for example, the processor 100 stores the measured distance as time-series data in a memory (not shown), and calculates the measured distance using a graph shown in FIG. Over time A process may be performed to display a graph showing the change. The vertical axis of the graph in Fig. 13 represents the distance information measured in step S270 in Fig. 8, and the horizontal axis represents time.

[0072] For example, if the second display DP2 has a touch panel function and the user designates the second measurement point 42 using the touch panel function, the stability of the tip of the first treatment tool 31 can be quantitatively grasped from the graph in Fig. 13. In other words, if the second measurement point 42 is designated in step S122 of Fig. 7, the subject associated with the second measurement point 42 can be treated as stationary, and therefore the behavior of the graph in Fig. 13 is considered to depend on the stability of the tip of the first treatment tool 31 being operated by the user. For example, when the user starts operating the first treatment tool 31, and as the user becomes accustomed to operating the first treatment tool 31, the behavior of the tip of the first treatment tool 31 stabilizes, and the range of variation in the measured distance narrows over time.

[0073] Therefore, for example, the processor 100 may perform a process of determining that the distance measurement is stable when the amount of change in the measured distance falls within a second predetermined range during a second predetermined period in the measurement process (step S200) of Fig. 8. Specifically, for example, the width shown in B1 of Fig. 13 corresponds to the second predetermined range, and the width shown in B2 corresponds to the second predetermined period.

[0074] Furthermore, as described above, since the position information of the first measurement point 41 and the position information of the second measurement point 42 can be obtained using only the first method, an example screen of the second display DP2 to which the method of this embodiment is applied can be, for example, as shown in FIG. 14. In FIG. 14, the second display DP2 displays the screen shown in A30, as well as the icons shown in A33, A34, and A35. The icons shown in A31, A32, A33, A34, and A35 in FIG. 14 correspond to the icons shown in A21, A22, A23, A24, and A25 in FIG. 11. On the other hand, the screen shown in A30 in FIG. 14 differs from the example screen shown in A20 in FIG. 11 in that the position information of the tip of the second treatment tool 32 is used as the position information of the second measurement point 42.

[0075] For example, although not shown in the flowchart, the processor 100 executes step S110 in a situation where at least the first treatment tool 31 and the second treatment tool 32 are displayed on the imager of the endoscope 20. Steps In S112, the processor 100 recognizes the first treatment tool 31 and the second treatment tool 32. Then, in step S114, the processor 100 recognizes the tip portion of the first treatment tool 31 and the tip portion of the second treatment tool 32. Then, in step S116, the processor 100 designates the tip portion of the first treatment tool 31 as the first measurement point 41, and designates the tip portion of the second treatment tool 32 as the second measurement point 42. This makes it possible to designate the second measurement point 42 instead of step S120. Then, the processor 100 further performs steps S130, S200, and S300, thereby realizing the display of the example screen in FIG. 14.

[0076] 14, when the position information of the tip of the second treatment tool 32 is used as the position information of the second measurement point 42, the stability of the tip of the second treatment tool 32 may be further determined. In other words, the treatment tool stability determination process (step S250) in FIG. 9 may be applied to both the first measurement point 41 and the second measurement point 42.

[0077] 10 may be displayed for each of the first measurement point 41 and the second measurement point 42. Alternatively, the icons shown in A13, A14, and A15 in FIG. 10 may be displayed with priority given to the first measurement point 41 or the second measurement point 42, whichever has the lower stability of the treatment tool.

[0078] For the above reasons, the image processing device 10 of this embodiment includes a processor 100 that performs display processing on the display DP. The processor 100 calculates three-dimensional position information of the first treatment tool 31 and a predetermined portion within the endoscopic image using an endoscopic image of the subject acquired by the endoscope 20 (step S100). The processor 100 also measures the distance between a first measurement point 41 on the distal end side of the first treatment tool 31 and a second measurement point 42 related to the predetermined portion based on the three-dimensional position information of the first treatment tool 31 and the predetermined portion (step S200), and performs display processing (step S300) on the display DP (second display DP2) to indicate whether the distance measurement is stable or not.

[0079] In this way, the processor 100 included in the image processing device 10 of this embodiment can calculate three-dimensional position information of the first measurement point 41 and the second measurement point 42 in the endoscopic image acquired from the endoscope 20. In addition, since a display process is performed to indicate whether or not the measurement of the distance between the first measurement point 41 and the second measurement point 42 is stable, the user can easily determine from the display DP whether or not the distance is being measured stably. This allows the user to perform treatment based on the appropriately measured distance.

[0080] For example, as described above, in endoscopic surgery, the endoscope 20 and the first treatment tool 31 are operated by the user's hands, which can lead to significant variations in the position information of the tip of the first treatment tool 31. Similarly, the influence of hand shake on the imager of the endoscope 20 can be significant. In this regard, by applying the method of this embodiment, the user can determine whether the distance measurement is stable or not, and can perform the procedure using the distance when the distance measurement is stable. This allows the user to perform more appropriate treatment. For example, when evaluating the surgical margin for partial resection of a malignant tumor, distance information of the measured surgical margin is important.

[0081] The aforementioned U.S. Patent Application Publication No. 2010 / 0317965 discloses a method for displaying measurements of the distance between points specified by the tip of a robotic tool used in surgical operations, but does not propose a method for displaying information regarding whether the distance measurement is stable.

[0082] The method of the present embodiment may also be realized as an endoscope system 1. That is, the endoscope system 1 of the present embodiment includes an image processing device 10 and an endoscope 20. By doing so, it is possible to obtain the same effects as those described above.

[0083] The technique of this embodiment may also be realized as a processing method. That is, the processing method of this embodiment causes a computer to perform the following processes: displaying an endoscopic image of a subject acquired by the endoscope 20 on a display DP; and calculating, using the endoscopic image of the subject, three-dimensional position information of the first treatment tool 31 and a predetermined portion within the endoscopic image. The processing method of this embodiment also causes the computer to perform the following processes: measuring the distance between a first measurement point 41 on the distal end of the first treatment tool 31 and a second measurement point 42 associated with the predetermined portion based on the three-dimensional position information of the first treatment tool 31 and the predetermined portion; and displaying an indication of whether the distance measurement is stable. This achieves the same effects as described above.

[0084] The method of this embodiment may also be implemented as a program. That is, the program of this embodiment causes a computer to execute the following steps: displaying an endoscopic image of a subject acquired by the endoscope 20 on a display DP; and calculating, using the endoscopic image of the subject, three-dimensional position information of the first treatment tool 31 and a predetermined portion within the endoscopic image. The program of this embodiment also causes the computer to execute the following steps: measuring the distance between a first measurement point 41 on the distal end of the first treatment tool 31 and a second measurement point 42 associated with the predetermined portion based on the three-dimensional position information of the first treatment tool 31 and the predetermined portion; and displaying a display indicating whether the distance measurement is stable. This achieves the same effects as described above.

[0085] The processor 100 may also calculate three-dimensional position information based on the left-eye image and the right-eye image, which are endoscopic images acquired by the endoscope 20. In this way, it is possible to construct an image processing device 10 that can determine whether or not the distance measured using the position information of the first measurement point 41 and the position information of the second measurement point 42 obtained by stereo matching is in a stable state.

[0086] Furthermore, the three-dimensional position information of the predetermined portion may be three-dimensional position information of the second treatment tool 32. Furthermore, the processor 100 may measure the distance between a first measurement point 41 on the distal end side of the first treatment tool 31 and a second measurement point 42 which is a position on the distal end side of the second treatment tool 32, based on the three-dimensional position information of the first treatment tool 31 and the predetermined portion. In this way, it is possible to construct an image processing device 10 which can determine whether or not the distance measurement using the distal end position of the first treatment tool 31 and the distal end position of the second treatment tool 32 is stable.

[0087] Furthermore, the three-dimensional position information of the predetermined portion may include three-dimensional position information of a second measurement point 42, which is the position of the subject corresponding to a position on the display DP (second display DP2) specified by the user. Furthermore, the processor 100 may measure the distance between the first measurement point 41 on the distal end side of the first treatment tool 31 and the second measurement point 42, based on the three-dimensional position information of the first treatment tool 31 and the predetermined portion. In this way, it is possible to construct an image processing device 10 that can determine whether or not distance measurement using the distal end position of the first treatment tool 31 and a position specified by the user is stable.

[0088] Furthermore, the processor 100 may determine that the distance measurement is stable when the first measurement point 41 and the second measurement point 42 are stably measured over a predetermined period of time or when the distance is stably measured over a predetermined period of time. In this way, it is possible to construct an image processing device 10 that quantifies the stability of distance measurement depending on the period of time.

[0089] Furthermore, the processor 100 may determine that the state is stable when at least the amount of movement of the first measurement point 41 is within a first predetermined range within a first predetermined period. In this way, a criterion for determining the stability of distance measurement can be established using the calculated position information of the first measurement point 41.

[0090] Furthermore, the processor 100 may determine that the state is stable when the amount of change in distance is within a second predetermined range within a second predetermined period. In this way, a criterion for determining the stability of distance measurement can be established using information on the measured distance.

[0091] The method of this embodiment is not limited to the above and can be modified in various ways. For example, the display format of the icon indicating the period during which the treatment tool is determined to be stable can be modified in various ways. For example, in addition to the bar graph icon shown in B11 of FIG. 15, the icon may be a pie chart icon shown in B12 or a meter display icon shown in B13. The meter display icon in B13 may further include an icon indicating the ratio of the first count value to a first predetermined value as shown in B14. Furthermore, the icon is not limited to a graph icon, and may be, for example, a fraction display icon as shown in B15. For example, the number on the denominator side shown in B16 corresponds to the first predetermined value, and the number on the numerator side shown in B17 corresponds to the first count value. This allows the user to easily determine the ratio of the period during which the treatment tool is determined to be stable to the period during which it is desirable for the treatment tool to be stable.

[0092] Furthermore, the display mode of the measured distance information is not limited to the mode shown in A11 of Fig. 10 and A21 of Fig. 12, and various modifications are possible. For example, as shown in Fig. 16, patterns of the display mode of the measured distance information may be stored in a memory (not shown), and the user may be allowed to select a pattern of the display mode.

[0093] Pattern P in Figure 16 -B In the case of the treatment tool being unstable, the display mode is a mode in which the measured number shown in B21 is iconized and a symbol icon shown in B22 is added. -B In FIG. 16, the display mode when the treatment tool is stable is the mode in which the measured number shown in B23 is iconized. By looking at the symbol-type icon shown in B22, the user can understand that the treatment tool is not stable. Alternatively, when comparing the icon shown in B21 with the icon shown in B23, the icon shown in B23 is displayed in bold. This makes it possible to make the visibility of the icon shown in B23 higher than that of the icon shown in B21, thereby suggesting to the user that the treatment tool is stable. Similarly, in the case of pattern P in FIG. 16, -BIn pattern P2 of FIG. 16, the icon when the treatment tool is stable is displayed in a darker color than the icon when the treatment tool is not stable, which indicates to the user that the treatment tool is stable. -B 3 is pattern P -B This is a display pattern in which a graph icon is combined with the icon B1. The graph icon shown in B24 indicates that the treatment tool is not stable, and the graph icon shown in B25 indicates that the treatment tool is stable. -B 4 is different from pattern P in that it does not display distance information if the treatment tool is not stable. -B Different from 1.

[0094] In this way, in the image processing device 10 of this embodiment, the processor 100 changes the display mode of the distance measurement value depending on whether or not the distance measurement is stable, thereby allowing the user to easily determine whether or not the distance measurement is stable.

[0095] The processor 100 may also display the period of stable distance measurement in a graph, allowing the user to visually grasp the stability of the distance measurement.

[0096] Furthermore, for example, when the stability of the treatment tool is insufficient, the processor 100 may perform processing to display the example screen shown in FIG. 17 on the second display DP2. In FIG. 17, the second display DP2 displays a screen shown in A40, an icon shown in A43, an icon shown in A44, and an icon shown in A45. The screen shown in A40 displays the first treatment tool 31 as well as an icon shown in A46. The icon shown in A46 suggests to the user at what position the tip of the first treatment tool 31 should be positioned to stabilize the tip of the first treatment tool 31. The icons shown in A43, A44, and A45 in FIG. 17 correspond to the icons shown in A13, A14, and A15 in FIG. 10, respectively. In addition, in FIG. 17, pattern P-B4 in FIG. 16 is used as the display mode of the icon for distance information. In other words, because the stability of the treatment tool is insufficient, the icon for the measured distance information is not displayed on the second display DP2 in FIG. 17.

[0097] For example, although not shown in a flowchart, when the result of step S254 is YES, the processor 100 may perform step S256 and also perform processing to store predetermined data in a memory (not shown). The predetermined data is, for example, position information of the tip of the first treatment tool 31 related to the frame for which the result of step S254 is determined to be YES. The processor 100 then searches for a frame including the predetermined data from the information of past frames stored in the memory, and performs processing to generate aggregate information of position information of the tip of the first treatment tool 31 based on the predetermined data of the searched frame.

[0098] Then, in step S290, processor 100 performs a process of generating predetermined graphic image data based on the aggregate information. The predetermined graphic image data can be said to be image information for keeping the movement amount of first measurement point 41 within a first predetermined range.

[0099] Then, in step S300, the processor 100 performs a process of superimposing and displaying an icon shown in A46 in Fig. 17 on the image of A40. Note that although the icon shown in A46 is a circular icon, it may be an icon made of other shapes such as an ellipse or a polygon.

[0100] In this way, the user can easily stabilize the tip of the first treatment tool 31. As a result, the example screen of FIG. 17 becomes like the example screen of FIG. 18. In FIG. 18, the second display DP2 displays a screen shown in A50, an icon shown in A53, an icon shown in A54, and an icon shown in A55. In addition, the screen shown in A50 displays the first treatment tool 31, as well as an icon shown in A51, an icon shown in A52, and an icon shown in A56. Since the tip of the first treatment tool 31 is stable, the icons shown in A51, A52, A53, A54, and A55 of FIG. 18 are displayed in the same manner as the icons shown in A21, A22, A23, A24, and A25 of FIG. 11, respectively. In this situation, since the tip of the first treatment tool 31 is stable, the movement amount of the position coordinates of the tip of the first treatment tool 31 in each past imaging frame is small. Therefore, the icon shown in A56 in FIG. 18 is displayed smaller than the icon shown in A46 in FIG.

[0101] Furthermore, if the stability of the treatment tool is insufficient, the processor 100 may perform processing to display the example screen shown in Fig. 19 on the second display DP2. In Fig. 19, the second display DP2 displays a screen shown in A60, an icon shown in A63, an icon shown in A64, and an icon shown in A65. Furthermore, the screen shown in A60 displays the first treatment tool 31, as well as icons shown in A66, A67, and A68.

[0102] For example, although illustration of a flowchart or the like is omitted, in step S290, the processor 100 performs processing to generate display data for a predetermined dot-like icon based on position information of the tip of the first treatment tool 31 included in the data of a frame for which the answer to step S254 was determined to be YES, among data of past frames stored in a memory (not shown). Then, in step S300, the processor 100 performs processing to superimpose the icons shown in A66, A67, and A68 on the image of A60 based on the display data generated in step S290.

[0103] Although not shown, when the stability of the treatment tool is insufficient, the icons A66, A67, and A68 in FIG. 19 and the icon A46 in FIG. 17 may be displayed together.

[0104] 17, 18, and 19, the second measurement point 42 is the point selected by the user on the touch panel in step S122 of Fig. 7, but the techniques shown in Fig. 17 and 18 may also be applied when the tip of the second treatment tool 32 is set as the second measurement point 42. For example, if the tip of the first treatment tool 31 and the tip of the second treatment tool 32 are both unstable, the icon shown in A46 in Fig. 17 or the icons shown in A66, A67, and A68 in Fig. 19 may be displayed at the tip of the first treatment tool 31 and the tip of the second treatment tool 32, respectively.

[0105] As described above, in the image processing device 10 of this embodiment, the processor 100 displays on the display DP (second display DP2) image information for keeping at least the amount of movement of the first measurement point 41 within the first predetermined range. In this way, the image processing device 10 can provide the user with assistance in stabilizing distance measurement.

[0106] Also, for example, the measurement process (step S200) may be performed as in the example process shown in the flowchart of Fig. 20. Fig. 20 differs from Fig. 8 in that an image stability determination process (step S210) and step S230 are added. Note that, in the following, explanations of processes similar to those already explained will be omitted as appropriate.

[0107] 20, the processor 100 performs an image stability determination process (step S210), and then performs a process of determining whether or not the image is stable (step S230). The image stability determination process (step S210) and step S230 will be described in detail later, but the processor 100 determines whether or not the imaging state by the imager of the endoscope 20 is stable. If the processor 100 determines that the image is stable (YES in step S230), it performs a treatment tool stability determination process (step S250), steps S270, and S290, as in FIG. 8, and ends the flow.

[0108] On the other hand, if the processor 100 determines that the image is not stable (NO in step S230), it performs step S290 and ends the flow. In other words, in Fig. 20, if the image is not stable, the stability of the first treatment tool 31 etc. is determined, and the endoscopic image related to the frame is updated without measuring the distance between the first measurement point 41 and the second measurement point 42.

[0109] Note that the processing example of the measurement processing (step S200) including the image stability determination processing (step S210) is not limited to that shown in FIG. 20 and various modifications are possible. For example, even if the result of step S230 is NO, the processor 100 may determine the stability of the first treatment tool 31 and measure the distance between the first measurement point 41 and the second measurement point 42. In this case, although not shown in the flowchart, step S230 may be omitted in FIG. 20. Alternatively, although not shown in the flowchart, the processor 100 may perform the image stability determination processing (step S210) and the treatment tool stability determination processing (step S250) in parallel in the measurement processing (step S200) and then perform step S230. In this case, if the processor 100 determines YES in step S230, it may perform step S270, and if the processor 100 determines NO in step S230, it may perform step S290.

[0110] A more detailed example of the image stability determination process (step S210) will be described using the flowchart in Fig. 21. The processor 100 performs a process to determine whether the endoscopic image includes features that are unsuitable for measurement (step S212). Features that are unsuitable for measurement in the endoscopic image include events that are unsuitable for stereo matching, such as being out of focus, lack of brightness throughout the image, or lack of contrast throughout the image.

[0111] When the processor 100 determines that the endoscopic image does not contain features unsuitable for measurement (NO in step S212), it increments the second count value by 1 (step S214). For example, when the processor 100 determines YES in step S214 for the next captured frame, it further increments the second count value by 1. In other words, as long as the endoscopic image is stable, the second count value is accumulated each time the number of captured frames increases. In other words, the second count value has the technical significance of being an index indicating the duration of time during which a state suitable for measurement has continued. Note that a second predetermined value may be further set as an upper limit of the second count value. In this way, when the second count value is equal to or greater than the second predetermined value, the user can determine that a state suitable for measuring the distance between the first measurement point 41 and the second measurement point 42 has continued for a sufficient period of time.

[0112] On the other hand, if the processor 100 determines that the endoscopic image contains features that are not suitable for measurement (YES in step S212), it sets the second count value to 0 (step S216). For example, when the second count value continues to be incremented due to a NO determination in step S212, a YES determination may be made in step S212 due to a predetermined circumstance. The predetermined circumstance may be, for example, a sudden and significant change in the appearance of the endoscopic image due to an external factor such as mist, or a significant shaking in the operation of the user holding the endoscope 20. When such a circumstance occurs, it is considered appropriate to set the second count value to 0 so as to restart the determination of the image stability from the beginning.

[0113] 21, step S216 may be appropriately determined by the user. For example, the second count value may be decremented by a predetermined number. Alternatively, the second count value may not be incremented or decremented. In other words, if the processor 100 determines NO in step S212, the processor 100 may proceed to step S218 without executing step S214.

[0114] After performing step S214 or step S216, processor 100 determines the stability of the image according to the second count value (step S218). Step S218 is the same process as step S259 described above. For example, processor 100 calculates the ratio of the accumulated second count value to a second predetermined value to determine the degree of stability of the endoscopic image. Then, in a subsequent step S290, processor 100 generates display data corresponding to the determination result of step S218, and in a subsequent step S300, displays an image corresponding to the display data generated in step S290 on second display DP2.

[0115] FIG. 22 shows an example of a screen when the processes of FIGS. 20 and 21 are applied. In FIG. 22, the screen shown in A70, the icon shown in A73, the icon shown in A74, and the icon shown in A75 are displayed on the second display DP2. The icon shown in A73 is a pie chart icon indicating the ratio of the period during which the endoscopic image is determined to be stable to the period during which it is desirable for the endoscopic image to be stable. In other words, the ratio of the second count value accumulated in step S214 to the second predetermined value is displayed using a graph icon. The icon shown in A74 is a text icon related to the stability of the endoscopic image. The icon shown in A75 is a symbol icon that succinctly indicates the degree of stability of the endoscopic image. In the situation shown in FIG. 22, the second count value added in step S214 of FIG. 21 has not reached the predetermined ratio to the second predetermined value, so the stability of the screen is insufficient and it is not suitable for measuring the distance between the first measurement point 41 and the second measurement point 42. The pie chart icon shown in A73 visually indicates the ratio of the second count value to the second predetermined value and that the second count value has not reached the predetermined ratio to the second predetermined value. The icons shown in A74 and A75 simply indicate that the endoscopic image is not stable.

[0116] The screen example in Fig. 22 is an example of a screen in which the process of step S230 in Fig. 20 is omitted. Therefore, steps S250 and S270 in Fig. 20 are performed, and the distance information icon shown in A71 in Fig. 22 and the arrow icon shown in A72 are displayed together. The distance information icon shown in A71 is the same as the icon A11 in Fig. 10. In Fig. 22, to simplify the explanation, the icons A13, A14, and A15 in Fig. 10 are not shown, but these icons may be displayed on the second display DP2.

[0117] Fig. 23 is another example of a screen when the processes of Figs. 20 and 21 are applied. In Fig. 23, the screen shown in A80, the icon shown in A83, the icon shown in A84, and the icon shown in A85 are displayed on the second display DP2. In the situation shown in Fig. 23, the second count value added in step S214 of Fig. 21 has reached the second predetermined value, so the stability of the endoscopic image is sufficient and it is suitable for measuring the distance between the first measurement point 41 and the second measurement point 42. The pie chart icon shown in A83 indicates that the second count value has reached the second predetermined value. The icons shown in A84 and A85 simply indicate that the endoscopic image is stable.

[0118] It should be noted that the icon modes shown in A73 and A75 of Fig. 22 are not limited to those shown in Fig. 22. Similarly, the icon modes shown in A83 and A85 of Fig. 23 are not limited to those shown in Fig. 23. For example, the icon modes shown in A73 of Fig. 22 and A83 of Fig. 23 may appropriately adopt the example described above in Fig. 15. Similarly, the icon modes shown in A75 of Fig. 22 and A85 of Fig. 23 may appropriately adopt the pattern described above in Fig. 12.

[0119] As described above, in the image processing device 10 of this embodiment, the processor 100 calculates a final value of the distance when the endoscopic image is stable for a predetermined period of time. The processor 100 also displays on the display DP (second display DP2) a message indicating that the endoscopic image is stable for the predetermined period of time. This allows the user to easily understand that the endoscopic image is stable.

[0120] Furthermore, the image stability determination process (step S210) may be performed as shown in the flowchart of FIG. 24. In FIG. 24, the processor 100 performs a process (step S222) of determining whether the first treatment tool 31 is displayed horizontally. For example, the processor 100 performs a process of calculating a direction vector parallel to the longitudinal direction of the first treatment tool 31 based on the treatment tool mask image of the first treatment tool 31 specified in step S110, and a process of calculating an angle between the direction vector and the X-axis direction. If the calculated angle is within a predetermined range, the processor 100 determines that the first treatment tool 31 is displayed horizontally, thereby realizing the process of step S222. Note that the predetermined range may be changed as appropriate depending on the stereo matching conditions.

[0121] If the processor 100 determines that the first treatment tool 31 is not displayed horizontally (NO in step S222), it performs a process of determining that the image is stable (step S224) and ends the flow. On the other hand, if the processor 100 determines that the first treatment tool 31 is displayed horizontally (YES in step S222), it performs a process of determining that the image is not stable (step S226) and ends the flow. In the subsequent step S290, the processor 100 performs a process of generating image data corresponding to step S224 or step S226. Then, in the subsequent step S300, the processor 100 performs a process of displaying the image data generated in step S290 on the second display DP2. Note that the image stability determination process (step S210) may be a combination of the process of FIG. 21 and the process of FIG. 24.

[0122] Fig. 25 is an example of a screen when the processing example of Fig. 24 is applied. Note that Fig. 25 omits the display of the icons shown in A81, A82, A83, A84, and A85 in Fig. 23 and the icons showing the first measurement point 41 and the second measurement point 42, but these may be displayed.

[0123] In FIG. 25, the screen shown in A90 is displayed on the second display DP2. In the screen shown in A90, the first treatment tool 31 is displayed parallel to the horizontal direction. In other words, the first treatment tool 31 is displayed parallel to the parallax direction. Therefore, the processor 100 determines YES in step S222 of FIG. 24 and performs step S226. As a result, the image shown in A97 is displayed superimposed on the image shown in A90 in steps S290 and S300. The image shown in A97 includes an instruction display indicating that the first treatment tool 31 is displayed horizontally on the screen and urging the user to tilt the first treatment tool 31 with respect to the horizontal direction. Note that tilting the first treatment tool 31 with respect to the horizontal direction means, for example, displacing the first treatment tool 31 by the user operating the first treatment tool 31. However, this is not limited to this, and may also mean, for example, rotating the scope of the endoscope 20 by the user operating the endoscope 20. For example, if the tip of the first treatment tool 31 is already positioned at the desired position, it is more appropriate to rotate the scope of the endoscope 20 rather than displacing the first treatment tool 31. Thus, in the image processing device 10 of this embodiment, when the first treatment tool 31 is parallel to the parallax direction in the stereoscopic view of the endoscopic image, the processor 100 displays on the display DP (second display DP2) an instruction display to tilt the first treatment tool 31 with respect to the parallax direction. This improves the accuracy of stereo matching. Because treatment tools have little texture, the accuracy of stereo matching may be reduced if the entire treatment tool is projected horizontally. Note that texture here refers to a pattern, design, or the like that appears due to the brightness and darkness of each pixel in the image. In this regard, applying the method of this embodiment can prevent a decrease in the accuracy of stereo matching.

[0124] 25A and 25B, an image shown in A98 may be displayed instead of the image shown in A97. Alternatively, the image shown in A97 and the image shown in A98 may be displayed together. The image shown in A98 is an object image indicating the direction in which to rotate the scope of the endoscope 20 in order to prompt the user to rotate the scope of the endoscope 20. That is, in the image processing device 10 of this embodiment, the processor 100 displays on the display DP (second display DP2) an object indicating the direction in which to tilt the first treatment tool 31 when the first treatment tool 31 is parallel to the parallax direction in the stereoscopic view of the endoscopic images. In this way, the user can easily determine the operation required to further improve the construction accuracy of the stereoscopic images.

[0125] Note that Figures 24 and 25 are examples of measuring distance using the first treatment tool 31, but when measuring distance using the first treatment tool 31 and the second treatment tool 32 as described above in Figure 14, the processing of Figure 24 may also be applied to the second treatment tool 32 to display the images shown in A97 and A98 of Figure 25, and various modifications are possible.

[0126] Furthermore, for example, the measurement process (step S200) of this embodiment may be performed as in the example process shown in the flowchart of Fig. 26. The example process of Fig. 26 differs from the example process of Fig. 8 in that smoothing process (step S280) is further performed after step S270.

[0127] The smoothing process (step S280) may be, for example, a process of smoothing the measured distance in the time direction. More specifically, for example, the processor 100 performs a process of calculating an average value of the distance information measured in step S270 and distance information in past frames stored in a memory (not shown). The average value here is an arithmetic mean value, for example, a value obtained by dividing the sum of the distance information stored in a memory (not shown) by the number of frames. The processor 100 then generates display data for the distance information based on the average value calculated in step S290. As a result, an icon representing the smoothed distance information is displayed on the second display DP2 (not shown), although this is not shown in the figure.

[0128] 26, the number of previous captured frames required to calculate the average value is preferably the same as the first predetermined value. In this way, more accurate distance information can be displayed on the second display DP2. Note that the three-dimensional position information of the first measurement point 41 and the second measurement point 42 calculated in step S130 may also be smoothed in the time direction.

[0129] In other words, smoothing the measured distance requires that the distance measurement be stable for a predetermined period of time, which may be, for example, the first predetermined period described above in FIG. 9 or the second predetermined period described above in FIG. 13.

[0130] Alternatively, instead of step S280, a process of smoothing the three-dimensional position information of the first measurement point 41 with respect to time may be performed. In this case, before performing step S270, a process of smoothing the three-dimensional position information of the first measurement point 41 with respect to time is performed. The same applies to the second measurement point 42.

[0131] Also, for example, a technique of smoothing in the spatial direction may be applied. More specifically, for example, by performing the measurement process (step S200) as in the processing example of Fig. 27, the Z coordinate of the three-dimensional position information of the first measurement point 41 may be smoothed in the spatial direction. The processing of Fig. 27 differs from Fig. 8 in that it further includes Z coordinate correction processing (step S260).

[0132] FIG. 28 is a detailed flowchart of the Z coordinate correction process (step S260). The processor 100 performs a process of setting a region of interest (step S262). Specifically, as shown in B30 of FIG. 29(A), the region of interest is set around the first measurement point 41. The processor 100 then performs a process of determining whether or not there is a past frame that can be used for correction (step S264). More specifically, the processor 100 determines whether or not data of a frame for which a YES determination has been made in step S254 of FIG. 9 has been made is stored in a memory (not shown).

[0133] If the processor 100 determines that there is a past frame available for correction (YES in step S264), it performs a process to correct the Z coordinates of the past frame and the current frame (step S265). On the other hand, if the processor 100 determines that there is no past frame available for correction (NO in step S264), it performs a process to correct the Z coordinate of the current frame (step S266).

[0134] More specifically, correcting the Z coordinate in step S266 refers to calculating the average or median of the Z coordinates of pixels associated with the distal edge of the first treatment tool 31 within the region of interest of the stereo image of the current frame. The pixels associated with the distal edge of the first treatment tool 31 within the region of interest are, for example, pixels associated with the region B31 in FIG. 29(A). Whether to use the average or median is up to the user. However, if the distribution of the Z coordinate data tends to differ from a normal distribution due to, for example, significant noise, the median is more appropriate than the average. For example, if the Z coordinate consists of five data values ​​(1, 3, 5, 6, 20), the average is 7 and the median is 5. However, if the data with Z=20 is treated as noise, the median is considered more appropriate than the average.

[0135] To correct the Z coordinate in step S265, for example, a calculation similar to that in step S266 may be performed for each of the past frames, and the median value of the Z coordinates calculated for each frame may be used.

[0136] The processor 100 then performs a process (step S268) to calculate three-dimensional position information of the first measurement point 41. For example, the process of step S268 can be realized by using a method of perspectively projecting the three-dimensional position information of the first measurement point 41 onto the reference image plane. For example, the position shown in B41 in FIG. 29(B) is the origin of the reference camera, the position shown in B42 is the three-dimensional position of the first measurement point 41 after correction, and the position shown in B43 is a position indicating the Z coordinate of the first measurement point 41 after correction. In the reference image plane shown in B40, the position shown in B44 is the position corresponding to the first measurement point 41 in the reference image, and the point shown in B45 is the intersection of the Z axis of the reference camera and the reference image plane. The line of sight vector LV is a vector directed from the origin position shown in B41 to the position shown in B44 and further directed to the three-dimensional position of the first measurement point 41 shown in B42.

[0137] In this case, the position information shown in B44 is known in step S100, and the Z coordinate of the position shown in B45 is known because it is the Z coordinate corrected in step S265 or step S266. Furthermore, the triangle formed by points B41, B42, and B43 is similar to the triangle formed by points B41, B44, and B45. Therefore, by using camera parameters such as focal length, the scale of the line of sight vector LV can be changed and the three-dimensional position information of the point shown in B42, i.e., the three-dimensional position information of the corrected first measurement point 41, can be obtained.

[0138] As described above, in the image processing device 10 of this embodiment, the processor 100 calculates three-dimensional position information of the first measurement point 41 using the line-of-sight vector LV, which is based on the median or average depth coordinates in the area surrounding the measurement position coordinate of the first measurement point 41 in the endoscopic image and the measurement position coordinate. This allows for more accurate calculation of the position information of the first measurement point 41 through stereo matching. Because treatment tools are made of metal and have little texture, reflections and glare occur, resulting in significant variation in the Z coordinate after stereo matching. In this regard, by applying the method of this embodiment, the Z coordinate can be spatially smoothed without smoothing the X and Y coordinates of the first measurement point 41, thereby further improving the calculation accuracy of the first measurement point 41. This allows for more accurate calculation of the measured distance.

[0139] Note that even when the tip of the second treatment tool 32 is set as the second measurement point 42, the above-described smoothing process in the time direction or the smoothing process in the space direction can be applied.

[0140] For the above reasons, in the image processing device 10 of this embodiment, the processor 100 measures the distance by performing smoothing processing in the time direction or the space direction on at least the first measurement point 41. In this way, the accuracy of distance measurement can be further improved.

[0141] 30, a position away from the position of the tip of the first treatment tool 31 may be designated as the first measurement point 41. In the screen example of A100, the second measurement point 42 is illustrated as the point designated in step S120 of FIG.

[0142] For example, the processor 100 can realize the screen example shown in A100 of FIG. 30 by performing processing that is a modification of the first measurement point designation processing (step S1110) shown in FIG. 31. In FIG. 31, the processor 100 functions as the measurement point selection unit 114 and performs processing (step S1112) to select a first treatment tool 31 from the recognized treatment tools. Step S1112 in FIG. 31 is processing that corresponds to steps S112 and S114 in FIG. 6. Then, the processor 100 performs processing (step S1114) to detect the tip portion and base portion of the selected first treatment tool 31. The base portion of the first treatment tool 31 is the base-most portion of the first treatment tool 31 that can be displayed on the second display DP2.

[0143] Thereafter, the processor 100 performs a process of calculating three-dimensional position information (step S1116). Step S1116 is the same process as the three-dimensional position information calculation process (step S130) of Fig. 5 etc. More specifically, for example, the processor 100 generates a treatment tool mask image related to the first treatment tool 31 in step S1112, and performs stereo matching based on the treatment tool mask image and stereo matching based on images of the subject obtained by excluding the portion related to the first treatment tool 31 from the stereo images in step S1116.

[0144] In this way, the processor 100 performs steps S1112, S1114, and S1116, thereby constructing a three-dimensional structure of the first treatment tool 31, for example, as shown in B51 of Fig. 32. Note that the area shown in B52 conceptually shows a part of the three-dimensional structure of the subject excluding the part related to the first treatment tool 31. Note that Fig. 32 does not specifically specify the structure of the first treatment tool 31, etc.

[0145] Returning to the flowchart of FIG. 31, the description will be continued. Thereafter, the processor 100 performs a process of calculating a first extension line PL1 (step S1118). Specifically, for example, the tip portion of the first treatment tool 31 detected in step S1114 is constructed three-dimensionally as shown in B53 in step S1116. Similarly, the base portion of the first treatment tool 31 detected in step S1114 is constructed three-dimensionally as shown in B54 in step S1116. Then, in step S1118, the first extension line PL1 is calculated based on the constructed tip portion and base portion. For example, the processor 100 performs a process of calculating a line passing through the coordinates of the center of gravity of the tip portion shown in B53 and the coordinates of the center of gravity of the base portion shown in B54. As a result, the first extension line PL1 becomes substantially the same line as a line passing through the tip of the first treatment tool 31 and parallel to the longitudinal direction of the first treatment tool 31. This makes it possible to accurately determine the direction in which the tip of the first treatment tool 31 is facing, and therefore the first measurement point 41 can be accurately specified at a desired position.

[0146] Processor 100 then performs a process (step S1119) of designating the intersection of first extension line PL1 and the subject as first measurement point 41. As a result, the intersection of first extension line PL1 determined in step S1118 and the subject shown in B52 is determined, as shown in B55 in Fig. 32. In other words, the point shown in B55 is designated as first measurement point 41, and the first measurement point designation process (step S1110) shown in Fig. 31 ends.

[0147] Although not shown in the flowchart, for example, between steps S1114 and S1116, by further specifying the second measurement point 42 using processing similar to step S122 in Figure 7, and then performing processing from step S1116 onwards, the state will be similar to the state at the end of the flow of the position specification calculation processing (step S100).

[0148] Also, although not shown in the flowchart, if it is desired to further specify a second measurement point 42 at a position away from the tip of the second treatment tool 32, the processor 100 selects the first treatment tool 31 and the second treatment tool 32 in step S1112, and performs steps S1114, S1116, S1118, and S1119 for the selected first treatment tool 31 and second treatment tool 32.

[0149] 31 and 32, the first measurement point 41 is specified by using a three-dimensional construction process based on stereo images, but the method of this embodiment is not limited to this, and the first measurement point 41 may be specified on a two-dimensional image. In this case, for example, the first measurement point specification process (step S2110) shown in the flowchart of FIG. 33 may be performed.

[0150] 33, the processor 100 functions as the measurement point selection unit 114, and performs a process (step S2112) of selecting a first treatment tool 31 from the recognized treatment tools, similar to step S1112 in Fig. 31. Then, the processor 100 performs a process (step S2114) of detecting the tip portion and base portion of the selected first treatment tool 31, similar to step S1114 in Fig. 31.

[0151] Thereafter, the processor 100 performs a process of calculating a second extension line PL2 (step S2116). Specifically, for example, in the reference image shown in B60 in FIG. 34, the processor 100 recognizes the area shown in B61 as the first treatment tool 31 in step S2112. Then, in step S2114, the processor 100 detects the tip portion shown in B62 and the base portion shown in B63. Then, in step S2116, the processor 100 calculates the second extension line PL2. For example, the processor 100 performs a process of calculating a line that passes through the coordinates of the center of gravity of the tip portion shown in B62 and the coordinates of the center of gravity of the base portion shown in B63. As a result, the second extension line PL2 becomes a line that passes through the center of gravity of the tip portion of the first treatment tool 31 recognized in the reference image and is substantially the same line as a line that is parallel to the longitudinal direction of the first treatment tool 31. In this way, the method for calculating the second extension line PL2 differs from the method for calculating the first extension line PL1 described above in that the processor 100 does not function as the three-dimensional construction unit 112.

[0152] Returning to the flowchart of Fig. 33, the explanation will be continued. Thereafter, the processor 100 performs a process (step S2118) of designating a predetermined position on the second extension line PL2 as the first measurement point 41. Specifically, for example, as shown at B65 in Fig. 34, the processor 100 performs a process of designating the position obtained by extending the second extension line PL2 by the length shown at B64 from the coordinates of the center of gravity of the tip portion of B62 as the first measurement point 41.

[0153] 34. As with the standard image shown in B60 of FIG. 34, the reference image is also subjected to the process shown in FIG. 33. 5 Similarly, the processor 100 performs a second measurement point designation process (step S120) and a three-dimensional position information calculation process (step S130), thereby completing the position designation calculation process (step S100). As a result, stereo matching based on the stereo images is performed, and three-dimensional position information of the first measurement point 41 is obtained.

[0154] If it is desired to further specify a second measurement point 42 at a position away from the tip of the second treatment tool 32 using the method of Figure 33, a process of selecting the first treatment tool 31 and the second treatment tool 32 is performed in step S2112, and then steps S2114, S2116, and S2118 are performed for the selected first treatment tool 31 and second treatment tool 32.

[0155] From the above, in the image processing device 10 of this embodiment, the processor 100 calculates three-dimensional position information by determining the point where a straight line based on three-dimensional position information of at least a portion of the shaft of the first treatment tool 31 and three-dimensional position information of the tip of the first treatment tool 31 intersects with the subject as the first measurement point 41. In this way, the first measurement point 41 can be specified at any position in an area that the tip of the first treatment tool 31 does not directly reach. For example, while it is desired to specify a desired position on the surface of a specific tissue as the first measurement point 41, there may be cases where the tip of the first treatment tool 31 cannot reach the desired position due to reasons such as an insufficient length of the shaft of the first treatment tool 31. In this regard, by applying the method of this embodiment, a position on an extension of the direction in which the tip of the first treatment tool 31 is facing can be treated as the first measurement point 41.

[0156] Furthermore, the processor 100 may calculate three-dimensional position information of the first measurement point 41 based on a predetermined position on a straight line based on two-dimensional position information of at least a part of the shaft portion of the first treatment tool 31 and two-dimensional position information of the tip of the first treatment tool 31. By doing so, the same effect as above can be obtained.

[0157] Furthermore, for example, when the desired distance is measured, the processor 100 may be configured to track the first measurement point 41 and the second measurement point 42 for a certain period of time. For example, the position designation calculation process (step S100) may be performed as in the processing example shown in the flowchart of FIG.

[0158] In FIG. 35, the processor 100 performs a process (step S102) to determine whether or not a certain time has elapsed since the start of tracking. If the processor 100 determines that the certain time has not elapsed since the start of tracking (NO in step S102), the processor 100 performs a process (step S104) to determine whether or not the measured distance of the previous frame is within a specified range. More specifically, in step S104, the processor 100 determines whether or not the distance measured in the previous measurement process (step S200) is within the range of the target value of the measurement. That is, the specified range in step S104 is the range of the target value when a target distance is to be measured. For example, in a tumor resection procedure, the resection area may be predetermined at a specified distance from the tumor based on the test results before the procedure, etc. In such a case, when measuring the width of the region to be resected, the target distance is set.

[0159] Furthermore, if the processor 100 determines that the measurement distance in the previous frame is not within the specified range (NO in step S104), it performs a first measurement point designation process (step S110), a second measurement point designation process (step S120), and a three-dimensional position information calculation process (step S130), and ends the flow. On the other hand, if the processor 100 determines that the measurement distance in the previous frame is within the specified range (YES in step S104), it performs a process of tracking the first measurement point 41 and the second measurement point 42 in the previous frame (step S106). After executing step S106, the processor 100 continues to track the first measurement point 41 and the second measurement point 42 by image processing until a certain time has elapsed. Note that the image processing method related to tracking is well known, so a description thereof will be omitted.

[0160] On the other hand, if the processor 100 determines that it is within a certain time since the start of tracking (YES in step S102), it performs the above-mentioned three-dimensional position information calculation process (step S130) and ends the flow. The specific length of the certain time may be determined appropriately by the user. If the result of step S102 is YES, tracking of the first measurement point 41 and the second measurement point 42 continues. In this case, the processor 100 does not newly perform the first measurement point designation process (step S110) or the second measurement point designation process (step S120), but performs stereo matching on the stereo image related to the frame newly acquired in the three-dimensional position information calculation process (step S130). Thereafter, the processor 100 superimposes the stereo image subjected to stereo matching on the image information of the first measurement point 41 and the second measurement point 42 related to step S106 performed in the previous frame, and displays them on the second display DP2. If the processor 100 performs step S130 because it determines YES in step S102, it may omit the measurement process (step S200). This is because the measured distance does not change while tracking is in progress. After the above-mentioned certain time has elapsed, the distance information stored in the memory (not shown) may be erased. This allows the processor 100 to determine NO in step S104 after the certain time has elapsed since tracking began.

[0161] An example of a screen when the processing of Fig. 35 is applied will be described using Fig. 36 and Fig. 37. Note that the example of the screen in Fig. 36 and Fig. 37 is an example in which the second measurement point 42 is designated as the tip of the second treatment tool 32, but this does not prevent the processing of Fig. 35 from being applied when the second measurement point 42 is designated in step S122 of Fig. 7.

[0162] For example, it is assumed that the example screen shown in A120 is displayed on the second display DP2 at the first timing. In the example screen shown in A120, the processor 100 designates the tip of the first treatment tool 31 as the first measurement point 41 and the tip of the second treatment tool 32 as the second measurement point 42 through the position designation calculation process (step S100). Then, the processor 100 performs the measurement process (step S200) and the display update process (step S300) to superimpose an image of distance information shown in A121 and an image of an arrow shown in A122 on the endoscopic image, and displays them on the second display DP2.

[0163] Also, the first timing in the screen example shown in A120 is a timing when the distance information related to the image shown in A121 is not within the range of the target value of measurement. In this case, the processor 100 determines NO in step S104 of Fig. 35 and performs the first measurement point designation process (step S110), the second measurement point designation process (step S120), and the three-dimensional position information calculation process (step S130) in this order. In other words, the position designation calculation process (step S100) shown in Fig. 35 is substantially the same as the position designation calculation process (step S100) of Fig. 5 at the first timing shown in the screen example shown in A120.

[0164] Thereafter, the user operates at least one of the first treatment tool 31 and the second treatment tool 32 to change the distance between the first measurement point 41 and the second measurement point 42, and a second timing is reached at which the example screen shown in A130 is displayed. In the example screen shown in A130, as in the example screen shown in A120, the processor 100 designates the tip of the first treatment tool 31 as the first measurement point 41 and the tip of the second treatment tool 32 as the second measurement point 42 by the position designation calculation process (step S100). Furthermore, the processor 100 performs the measurement process (step S200) and the display update process (step S300) to superimpose an image of the distance information shown in A131 and an image of an arrow shown in A132 on the endoscopic image and display them on the second display DP2.

[0165] The second timing in the example screen shown in A130 is the timing when the distance information related to the image shown in A131 falls within the range of the measurement target value. In this case, the processor 100 determines YES in step S104 of FIG. 35 and performs step S106. As a result, the example screen shown in A130 becomes the same as that shown in FIG. 3 7 The screen will look like the example shown in A140.

[0166] Figure 3 7 In the example screen shown in A140, when tracking has started, the processor 100 performs display processing on the second display DP2 so that the display manner of the icon indicating the first measurement point 41 and the display manner of the icon indicating the second measurement point 42 change as shown in A141 and A142. This allows the user to recognize that step S106 has been executed and tracking has started. Note that, just as the display of the icon indicating the first measurement point 41 is not essential as described above, it is not always necessary to change the display manner of the icon indicating the first measurement point 41. The same applies to the icon indicating the second measurement point 42.

[0167] 37 illustrates a method for enabling the user to distinguish whether tracking has started by changing the display manner of the first measurement point 41 and the second measurement point 42. However, this is not limiting. For example, the start of tracking may be notified by a predetermined means without changing the display manner of the first measurement point 41 and the second measurement point 42. The predetermined means may be, for example, displaying a message on the second display DP2 indicating that tracking has started, displaying a mark including a predetermined color on the second display DP2, or outputting a predetermined sound. Furthermore, although not illustrated, after starting tracking, the processor 100 may further display on the second display DP2 the time remaining until tracking ends.

[0168] Then, at a third timing, which is after the second timing and before a certain time has elapsed since the second timing, the example screen shown in A150 is displayed. The user determines that the desired distance has been measured and further operates the first treatment tool 31 for reasons such as performing a subsequent treatment. Therefore, the position of the tip of the first treatment tool 31 at the third timing is different from the position of the tip of the first treatment tool 31 at the second timing. Similarly, the position of the tip of the second treatment tool 32 at the third timing is different from the position of the tip of the second treatment tool 32 at the second timing.

[0169] In this case, based on the determination of YES in step S102 for the frame, the processor 100 performs stereo matching based on the stereo images captured by the imager of the endoscope 20 for the frame. As a result, the positions of the first treatment tool 31 and the second treatment tool 32 in the example screen of A150 differ from the positions of the first treatment tool 31 and the second treatment tool 32 in the example screen of A140. Meanwhile, since the processor 100 executed step S106 for the previous frame, it is tracking the first measurement point 41 and the second measurement point 42 at the second timing. The tracked first measurement point 41 is displayed at the position indicated by A151 at the third timing, and the tracked second measurement point 42 is displayed at the position indicated by A152 at the third timing. The position indicated by A151 is the same as the position indicated by A141, and the position indicated by A152 is the same as the position indicated by A142. Furthermore, since the first measurement point designation process (step S110) is not performed at the third timing, the position of the tip of the first treatment tool 31 does not match the position shown in A151. Similarly, since the second measurement point designation process (step S120) is not performed at the third timing, the position of the tip of the second treatment tool 32 does not match the position shown in A152.

[0170] Note that FIG. 35 illustrates an example of a process in which the first measurement point 41 and the second measurement point 42 are tracked when the desired distance is measured. However, the present invention is not limited to this example. For example, the user may be able to track the first measurement point 41 and the second measurement point 42 at any timing. For example, although not illustrated, the image processing device 10, the endoscope 20, etc. may include an operation button or the like that can be operated by the user, and step S104 in FIG. 35 may be a process in which the processor 100 determines whether or not it has received an instruction signal based on the user's operation of the operation button. From the above, in the image processing device 10 of this embodiment, when a distance measurement value is obtained or an instruction signal is input by the user, the processor 100 tracks the first measurement point 41 and the second measurement point 42 at the time the measurement value was obtained (step S106) and displays the tracking positions of the first measurement point 41 and the second measurement point 42 on the display DP (second display DP2). In this way, the user can perform tasks other than measurement using the treatment tool used for measurement while understanding the desired measurement value. This can improve the convenience of the treatment.

[0171] Also, for example, a first treatment tool 31 to which a predetermined texture (e.g., a pattern) has been applied in advance may be used for treatment, and the three-dimensional position information calculation process (step S130) of Figure 5 may be performed to calculate the three-dimensional position information of the first measurement point 41.

[0172] The treatment tool used in the procedure according to this embodiment includes a metal portion and a handle portion. However, as shown in B70 of FIG. 38, the difference in pixel values ​​of the pixels relating to the metal portion between the standard image and the reference image is small, which may reduce the accuracy of stereo matching. The same is true for the handle portion. Therefore, for example, as shown in B71 or B72, by applying a predetermined texture to the metal portion and the handle portion of the first treatment tool 31, the accuracy of stereo matching can be improved. The predetermined texture is not limited to the example shown in B71 or B72, and various modifications are possible.

[0173] The predetermined texture can be realized, for example, by directly painting the first treatment tool 31 with a color.

[0174] Alternatively, the processor 100 may perform a process of irradiating the first treatment tool 31 with a predetermined pattern light from a light source device included in the tip of the endoscope 20, acquire a standard image and a reference image from the endoscope 20 while the predetermined pattern light is irradiated onto the first treatment tool 31, and perform stereo matching.

[0175] 14, when the position of the tip of the second treatment tool 32 is set as the second measurement point 42, a predetermined texture may be applied to both the first treatment tool 31 and the second treatment tool 32, but a certain degree of effect can be expected even if a predetermined texture is applied only to the first treatment tool 31. For the above reasons, the image processing device 10 of this embodiment includes the first treatment tool 31, to which a texture is applied to improve the measurement accuracy of the first measurement point 41. In this way, the processor 100 can more accurately determine the parallax, and therefore can more accurately calculate the three-dimensional position information of the first measurement point 41. This allows the distance between the first measurement point 41 and the second measurement point 42 to be measured more accurately.

[0176] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term with a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of the image processing device, endoscope system, image processing method, program, etc. are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0177] 1... endoscope system, 10... image processing device, 11... first output Interface, 12...2nd output Interface, 20...endoscope, 31...first treatment tool, 32...second treatment tool, 41...first measurement point, 42...second measurement point, 100...processor, 110...position designation calculation unit, 112...3D construction unit, 114...measurement point selection unit, 120...distance calculation unit, 122...measurement execution determination unit, DP...display, DP1...first display, DP2...second display, LV...line of sight vector, P-A1, P-A2, P-A3, P-A4, P-B1, P-B2, P-B3, P-B4...pattern, PL1...first extension line, PL2...second extension line

Claims

1. a processor that performs display processing on the display; The processor: Using an endoscopic image of a subject acquired by an endoscope, three-dimensional position information of the first treatment tool and the second treatment tool within the endoscopic image is calculated; measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point at a position on the distal end side of the second treatment tool based on the three-dimensional position information of the first treatment tool and the second treatment tool; An image processing device characterized in that a display process is performed on the display to indicate whether or not the distance measurement is stable.

2. a processor that performs display processing on the display; The processor: Using an endoscopic image of a subject acquired by an endoscope, calculate three-dimensional position information of the first treatment tool and a predetermined portion within the endoscopic image; measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point related to the predetermined portion based on the three-dimensional position information of the first treatment tool and the predetermined portion; performing a display process on the display to indicate whether the distance measurement is stable or not; an image processing device that, when the first treatment tool is parallel to the parallax direction in stereoscopic viewing of the endoscopic image, displays on the display an instruction display to tilt the first treatment tool with respect to the parallax direction.

3. a processor that performs display processing on the display; The processor: Using an endoscopic image of a subject acquired by an endoscope, calculate three-dimensional position information of the first treatment tool and a predetermined portion within the endoscopic image; measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point related to the predetermined portion based on the three-dimensional position information of the first treatment tool and the predetermined portion; performing a display process on the display to indicate whether the distance measurement is stable or not; an image processing device that displays, on the display, an object indicating a direction in which the first treatment tool is tilted when the first treatment tool is parallel to a parallax direction in stereoscopic viewing of the endoscopic image;

4. a processor that performs display processing on the display; The processor: Using an endoscopic image of a subject acquired by an endoscope, calculate three-dimensional position information of the first treatment tool and a predetermined portion within the endoscopic image; measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point related to the predetermined portion based on the three-dimensional position information of the first treatment tool and the predetermined portion; performing a display process on the display to indicate whether the distance measurement is stable or not; An image processing device characterized in that the distance is measured by performing a smoothing process to determine the three-dimensional position information of the first measurement point using a gaze vector based on the median or average depth coordinates in the surrounding area of ​​the measurement position coordinates of the first measurement point in the endoscopic image and the measurement position coordinates.

5. a processor that performs display processing on the display; The processor: Using an endoscopic image of a subject acquired by an endoscope, calculate three-dimensional position information of the first treatment tool and a predetermined portion within the endoscopic image; measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point related to the predetermined portion based on the three-dimensional position information of the first treatment tool and the predetermined portion; performing a display process on the display to indicate whether the distance measurement is stable or not; An image processing device characterized in that the three-dimensional position information is obtained by using a point where a straight line based on the three-dimensional position information of at least a portion of the shaft portion of the first treatment tool and the three-dimensional position information of the tip of the first treatment tool intersects with the subject as the first measurement point.

6. a processor that performs display processing on the display; The processor: Using an endoscopic image of a subject acquired by an endoscope, calculate three-dimensional position information of the first treatment tool and a predetermined portion within the endoscopic image; measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point related to the predetermined portion based on the three-dimensional position information of the first treatment tool and the predetermined portion; performing a display process on the display to indicate whether the distance measurement is stable or not; an image processing device that calculates the three-dimensional position information of the first measurement point based on a predetermined position on a straight line based on two-dimensional position information of at least a portion of the shaft portion of the first treatment tool and the two-dimensional position information of the tip of the first treatment tool.

7. 7. The image processing device according to claim 2, The three-dimensional position information of the predetermined portion is the three-dimensional position information of the second measurement point, which is a position of the subject corresponding to a position on the display designated by a user; The processor: an image processing device that measures the distance between the first measurement point on the tip side of the first treatment tool and the second measurement point based on the three-dimensional position information of the first treatment tool and the specified part.

8. 7. The image processing device according to claim 1, The processor: An image processing device characterized in that it determines that the measurement of the distance is in the stable state when the first measurement point and the second measurement point are measured stably over a predetermined period of time or when the distance is measured stably over the predetermined period of time.

9. 9. The image processing device of claim 8, The processor: An image processing apparatus characterized in that it determines that the stable state is present when the amount of movement of at least the first measurement point is within a first predetermined range within a first predetermined period.

10. 10. The image processing device of claim 9, The processor:

10. An image processing device comprising: displaying, on the display, image information for keeping the movement amount of at least the first measurement point within the first predetermined range.

11. 9. The image processing device of claim 8, The processor: The image processing device is characterized in that it determines that the stable state is present when the amount of change in the distance is within a second predetermined range within a second predetermined period.

12. 7. The image processing device according to claim 1, The processor:

10. An image processing device comprising: an image processing device that changes a display mode of the measured value of the distance depending on whether the measurement of the distance is in the stable state or not.

13. 7. The image processing device according to claim 1, The processor:

2. An image processing device comprising: a display unit for displaying a period during which the distance measurement is in a stable state as a graph;

14. 7. The image processing device according to claim 1, The processor: calculating a definitive value of the distance when the endoscopic image is stable for a predetermined period of time; An image processing device characterized in that a display indicating that the endoscopic image is stable is displayed on the display during the predetermined period.

15. 7. The image processing device according to claim 1, The processor: An image processing device characterized in that when the distance measurement value is obtained or an instruction signal is input by the user, tracking is performed between the first measurement point and the second measurement point when the measurement value was obtained, and the tracking positions of the first measurement point and the second measurement point are displayed on the display.

16. The image processing device according to any one of claims 1 to 6, An endoscope and An endoscope system comprising:

17. A process of displaying an endoscopic image of a subject acquired by the endoscope on a display; a process of calculating three-dimensional position information of a first treatment tool and a second treatment tool within the endoscopic image using the endoscopic image of the subject; a process of measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point at a position on the distal end side of the second treatment tool based on the three-dimensional position information of the first treatment tool and the second treatment tool; and displaying an indication of whether the distance measurement is stable or not. An image processing method characterized by being executed by a computer.

18. A process of displaying an endoscopic image of a subject acquired by the endoscope on a display; a process of calculating three-dimensional position information of a first treatment tool and a second treatment tool within the endoscopic image using the endoscopic image of the subject; a process of measuring a distance between a first measurement point on the distal end side of the first treatment tool and a second measurement point at a position on the distal end side of the second treatment tool based on the three-dimensional position information of the first treatment tool and the second treatment tool; and displaying an indication of whether the distance measurement is stable or not. A program that is executed by a computer.

Images