Image processing device, endoscope system, image processing method, and program

Abstract

To provide an image processing device, etc., that displays information regarding whether the measurement of the distance between specified points, such as the tip of a robotic tool used in surgical procedures, is stable. [Solution] The image processing device 10 includes a processor 100 that performs display processing on the display DP. The processor 100 uses the endoscopic image of the subject acquired by the endoscope 20 to calculate three-dimensional positional information of the first treatment instrument 31 and a predetermined part within the endoscopic image (step S100). The processor 100 also measures the distance between a first measurement point 41 on the tip side of the first treatment instrument 31 and a second measurement point 42 relating to the predetermined part based on the three-dimensional positional information of the first treatment instrument 31 and the predetermined part (step S200), and performs display processing (step S300) on the display DP (second display DP2) to indicate whether the distance measurement is in a stable state.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an endoscope system, an image processing method, and a program.

Background Art

[0002] Conventionally, an image processing apparatus that displays an image acquired from an endoscope in a surgical operation or the like on a display has been known. Patent Document 1 discloses a method of displaying a measurement value obtained by measuring the distance between points specified at the tip of a robotic tool used in a surgical operation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 discloses a method of displaying a measurement value obtained by measuring the distance between points specified at the tip of a robotic tool used in a surgical operation, but does not propose a method of displaying information on whether the distance measurement is stable.

Means for Solving the Problems

[0005] One aspect of the present disclosure relates to an image processing apparatus including a processor that performs display processing on a display, the processor calculating three-dimensional position information of a first treatment tool and a predetermined portion in the endoscope image using the endoscope image of a subject acquired by the endoscope, measuring the distance between a first measurement point on the tip side of the first treatment tool and a second measurement point related to the predetermined portion based on the three-dimensional position information of the first treatment tool and the predetermined portion, and performing display processing on the display to indicate whether the measurement of the distance is in a stable state.

[0006] Other aspects of this disclosure relate to an endoscopic system including the image processing apparatus described above and an endoscope.

[0007] Other aspects of this disclosure relate to an image processing method that causes a computer to perform the following steps: display an endoscopic image of a subject acquired by an endoscope on a display; calculate three-dimensional positional information of a first treatment instrument and a predetermined portion within the endoscopic image using the endoscopic image of the subject; measure the distance between a first measurement point on the tip side of the first treatment instrument and a second measurement point relating to the predetermined portion based on the three-dimensional positional information of the first treatment instrument and the predetermined portion; and display whether or not the measurement of the distance is in a stable state.

[0008] Other aspects of this disclosure relate to a program that causes a computer to perform the following processes: displaying an endoscopic image of a subject acquired by an endoscope on a display; calculating three-dimensional positional information of a first treatment instrument and a predetermined part within the endoscopic image using the endoscopic image of the subject; measuring the distance between a first measurement point on the tip side of the first treatment instrument and a second measurement point relating to the predetermined part based on the three-dimensional positional information of the first treatment instrument and the predetermined part; and displaying whether or not the measurement of the distance is in a stable state. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram illustrating an example configuration of an image processing unit included in an endoscope system. [Figure 2] A diagram illustrating the endoscopic system in more detail. [Figure 3] Another diagram illustrating the endoscopic system in more detail. [Figure 4] A flowchart illustrating an example of the processing method according to this embodiment. [Figure 5] A flowchart illustrating an example of position-based calculation processing. [Figure 6] A flowchart illustrating an example of the process for specifying the first measurement point. [Figure 7]Flowchart for explaining an example of the second measurement point designation process. [Figure 8] Flowchart for explaining an example of the measurement process. [Figure 9] Flowchart for explaining an example of the treatment tool stability judgment process. [Figure 10] Figure for explaining an example of a screen when the treatment tool is not stable. [Figure 11] Figure for explaining an example of a screen when the treatment tool is stable. [Figure 12] Figure for explaining an example of a mode indicating the stability of the treatment tool. [Figure 13] Figure for explaining an example of displaying the measured distance as time-series data. [Figure 14] Figure for explaining an example of a screen when measuring the distance with two treatment tools. [Figure 15] Figure for explaining an example of an icon indicating the period during which the distance measurement is stable. [Figure 16] Figure for explaining an example of the display mode of the measured distance [Figure 17] Figure for explaining an example of a screen including an image showing the range where the tip of the first treatment tool is stable. [Figure 18] Another figure for explaining an example of a screen including an image showing the range where the tip of the first treatment tool is stable. [Figure 19] Figure for explaining an example of a screen including the tip position information of the first treatment tool in the past frame. [Figure 20] Flowchart for explaining another example of the measurement process. [Figure 21] Flowchart for explaining an example of the image stability judgment process. [Figure 22] Figure for explaining an example of a screen including a display indicating the stability of the image. [Figure 23] Another figure for explaining an example of a screen including a display indicating the stability of the image. [Figure 24] Flowchart for explaining another example of the image stability judgment process. [Figure 25] Figure for explaining an example of a screen including an image prompting the rotation of the scope. [Figure 26]A flowchart for explaining another example of measurement processing. [Figure 27] A flowchart for explaining another example of measurement processing. [Figure 28] A flowchart for explaining an example of Z - coordinate correction processing. [Figure 29] (A) is a diagram for explaining the region of interest, and (B) is a diagram for explaining a method of calculating the three - dimensional position information of the first measurement point after correction. [Figure 30] A diagram for explaining an image example including the first measurement point specified by another example of the first measurement point specification process. [Figure 31] A flowchart for explaining another example of the first measurement point specification process. [Figure 32] A diagram for explaining the specification of the first measurement point in more detail. [Figure 33] A flowchart for explaining another example of the first measurement point specification process. [Figure 34] Another diagram for explaining the specification of the first measurement point in more detail. [Figure 35] A flowchart for explaining another example of the position specification calculation process. [Figure 36] A diagram for explaining the tracking of the first measurement point and the second measurement point. [Figure 37] Another diagram for explaining the tracking of the first measurement point and the second measurement point. [Figure 38] A diagram for explaining an example of the texture attached to the first treatment tool.

Mode for Carrying Out the Invention

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail. Note that the embodiments described below do not unduly limit the content described in the claims, and not all of the configurations described in the embodiments are essential constituent elements.

[0011] Figure 1 is a block diagram illustrating an example configuration of the endoscope system 1 of this embodiment. The endoscope system 1 of this embodiment includes an image processing device 10 and an endoscope 20. The image processing device 10 includes a processor 100. The processor 100 of this embodiment is composed of the following hardware. The hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. For example, the hardware may consist of one or more circuit devices or one or more circuit elements mounted on a circuit board. One or more circuit devices are, for example, ICs. One or more circuit elements are, for example, resistors, capacitors, etc.

[0012] Furthermore, for example, the image processing apparatus 10 of this embodiment may be configured to include a memory (not shown in Figure 1) and a processor 100 that operates based on the information stored in the memory. This allows the processor 100 to function as a position specification calculation unit 110, a 3D construction unit 112, a measurement point selection unit 114, a distance calculation unit 120, a measurement execution determination unit 122, etc., as described later. For the sake of clarity, the processor 100 will be used as the main component of the processing related to the method of this embodiment described below. The information stored in the memory includes, for example, programs and various types of data. The processor 100 can be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), etc. The memory may be a volatile memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), a non-volatile memory such as ROM (Read Only Memory), a register, a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, memory stores instructions that can be read by the computer, and when these instructions are executed by the processor 100, the functions of each part are realized as processing. The instructions here may be instructions from the instruction set that makes up the program, or they may be instructions that instruct the hardware circuit of the processor 100 to operate. Memory is also called a storage device.

[0013] Furthermore, the above-mentioned program can be stored in a non-temporary information storage medium, such as a computer-readable medium. This information storage medium can be implemented as, for example, an optical disc, a memory card, a hard disk drive, or non-volatile memory.

[0014] The method of this embodiment can be applied, for example, to procedures in endoscopic surgery as shown in Figure 2. In this case, the endoscopic system 1 can be said to function more specifically as an endoscopic surgical system. Note that Figure 2 is a conceptual diagram and does not strictly illustrate endoscopic surgery, for example, by omitting the illustration of the trocar.

[0015] In Figure 2, the image processing device 10 is connected to the endoscope 20 and also to the display DP. In the endoscopic surgical procedure shown in Figure 2, multiple holes are made in the body wall of the patient, and the endoscope 20 is inserted into the body cavity through one of the holes, while treatment instruments are inserted into the body cavity through the other holes. In this case, the endoscope 20 is a rigid endoscope, with most of the insertion portion being rigid. Although Figure 2 shows the insertion of a first treatment instrument 31 and a second treatment instrument 32 into the body cavity, the number of treatment instruments is not limited to two.

[0016] Furthermore, since rigid endoscopes are well known, detailed illustrations are omitted, but the tip of the rigid endoscope 20 includes an imager. In other words, the image processing device 10 receives an image signal from an imager (not shown) located at the tip of the endoscope 20 via a cable (not shown). In the following description, the endoscope 20 may be referred to as a scope. The image processing device 10 then generates a display image based on the received image signal, as shown in A1 of Figure 2, and displays it on the display DP. In other words, the processor 100 controls the display DP. In this embodiment, the image captured by the imager located at the tip of the endoscope 20 is called an endoscopic image. In the following description, an endoscopic image may be simply referred to as an image. With the endoscopic system 1 configured in this way, the endoscopic image obtained by the endoscope 20 is displayed on the display DP, so that the user can observe the biological tissue in the body cavity and perform procedures on the biological tissue with instruments. In this embodiment, the user refers to, for example, the surgeon handling the instruments, the scopist operating the endoscope 20, and all persons involved in the procedure. In recent years, some endoscopic surgical procedures utilize surgical robot systems where the user controls the robotic arms by operating a console. However, not all facilities employ such systems, and in some cases, the endoscope 20, first treatment instrument 31, second treatment instrument 32, etc., are operated manually by the user.

[0017] Furthermore, although only one display DP is shown in Figure 2, the number of display DPs that the image processing device 10 can connect to is not limited to one. For example, as shown in Figure 3, the image processing device 10 may be able to connect to a first display DP1 and a second display DP2. In this case, the processor 100, which is not shown in Figure 2, may be connected to the first display DP1 via the first interface 11 and to the second display DP2 via the second interface 12. In this case, the processor 100 that displays an image on the first display DP1 and the processor 100 that displays an image on the second display DP2 may be the same or different. In the following explanation, the first display DP1 and the second display DP2 may be referred to separately.

[0018] Furthermore, the display DP of this embodiment may be a display device having a first display mode for displaying stereoscopic images. A stereoscopic image is an image that gives the user a sense of depth. Specifically, this can be achieved by using a parallax barrier type display device, but it may also be achieved using a lenticular method, and various methods can be employed. A parallax barrier type display device is a method of displaying stereoscopic images by using a parallax barrier to allow the user's left eye to view the image for the left eye and the user's right eye to view the image for the right eye. Furthermore, the display DP of this embodiment may be a display device that further has a second display mode for displaying endoscopic images in a planar manner, or it may be configured to allow switching between the first display mode and the second display mode. For example, switching between the first display mode and the second display mode can be achieved by switching the parallax barrier function on and off.

[0019] Furthermore, the display DP of this embodiment may further include a touch panel function. In other words, the display DP of this embodiment may further include a pointing device for indicating any position on the screen. The detection method is not particularly limited and can be a capacitive method, resistive method, ultrasonic method, infrared method, electromagnetic induction method, etc., and can be determined by the user as appropriate. For example, although not shown in the figures, the display DP includes a touch panel control circuit that detects the user's touch operation. The touch panel control circuit detects the user's touch operation and outputs coordinate data on the display DP identified by the user's touch operation to the processor 100. A user's touch operation is, for example, an operation in which the user's finger touches the surface of the display DP, but may further include operations in which the user's finger slides while in contact with the surface of the display DP, operations in which the user's finger lifts off the surface of the display DP, etc. Furthermore, a user's touch operation is not limited to an operation performed by the user's finger, but may also be an operation performed by a stylus or the like held by the user. A stylus is a device that enables touch operations equivalent to those performed by a user's finger. For example, when using a touch panel function that employs a capacitive detection method, the tip of the stylus is configured to include a conductor.

[0020] Furthermore, the images displayed on the first display DP1 and the second display DP2 may be different. Specifically, for example, during a procedure, the image mainly observed by the user may be displayed on the first display DP1, and predetermined information may be displayed on the second display DP2. The predetermined information is information such as the distance between two predetermined measurement points specified by a predetermined method, and the details will be described later. The two predetermined measurement points are, for example, the first measurement point 41 and the second measurement point 42, which will be described later. The method of this embodiment relates to the specification of two measurement points, the measurement of the distance between the two measurement points, and the determination of whether measurement is possible or not.

[0021] In this embodiment, for example, as shown in Figure 10, which will be described later, a screen example to which the method of this embodiment is applied is assumed to be displayed on the second display DP2. However, this is merely an example and does not limit other display methods. For example, a screen similar to the screen shown in Figure 10 may be displayed on a part of the first display DP1. The same applies to the screen examples shown in Figures 11, 14, 17, 18, 19, 22, 23, 25, 30, 36, and 37, which will be described later. Furthermore, although these screen examples are shown in plan view for convenience, this does not prevent them from being displayed as stereoscopic images when actually applying the method of this embodiment.

[0022] Furthermore, in this embodiment, as will be described later, two methods for determining the position information of two predetermined measurement points are exemplified: a first method that calculates based on the tip position of the treatment instrument displayed on the second display DP2, and a second method that calculates based on the position of the subject corresponding to the position on the second display DP2 specified by the user. In this embodiment, for the sake of simplicity, when determining the position information of two predetermined measurement points using both the first and second methods, the measurement point whose position information is determined using the first method will be designated as the first measurement point 41, and the measurement point whose position information is determined using the second method will be designated as the second measurement point 42. Note that, as will be described later in Figure 14, the position information of the first measurement point 41 and the second measurement point 42 can also be determined using only the first method.

[0023] The processor 100 included in the image processing apparatus 10 of this embodiment functions as a position designation calculation unit 110 as shown in Figure 1, and calculates the position information of two predetermined measurement points. The position designation calculation unit 110 also includes a 3D construction unit 112. In other words, the position information of two predetermined measurement points is 3D position information, and as will be described later, the processor 100 can calculate the 3D position information of two predetermined measurement points by functioning as a 3D construction unit 112.

[0024] Furthermore, the processor 100 functions as the distance calculation unit 120 in Figure 1, calculating distance based on the position information calculated by the position specification calculation unit 110. Specifically, the processor 100 calculates the distance between the first measurement point 41 and the second measurement point 42 based on the three-dimensional position information of the first measurement point 41 and the three-dimensional position information of the second measurement point 42. Hereafter, the distance between the first measurement point 41 and the second measurement point 42 may be simply referred to as "distance". The distance calculation unit 120 also includes a measurement execution determination unit 122. In other words, the processor 100 functions as a measurement execution determination unit 122, determining whether the distance measurement is stable enough to perform distance measurement.

[0025] In this embodiment, stable distance measurement specifically means that, for example, the variation in the positional information of the first measurement point 41 and the variation in the positional information of the second measurement point 42 are small. For example, the processor 100 determines whether or not the distance measurement is stable by the treatment instrument stability determination process (step S250) described later. Stable distance measurement may also mean that the variation in the measured distance is small, as will be described later in Figure 13. Furthermore, stable distance measurement may also include stable endoscopic images, as will be described later in detail in Figure 21, etc.

[0026] The processor 100 then functions as a video processor and displays the measured distance information on the display DP. More specifically, using Figure 3 as an example, the processor 100 performs the process of converting the measured distance information into icons and the process of displaying them on the second display DP2 via the second interface 12. The processor 100 can also display information other than the measured distance information, and details will be described later in Figure 10, etc.

[0027] The imager in this embodiment is, for example, a three-dimensional camera, and more specifically, a stereo camera. The stereo camera consists of a reference camera and a reference camera. In this embodiment, the image captured by the reference camera is called the reference image, the image captured by the reference camera is called the reference image, and the set of the reference image and the reference image is called a stereo image. The camera parameters of the reference camera and the camera parameters of the reference camera are adjusted to be equal, with only the position where the reference camera captures images differing from the position where the reference camera captures images. Furthermore, the arrangement of the reference camera and the reference camera is designed such that the optical axis of the reference camera and the optical axis of the reference camera are parallel to each other, and the imaging planes of the reference camera and the imaging planes of the reference camera are on the same plane and aligned horizontally. In actual imaging, slight deviations in the arrangement may occur, so it is possible to correct the image to take these deviations into account.

[0028] In a stereo camera, a reference camera is placed on one side (left or right) and a reference camera on the other, thereby capturing images for the left eye and the right eye. Based on these left and right eye images, the three-dimensional position information of the subject can be calculated. In other words, the processor 100 calculates three-dimensional position information based on the left and right eye images, which are endoscopic images acquired by the endoscope 20. Hereafter, the left side will be referred to as the reference camera and the right side as the reference camera. In other words, hereafter, the left eye image will be referred to as the reference image and the right eye image as the reference image.

[0029] Since the method for acquiring 3D position information using a stereo camera is well known, illustrations are omitted. However, for example, if we denote the desired position of the object being imaged in 3D space as position P, then the 2D coordinates of position P1 on the reference image corresponding to position P and the 2D coordinates of position P2 on the reference image corresponding to position P will not coincide, resulting in a discrepancy. This discrepancy is called parallax, and since the 3D coordinates of position P are unknown, the parallax is also unknown.

[0030] The processor 100 functions as a 3D construction unit 112. When it acquires a stereo image from the endoscope 20, it searches the reference image for the pixel corresponding to the pixel at position P1, i.e., the pixel of point P2. Based on the 2D coordinates of the pixel at position P1 and the 2D coordinates of the found pixel at point P2, it calculates the parallax of point P. This method of calculating parallax is called stereo matching. Once the parallax of point P is determined, the processor 100 calculates the 3D coordinates of point P using the principle of triangulation, based on the position information of the center of the reference camera, the position information of the center of the reference camera, the position information of point P1, and the position information of point P2. The higher the accuracy of the search for the pixel at position P2, the higher the accuracy of the parallax calculation, and the more accurately the 3D coordinates of position P can be determined. By repeating the above calculations for each pixel on the reference image, depth information of the subject and 3D position information of the subject can be obtained.

[0031] Various methods have been proposed as algorithms for stereo matching. In this embodiment, for example, a method that calculates disparity based on semi-global block matching can be adopted, but this does not preclude the use of other algorithms. Furthermore, in order to implement these algorithms in the image processing device 10, programming using a predetermined open-source library can be performed, and the program based on this programming can be stored in the non-temporary information storage medium described above.

[0032] Furthermore, for the sake of clarity, Figure 3 illustrates the X, Y, and Z axes as three mutually orthogonal axes. Hereafter, the terms "X direction," "Y direction," and "Z direction" may be used. The "X direction" is the direction along the X axis, which is parallel to the horizontal direction of the second display DP2. The "Y direction" is the direction along the Y axis, which is parallel to the vertical direction of the second display DP2. The "Z direction" is the direction along the Z axis, which is parallel to the depth direction of the second display DP2. The "X direction" may also be called the "horizontal direction." In this embodiment, when a stereo camera is used as the imager, the reference camera and reference camera are arranged horizontally as described above, so the X direction can also be called the "parallax direction."

[0033] Hereafter, examples of processing related to the method of this embodiment and examples of screens when this processing example is applied will be shown. Although these screen examples are shown assuming the endoscopic surgical procedure described above, the method of this embodiment is not limited to endoscopic surgical procedures using the endoscope 20 as a rigid endoscope. For example, the endoscopic system 1 of this embodiment may be configured to include the endoscope 20 as a flexible endoscope and the image processing device 10. This makes it possible to use the method of this embodiment for the purpose of measuring the distance from the insertion point of the endoscope 20 to a desired position in, for example, an endoscopic examination.

[0034] Figure 4 is a flowchart illustrating an example of processing related to the method of this embodiment. The processor 100 performs a process (step S10) to determine whether or not measurement has started. If the processor 100 determines that measurement has started (YES in step S10), it performs the processing from the position specification calculation process (step S100) onward. If it determines that measurement has not started (NO in step S10), it performs step S10 again.

[0035] The process related to step S10 can be implemented by various methods. For example, the user may press a predetermined button included in the endoscope 20, or a button included in the treatment instrument, or a button provided on the operation panel (not shown) of the image processing device 10, or the user may step on a foot pedal (not shown) connected to the image processing device 10. Alternatively, the image processing device 10 may include a voice command device so that the processor 100 can determine YES in step S10 based on a predetermined voice command. Alternatively, the memory (not shown) may include a gesture recognition program so that when the user operates the treatment instrument so that the tip of the instrument performs a predetermined gesture, the processor 100 can determine YES in step S10 based on the gesture recognition program. Alternatively, when one treatment instrument is detected from the endoscope image and the user touches the touch panel of the second display DP2 as described later, the processor 100 can determine YES in step S10. Alternatively, when multiple treatment instruments are detected from the endoscope image, the processor 100 can determine YES in step S10.

[0036] Subsequently, the processor 100 performs position specification calculation processing (step S100), measurement processing (step S200), and display update processing (step S300), and then performs processing to determine whether or not to terminate the measurement (step S400). If the processor 100 determines that it will not terminate the measurement (NO in step S400), it performs the position specification calculation processing (step S100) again, and if it determines that it will terminate the measurement (NO in step S400), it terminates the flow. Details of the position specification calculation processing (step S100) and measurement processing (step S200) will be described later, but the position specification calculation processing (step S100) calculates the 3D position information of the first measurement point 41 and the second measurement point 42. Then the processor 100 functions as a distance calculation unit 120 and measures the distance between the first measurement point 41 and the second measurement point 42 by the measurement processing (step S200). Then, in step S300, the newly generated or updated image data from the position specification calculation process (step S100) and the measurement process (step S200) is displayed on the second display DP2.

[0037] Then, until an event occurs in step S400 that results in YES, the processor 100 performs position specification calculation processing (step S100), measurement processing (step S200), and display update processing (step S300) at the timing when it acquires one frame period of endoscopic images from the endoscope 20.

[0038] In other words, in this embodiment, the processor 100 continues to measure the distance until an event occurs that results in YES in step S400. An event that results in YES in step S400 is, for example, the user pressing a button to terminate the program related to measurement, or a certain amount of time elapsed since the timing of YES in step S10.

[0039] The position specification calculation process (step S100) will be explained in more detail using the flowchart in Figure 5. Note that the position specification calculation process (step S100) in Figure 5 is the process of determining the first measurement point 41 using the first method described above and determining the second measurement point 42 using the second method described above. In Figure 5, after the processor 100 performs the first measurement point specification process (step S110) and the second measurement point specification process (step S120), it performs the 3D position information calculation process (step S130) and then terminates the flow.

[0040] The first measurement point designation process (step S110) will be explained in more detail using the flowchart in Figure 6. After the processor 100 performs the process of recognizing each treatment instrument in the image (step S112), it performs the process of recognizing the tip of each treatment instrument (step S114). For example, since treatment instruments are made of metal, they have a higher brightness value than tissues among the subjects. Therefore, the processor 100 converts the endoscopic image into a brightness image, detects pixels with a brightness value above a predetermined threshold, groups these pixel clusters by contour detection, and detects groups of a predetermined number of pixels or more as treatment instruments, thereby realizing step S112. Furthermore, the processor 100 can realize step S114 by considering the tip of the pixel group detected in step S112 as the tip of the treatment instrument. Steps S112 and S114 may be realized by other methods; for example, treatment instruments may be recognized using a trained model that has been machine-learned to recognize the presence or absence of treatment instruments in an endoscopic image. In this case, the trained model is, for example, a convolutional neural network (CNN). Furthermore, in endoscopic images, the position of the instrument in the Z direction is located in front of the position of the tissue in the Z direction, so the image of the part related to the instrument can be considered a foreground image, and the image of the part other than the instrument can be considered a background image. Therefore, step S112 may be a process to extract the region related to the instrument as a foreground image from the endoscopic image and obtain an instrument mask image, which is an image in which the region other than the region related to the foreground image is masked. Note that a method for automatically generating a foreground mask image when the region related to the foreground image is specified can be realized by storing a program using a known software library in the non-temporary information storage medium mentioned above.

[0041] Subsequently, the processor 100 performs a process (step S116) to designate the selected tip portion as the first measurement point 41. If multiple treatment instruments are displayed on the second display DP2, in step S114, the processor 100 determines that the number of recognized treatment instrument tip portions is equal to the number of treatment instruments displayed on the second display DP2. The processor 100 then functions as a measurement point selection unit 114 and, in step S116, selects the tip portion of a desired treatment instrument as the first measurement point 41. Step S116 may be, for example, a notification process prompting the selection of the tip portion of a desired treatment instrument, or it may be a process in which the processor 100 selects the tip portion of a predetermined treatment instrument. A predetermined treatment instrument is, for example, a treatment instrument whose tip portion coordinates are closest to the coordinates of the center of the second display DP2. Note that, for example, if the first measurement point designation process (step S110) is performed when only one treatment instrument is displayed on the second display DP2, step S116 may be omitted.

[0042] Figure 7 is a flowchart illustrating an example of the second measurement point specification process (step S120). The processor 100 specifies the second measurement point 42 based on the part specified by the user (step S122). For example, as described above, when the second display DP2 functions as a touch panel, the user touches a desired position among the subjects displayed on the second display DP2. The touch panel control circuit of the second display DP2 then outputs position information data based on the touched position to the processor 100. The processor 100 then converts the position information data received from the touch panel control circuit into position information on the endoscopic image. In other words, the second measurement point 42 is the position of the subject corresponding to the position on the second display DP2 specified by the user.

[0043] Returning to Figure 5, the 3D position information calculation process (step S130) will be explained. As mentioned above, the processor 100 calculates the 3D position information of the first measurement point 41 and the second measurement point 42 by constructing a 3D model of the subject in accordance with the stereo matching algorithm and calculating the depth direction. More specifically, for example, the processor 100 performs preprocessing such as resizing on the stereo image, and then performs a process to parallelize the stereo image. The process of parallelizing the stereo image is performed using pre-determined intrinsic and extrinsic parameters. Intrinsic parameters include, for example, distortion due to the lenses of the reference camera and the reference camera, the focal length of the lens, and the pixel pitch. The extrinsic parameters refer to the position and rotation amount of the reference camera relative to the reference camera. Furthermore, if the processor 100 has acquired the treatment instrument mask image described above, it may perform preprocessing for stereo matching on the treatment instrument mask image because the treatment instrument mask image is deformed due to the stereo image parallelization process described above.

[0044] The processor 100 then performs algorithmic stereo matching, such as semi-global block matching. This allows it to calculate depth information based on parallax and determine the three-dimensional position information of the tip portion of the first treatment tool 31 selected in step S116.

[0045] The measurement process (step S200) will be explained in more detail using the flowchart in Figure 8. The processor 100 functions as a measurement execution determination unit 122 and performs a treatment instrument stability determination process (step S250). After that, the processor 100 performs a process to measure the distance between the first measurement point 41 and the second measurement point 42 (step S270). Details of the treatment instrument stability determination process (step S250) will be described later. Step S270 calculates the distance between the first measurement point 41 and the second measurement point 42 based on the three-dimensional position information of the first measurement point 41 and the second measurement point 42 calculated in step S130 in Figure 6. After performing step S270, the processor 100 performs a process to generate display data (step S290). The display data in step S290 includes display data that visualizes the information related to the stability of the treatment instrument obtained by the treatment instrument stability determination process (step S250) described later, and display data that visualizes the distance information obtained in step S270.

[0046] The procedure instrument stability determination process (step S250) will be explained in more detail using the flowchart in Figure 9. The processor 100 performs a process to determine whether or not there is data from the previous frame (step S252). If there is data from the previous frame (YES in step S252), it performs a process to determine whether or not the amount of movement of the first measurement point 41 is within a first predetermined range (step S254). The amount of movement of the first measurement point 41 refers to the magnitude of the distance between the three-dimensional position coordinates of the first measurement point 41 in the previous frame and the three-dimensional position coordinates of the first measurement point 41 in the current frame. Since the first procedure instrument 31 and the endoscope 20 are operated by the user, the endoscope image may be captured as if the tip of the first procedure instrument 31 is moving. Therefore, the first predetermined range is set as the allowable range for displacement of the tip position of the first procedure instrument 31, and step S254 monitors whether or not the displacement that occurs with the update of the frame is acceptable. The first predetermined range may also be an acceptable range for changes in the tilt of the first treatment tool 31 relative to the horizontal direction.

[0047] On the other hand, if there is no data for the previous frame (NO in step S252), the processor 100 performs a process to increment the first count value by 1 (step S256). Details of the first count value will be described later. If NO is obtained in step S252, in other words, it means that it is the first frame of the endoscopic image captured by the imager.

[0048] Then, if the processor 100 determines that the amount of movement of the first measurement point 41 was within the first predetermined range (YES in step S254), it performs the process of adding 1 to the first count value (step S256).

[0049] If the processor 100 determines YES again in step S254, it further increments the first count value by 1. In other words, if the position of the tip of the first treatment instrument 31 is stable, the first count value will accumulate and be added as the number of imaging frames increases. A first predetermined value may also be set as an upper limit for the first count value. This allows the user to determine that the position of the tip of the first treatment instrument 31 is stable enough that the accuracy of the measurement distance between the first measurement point 41 and the second measurement point 42 is sufficiently high when the first count value is equal to or greater than the first predetermined value. If the processor 100 determines YES in step S254 when the first count value has reached the first predetermined value, step S256 may be a process to maintain the first count value at the first predetermined value. More specifically, for example, if it is desirable for the tip of the first treatment instrument 31 to be stable for a period obtained by multiplying the period of one frame by 30 frames, the first predetermined value should be set to 30.

[0050] In this embodiment, the period obtained by multiplying the period of one frame by a first predetermined value is called the first predetermined period. In other words, if the tip position of the first treatment tool 31 is stable for the first predetermined period, the user can determine that the accuracy of the measurement distance between the first measurement point 41 and the second measurement point 42 is sufficiently high. In other words, the first count value has technical significance as an indicator of the period during which the tip position of the first treatment tool 31 remains stable.

[0051] Although flowcharts and other illustrations are omitted, for example, the processor 100 may further perform a process in the treatment instrument stability determination process (step S250) that calculates the ratio between a first count value and a first predetermined value and stores the said ratio.

[0052] On the other hand, if the processor 100 determines that the amount of movement of the first measurement point 41 is not within the first predetermined range (NO in step S254), it performs a process to set the first count value to 0 (step S258). The fact that the amount of movement of the first measurement point 41 is not within the first predetermined range means that the position of the tip of the first treatment tool 31 is not stable, and it is considered that a certain period of time will be required until the position of the tip of the first treatment tool 31 stabilizes again, so the first count value is set to 0.

[0053] The processing when NO is determined in step S254 is not limited to this, and the user may change it as appropriate. For example, the processor 100 may subtract a predetermined number from the first count value in step S258, or it may perform step S259, described later, without adding or subtracting from the first count value. For example, if the tip of the first treatment tool 31 is stable, but NO is determined in step S254 due to the occurrence of temporary noise, it is not considered necessary to set the first count value to 0.

[0054] Then, after performing step S256 or step S258, the processor 100 performs a process (step S259) to determine the stability of the first treatment instrument 31 according to the first count value. For example, at step S259, the processor 100 indicates the percentage of the accumulated first count value relative to the target value and determines whether or not the position of the tip of the first treatment instrument 31 is stable.

[0055] Figure 10 shows an example of the screen of the second display DP2 to which the method of this embodiment is applied. Based on the endoscopic image captured by the imager of the endoscope 20, the processor 100 displays the screen shown in A10, the icons shown in A13, A14, and A15 on the second display DP2.

[0056] In the screen shown at A10 in Figure 10, the first treatment tool 31 is recognized by the first measurement point designation process (step S110), and the tip of the first treatment tool 31 is displayed as the first measurement point 41. Note that Figure 10 conveniently displays only the first treatment tool 31 related to the first measurement point 41 selected in step S116, but other treatment tools may also be displayed on the screen at A10. The same applies to the screen examples shown in Figures 11, 14, 17, 18, 19, 22, 23, 25, 30, 36, and 37, which will be described later.

[0057] The icon shown in A13 represents the ratio of the period during which the instrument is judged to be stable to the period during which stability of the instrument is desirable. In other words, the ratio of the first count value accumulated in step S256 to the first predetermined value is displayed using a graph icon. Note that the form of the icon shown in A13 is not limited to this, and further details will be described later.

[0058] The icon shown in A14 is a text icon related to the stability of distance measurement, or in other words, a text icon related to the stability of the instrument. The icon shown in A15 is a symbolic icon that concisely indicates the degree of stability. Note that the form of the icon shown in A15 is not limited to this, and further details will be described later in Figure 12.

[0059] In the example screen shown in Figure 10, the icon shown in A13 indicates that the ratio of the first count value to the first predetermined value is not sufficient to determine that the treatment device is stable for that period. The icons shown in A14 and A15 clearly indicate that the treatment device is not stable.

[0060] Meanwhile, the user specifies the second measurement point 42 using the touch panel function of the second display DP2. Then, based on the measurement results measured in step S270 of Figure 8, the distance information icon shown in A11 and the arrow icon shown in A12 are displayed. In other words, the processor 100 generates the icons shown in A11, A12, A13, A14, and A15, the icon representing the first measurement point 41, and the icon representing the second measurement point 42 in step S290 of Figure 8. Then, in step S300, the processor 100 displays these icons together with the endoscopic image on the second display DP2.

[0061] In this embodiment, the distance information icon shown in A11 is configured not to include unit information, but it may be possible to display it with specific units, such as "100mm," or to allow switching between a display mode that includes units and a display mode that does not include units. Furthermore, for example, a table of units for representing length and their conversions may be stored in memory (not shown), and when the user selects a unit of length, the distance information icon shown in A11 may change to an icon based on a number corresponding to the selected unit.

[0062] Furthermore, for the sake of explanation, icons indicating the first measurement point 41 and the second measurement point 42 are superimposed on the screen shown in A10 and displayed on the second display DP2, but this is not mandatory and can be decided by the user as appropriate. Note that in Figure 10, the tip position of the first treatment instrument 31 and the display position of the icon indicating the first measurement point 41 are shown offset for convenience. The same applies to Figures 11, 14, 17, 18, 19, 22, 23, 25, 29(A), 36, and 37, which will be described later.

[0063] Furthermore, in Figure 10, the arrow icon shown at A12 is not mandatory; users can decide whether to display it as they see fit. The same applies to Figures 11, 14, 22, 23, 36, and 37, which will be discussed later.

[0064] Furthermore, the dotted line frame shown in A16 is a frame indicating that the first treatment instrument 31 has been recognized by image processing, and may be displayed on the second display DP2 to supplementarily indicate that the first treatment instrument 31 has been recognized, but it is not necessarily required to display it. In the following sections, the explanation and illustration of the dotted line frame may be omitted as appropriate.

[0065] Figure 11 shows another example of the screen of the second display DP2 to which the method of this embodiment is applied. Similar to Figure 10, the second display DP2 displays the screen shown at A20, as well as the graph shown at A23 and the character icons shown at A24. The character icons shown at A24 include the character icons shown at A25. Also, similar to the case of Figure 10, the first treatment tool 31 is recognized by the first measurement point specification process (step S110) described above, and the tip of the first treatment tool 31 is displayed as the first measurement point 41. Meanwhile, the user touches the second display DP2 as a touch panel, and the second measurement point 42 is specified. Then, based on the measurement result measured in step S270 of Figure 8, the distance information icon shown at A21 and the arrow icon shown at A22 are displayed.

[0066] Figure 10 shows an example screen when the instrument is not sufficiently stable, while Figure 11 shows an example screen when the instrument is sufficiently stable. Therefore, there are differences in the various icons displayed on the second display DP2. For example, the icon shown in A23 indicates that the ratio of the first count value to the first predetermined value is sufficient for the period during which the instrument can be judged to be stable. The icons shown in A24 and A25 clearly indicate that the instrument is stable. In addition, the distance information icon shown in A11 of Figure 10 and the distance information icon shown in A21 of Figure 11 also differ in their display modes, but the details will be described later in Figure 16.

[0067] Furthermore, the symbolic icons indicating the stability of the treatment instrument are not limited to the forms shown in A15 of Figure 10 and A25 of Figure 11, and various modifications are possible. For example, as shown in Figure 12, the stability patterns of the treatment instrument may be stored in a memory (not shown) according to the stability of the treatment instrument, and the pattern can be selected according to the desired level of stability of the treatment instrument. In Figure 12, the stability of the treatment instrument is, for example, the ratio of the first count value to the first predetermined value.

[0068] For example, if the first predetermined value is set to 30, the stability of the treatment device will be 100% if the processor 100 determines YES 30 times in a row in step S254. If the processor 100 determines YES 18 times in a row in step S254, the stability of the treatment device will be 60%.

[0069] For example, if a certain degree of error in the measured distance is acceptable, it may be acceptable to use the distance measurement taken when the stability of the instrument exceeds 60%. In this case, for example, using pattern P-A1 in Figure 12, the stability of the instrument may be divided into cases where the stability is 100%, 80% or more but less than 100%, 60% or more but less than 80%, and 0% or more but less than 60%. Note that "80%~100%" in Figure 12 is a notation meaning "80% or more but less than 100%", and the same applies to other notations. Also, for example, if it is not necessary to distinguish whether the stability of the instrument is 100% or not, pattern P-A2 in Figure 12 may be used. Pattern P-A2 differs from pattern P-A1 in that it uses the same display method for cases where the stability of the instrument is 100% and 80% or more but less than 100%.

[0070] Furthermore, if it is desirable to determine the measured distance as accurately as possible, it is desirable that the stability of the instrument reaches 100%. In this case, a pattern like pattern P-A3 in Figure 12 may be used. Pattern P-A3 is a pattern that consists of two forms: one for when the stability of the instrument is 100% and another for when it is not. In addition, while pattern P-A3 indicates the stability of the instrument using symbols, it may also be indicated using letters, as in pattern P-A4, for example. In this way, the user can easily understand the stability of the instrument.

[0071] Although the flowchart is not shown here, for example, the processor 100 may store the measured distance as time-series data in a memory not shown, and perform a process to display a graph showing the change in the measured distance over time, as shown in Figure 13. In the graph of Figure 13, the vertical axis represents the distance information measured in step S270 of Figure 8, and the horizontal axis represents time.

[0072] For example, if the second display DP2 has a touch panel function and the user specifies the second measurement point 42 using this touch panel function, the stability of the tip of the first treatment tool 31 can be quantitatively grasped by the graph in Figure 13. In other words, when the second measurement point 42 is specified by step S122 in Figure 7, the subject related to the second measurement point 42 can be treated as stationary, so the behavior of the graph in Figure 13 is considered to depend on the stability of the tip of the first treatment tool 31 being operated by the user. For example, as the user starts operating the first treatment tool 31 and becomes accustomed to operating the first treatment tool 31, the behavior of the tip of the first treatment tool 31 stabilizes, and the range of variation in the measured distance narrows over time.

[0073] Therefore, for example, in the measurement process (step S200) shown in Figure 8, the processor 100 may perform a process to determine that the distance measurement is stable if the amount of change in the measured distance falls within the second predetermined range during the second predetermined period. Specifically, for example, the width shown in B1 in Figure 13 corresponds to the second predetermined range, and the width shown in B2 corresponds to the second predetermined period.

[0074] Furthermore, as mentioned above, since the position information of the first measurement point 41 and the position information of the second measurement point 42 can be obtained using only the first method, an example screen of the second display DP2 to which the method of this embodiment is applied can be as shown in Figure 14, for example. In Figure 14, the second display DP2 displays the screen shown in A30, as well as the icons shown in A33, A34, and A35. The icons shown in A31, A32, A33, A34, and A35 in Figure 14 correspond to the icons shown in A21, A22, A23, A24, and A25 in Figure 11. On the other hand, the screen shown in A30 of Figure 14 differs from the example screen shown in A20 of Figure 11 in that the position information of the tip of the second treatment tool 32 is used as the position information of the second measurement point 42.

[0075] For example, although the flowchart is not shown, the processor 100 executes step S110 when at least the first treatment instrument 31 and the second treatment instrument 32 are visible on the imager of the endoscope 20. Then, in S112, the processor 100 recognizes the first treatment instrument 31 and the second treatment instrument 32. Then, in step S114, the processor 100 recognizes the tip portion of the first treatment instrument 31 and the tip portion of the second treatment instrument 32. Then, in step S116, the processor 100 designates the tip portion of the first treatment instrument 31 as the first measurement point 41 and the tip portion of the second treatment instrument 32 as the second measurement point 42. This allows the second measurement point 42 to be designated instead of step S120. Then, by further performing steps S130, S200, and S300, the processor 100 can realize the display of the example screen in Figure 14.

[0076] Furthermore, if the position information of the tip of the second treatment tool 32 is used as the position information of the second measurement point 42, as shown in Figure 14, the stability of the tip of the second treatment tool 32 may be further determined. In other words, the treatment tool stability determination process (step S250) shown in Figure 9 may be applied to both the first measurement point 41 and the second measurement point 42.

[0077] Although not shown in the diagram, for example, the icons shown in A13, A14, and A15 of Figure 10 may be displayed for the first measurement point 41 and the second measurement point 42, respectively. Alternatively, the icons shown in A13, A14, and A15 of Figure 10 may be displayed prioritizing the one with lower stability of the treatment instrument between the first measurement point 41 and the second measurement point 42.

[0078] Based on the above, the image processing apparatus 10 of this embodiment includes a processor 100 that performs display processing on the display DP. The processor 100 uses the endoscopic image of the subject acquired by the endoscope 20 to calculate three-dimensional positional information of the first treatment instrument 31 and a predetermined part within the endoscopic image (step S100). The processor 100 also measures the distance between a first measurement point 41 on the tip side of the first treatment instrument 31 and a second measurement point 42 relating to the predetermined part based on the three-dimensional positional information of the first treatment instrument 31 and the predetermined part (step S200), and performs display processing (step S300) on the display DP (second display DP2) to indicate whether the distance measurement is in a stable state.

[0079] As described above, the processor 100 included in the image processing device 10 of this embodiment can calculate the three-dimensional position information of the first measurement point 41 and the second measurement point 42 in the endoscopic image acquired from the endoscope 20. Furthermore, since it performs display processing to indicate whether the measurement of the distance between the first measurement point 41 and the second measurement point 42 is in a stable state, the user can easily determine from the display DP whether the distance is being measured stably or not. As a result, the user can perform procedures based on the appropriately measured distance.

[0080] For example, as mentioned above, in endoscopic surgery, the endoscope 20 and the first treatment instrument 31 are manipulated by the user's hands, which can lead to significant variability in the positional information of the tip of the first treatment instrument 31. Similarly, the effect of hand tremors on the imager of the endoscope 20 can be significant. In this respect, by applying the method of this embodiment, the user can determine whether the distance measurement is stable or not, and therefore the user can perform the procedure using the distance when the distance measurement is stable. This allows the user to perform a more appropriate procedure. For example, when evaluating the surgical margin for partial resection of a malignant tumor, the measured surgical margin distance information is of great importance.

[0081] The aforementioned U.S. Patent Application Publication No. 2010 / 0317965 discloses a method for displaying a measurement of the distance between points specified at the tip of a robotic tool used in surgical procedures, but it does not propose a method for displaying information regarding whether or not the distance measurement is stable.

[0082] Furthermore, the method of this embodiment may be implemented as an endoscope system 1. That is, the endoscope system 1 of this embodiment includes an image processing device 10 and an endoscope 20. By doing so, the same effects as described above can be obtained.

[0083] Furthermore, the method of this embodiment may also be implemented by a processing method. Specifically, the processing method of this embodiment involves having a computer perform the following processes: displaying the endoscopic image of the subject acquired by the endoscope 20 on the display DP; and calculating the three-dimensional positional information of the first treatment instrument 31 and a predetermined part within the endoscopic image using the endoscopic image of the subject. In addition, the processing method of this embodiment involves having the computer perform the following processes: measuring the distance between a first measurement point 41 on the tip side of the first treatment instrument 31 and a second measurement point 42 relating to the predetermined part, based on the three-dimensional positional information of the first treatment instrument 31 and the predetermined part; and displaying whether or not the distance measurement is in a stable state. By doing so, the same effects as described above can be obtained.

[0084] Furthermore, the method of this embodiment may be implemented as a program. Specifically, the program of this embodiment causes the computer to perform the steps of: displaying the endoscopic image of the subject acquired by the endoscope 20 on the display DP; and calculating the three-dimensional positional information of the first treatment instrument 31 and a predetermined part within the endoscopic image using the endoscopic image of the subject. The program of this embodiment also causes the computer to perform the steps of: measuring the distance between a first measurement point 41 on the tip side of the first treatment instrument 31 and a second measurement point 42 relating to the predetermined part based on the three-dimensional positional information of the first treatment instrument 31 and the predetermined part; and displaying whether or not the distance measurement is in a stable state. By doing so, the same effects as described above can be obtained.

[0085] Furthermore, the processor 100 may calculate three-dimensional position information based on the left-eye image and the right-eye image, which are endoscopic images acquired by the endoscope 20. In this way, an image processing device 10 can be constructed that can determine whether or not the distance measured using the position information of the first measurement point 41 and the position information of the second measurement point 42 obtained by stereo matching is in a stable state.

[0086] Furthermore, the three-dimensional position information of the predetermined part may be the three-dimensional position information of the second treatment tool 32. The processor 100 may also measure the distance between a first measurement point 41 on the tip side of the first treatment tool 31 and a second measurement point 42, which is the position on the tip side of the second treatment tool 32, based on the three-dimensional position information of the first treatment tool 31 and the predetermined part. In this way, an image processing device 10 can be constructed that can determine whether the distance measurement using the tip position of the first treatment tool 31 and the tip position of the second treatment tool 32 is in a stable state.

[0087] Furthermore, the three-dimensional position information of the predetermined part may include the three-dimensional position information of a second measurement point 42, which is the position of the subject corresponding to the position on the display DP (second display DP2) specified by the user. The processor 100 may also measure the distance between the first measurement point 41 on the tip side of the first treatment tool 31 and the second measurement point 42 based on the three-dimensional position information of the first treatment tool 31 and the predetermined part. In this way, an image processing device 10 can be constructed that can determine whether the measurement of the distance using the tip position of the first treatment tool 31 and the position specified by the user is in a stable state.

[0088] Furthermore, the processor 100 may determine that the distance measurement is stable if the first measurement point 41 and the second measurement point 42 are measured stably over a predetermined period of time, or if the distance is measured stably over a predetermined period of time. In this way, an image processing device 10 can be constructed that quantifies the stability of distance measurement over a period of time.

[0089] Furthermore, the processor 100 may determine that the system is in a stable state if the amount of movement of the first measurement point 41 is within a first predetermined range within a first predetermined period. In this way, the calculated positional information of the first measurement point 41 can be used to construct a criterion for determining the stability of distance measurement.

[0090] Furthermore, the processor 100 may determine that the system is in a stable state if the amount of change in distance is within a second predetermined range within a second predetermined period. In this way, a criterion for determining the stability of distance measurement can be established using the measured distance information.

[0091] The method of this embodiment is not limited to the above and can be modified in various ways. For example, the display method of the icon indicating the period during which the treatment device is judged to be stable can be modified in various ways. For example, in addition to the bar graph type icon shown in B11 of Figure 15, it may also be a pie chart type icon shown in B12, or a meter display type icon shown in B13. Furthermore, the meter display type icon in B13 may further include an icon showing the ratio of the first count value to the first predetermined value, as shown in B14. Moreover, it is not limited to graph type icons, but may also be a fraction display type icon, for example, shown in B15. For example, the number on the denominator side shown in B16 corresponds to the first predetermined value, and the number on the numerator side shown in B17 corresponds to the first count value. This allows the user to easily determine the ratio of the period during which the treatment device is judged to be stable to the period during which it is desirable for the treatment device to be stable.

[0092] Furthermore, the display modes for the measured distance information are not limited to those shown in A11 of Figure 10 and A21 of Figure 12, and various modifications are possible. For example, as shown in Figure 16, patterns for displaying the measured distance information may be stored in a memory (not shown), allowing the user to select a display mode.

[0093] In pattern PB-1 of Figure 16, the display when the instrument is unstable is an iconized representation of the measured numbers shown in B21, with a symbolic icon shown in B22 added. Also in pattern PB-1, the display when the instrument is stable is an iconized representation of the measured numbers shown in B23. The user can understand that the instrument is unstable by looking at the symbolic icon shown in B22. Alternatively, comparing the icon shown in B21 and the icon shown in B23, the icon shown in B23 is displayed in bolder font. This makes the icon in B23 more visible than the icon in B21, thus suggesting to the user that the instrument is stable. Similarly, in pattern PB-2 of Figure 16, the icon for a stable instrument is displayed in a darker color than the icon for an unstable instrument, thus suggesting to the user that the instrument is stable. Furthermore, pattern PB-3 of Figure 16 is a display pattern that combines the icons of pattern PB-1 with graph icons. The graph icon shown in B24 indicates that the instrument is unstable, while the graph icon shown in B25 indicates that the instrument is stable. Furthermore, pattern PB-4 differs from pattern PB-1 in that it does not display distance information when the instrument is unstable.

[0094] In this embodiment, the image processing device 10 changes the display mode of the distance measurement value depending on whether the distance measurement is stable or not. This allows the user to easily determine whether the distance measurement is stable or not.

[0095] Furthermore, the processor 100 may display the period of stable distance measurement in a graph. In this way, the user can visually grasp the stability of distance measurement.

[0096] Furthermore, for example, if the stability of the treatment instrument is insufficient, the processor 100 may perform the process of displaying the example screen shown in Figure 17 on the second display DP2. In Figure 17, the second display DP2 displays the screen shown in A40, the icons shown in A43, A44, and A45. In addition to displaying the first treatment instrument 31 on the screen shown in A40, the icon shown in A46 is also displayed. The icon shown in A46 is an icon that suggests to the user at what position the tip of the first treatment instrument 31 should be positioned for the tip of the first treatment instrument 31 to be stable. Note that the icons shown in A43, A44, and A45 in Figure 17 correspond to the icons shown in A13, A14, and A15 in Figure 10, respectively. Also, in Figure 17, the pattern P-B4 in Figure 16 is used as the display mode for the distance information icon. In other words, because the stability of the treatment instrument is insufficient, the icon related to the measured distance information is not displayed on the second display DP2 in Figure 17.

[0097] For example, although flowcharts and other diagrams are omitted, if the processor 100 determines YES in step S254, it may perform step S256 and store predetermined data in a memory not shown. The predetermined data is, for example, the position information of the tip of the first treatment tool 31 related to the frame for which YES was determined in step S254. The processor 100 then searches the memory for frames containing the predetermined data from the information of past frames stored in the memory and generates a set of position information for the tip of the first treatment tool 31 based on the predetermined data of the searched frame.

[0098] Then, in step S290, the processor 100 performs a process to generate predetermined graphic image data based on the aggregate information. The predetermined graphic image data can be described as image information for keeping the movement amount of the first measurement point 41 within a first predetermined range.

[0099] Then, in step S300, the processor 100 performs the process of superimposing the icon shown in A46 of Figure 17 onto the image A40. Note that although the icon shown in A46 is a circular icon, it may also be an icon made up of other shapes such as an ellipse or polygon.

[0100] In this way, the user can easily stabilize the tip of the first treatment instrument 31. As a result, the screen example in Figure 17 becomes the screen example shown in Figure 18. In Figure 18, the second display DP2 displays the screen shown at A50, the icons shown at A53, A54, and A55. In addition to the first treatment instrument 31 being displayed on the screen shown at A50, the icons shown at A51, A52, and A56 are also displayed. Since the tip of the first treatment instrument 31 is stable, the icons shown at A51, A52, A53, A54, and A55 in Figure 18 are displayed in the same manner as the icons shown at A21, A22, A23, A24, and A25 in Figure 11. In this situation, since the tip of the first treatment instrument 31 is stable, the amount of movement of the position coordinates of the tip of the first treatment instrument 31 in each past imaging frame is small. Therefore, the icon shown in A56 of Figure 18 is displayed smaller than the icon shown in A46 of Figure 17.

[0101] Furthermore, if the stability of the treatment instrument is insufficient, the processor 100 may perform the process of displaying the example screen shown in Figure 19 on the second display DP2. In Figure 19, the second display DP2 displays the screen shown at A60, the icons shown at A63, A64, and A65. In addition, the screen shown at A60 displays the first treatment instrument 31, as well as the icons shown at A66, A67, and A68.

[0102] For example, although the flowchart and other diagrams are omitted, in step S290, the processor 100 generates display data for predetermined dot-shaped icons based on the position information of the tip of the first treatment tool 31, which is included in the data of the frames that were determined to be YES in step S254, from the data of past frames stored in memory (not shown). Then, in step S300, the processor 100 superimposes the icons shown in A66, A67, and A68 onto the image of A60 based on the display data generated in step S290.

[0103] Furthermore, although not shown in the diagram, if the stability of the treatment instrument is insufficient, the icons shown in A66, A67, and A68 in Figure 19 and the icon shown in A46 in Figure 17 may be displayed together.

[0104] In the example screens in Figures 17, 18, and 19, the second measurement point 42 is the point selected by the user on the touch panel in step S122 of Figure 7. However, even when the tip of the second treatment instrument 32 is used as the second measurement point 42, the methods shown in Figures 17 and 18 may also be applied. For example, if both the tip of the first treatment instrument 31 and the tip of the second treatment instrument 32 are unstable, the icons shown in A46 of Figure 17, or the icons shown in A66, A67, and A68 of Figure 19, may be displayed on the tips of the first treatment instrument 31 and the second treatment instrument 32, respectively.

[0105] In this embodiment, the image processing device 10 displays image information on the display DP (second display DP2) to ensure that the movement of at least the first measurement point 41 is within a first predetermined range. By doing so, the image processing device 10 can provide the user with assistance in stabilizing distance measurement.

[0106] Furthermore, for example, the measurement process (step S200) may be performed as shown in the flowchart of Figure 20. Figure 20 differs from Figure 8 in that it includes the image stability determination process (step S210) and step S230. In the following sections, explanations of processes similar to those already described will be omitted as appropriate.

[0107] In Figure 20, after performing the image stability determination process (step S210), the processor 100 performs a process to determine whether the image is stable or not (step S230). Details of the image stability determination process (step S210) and step S230 will be described later, but the processor 100 determines whether the imaging state by the imager of the endoscope 20 is stable or not. If the processor 100 determines that the image is stable (YES in step S230), it performs the treatment instrument stability determination process (step S250), step S270, and step S290 as in Figure 8, and terminates the flow.

[0108] On the other hand, if the processor 100 determines that the image is unstable (NO in step S230), it performs step S290 and terminates the flow. In other words, Figure 20 shows that if the image is unstable, the stability of the first treatment instrument 31, etc., is determined, and the endoscopic image related to the frame is updated without measuring the distance between the first measurement point 41 and the second measurement point 42.

[0109] Note that the example of the measurement process (step S200) including the image stability judgment process (step S210) is not limited to Figure 20, and various modifications are possible. For example, even if step S230 is NO, the processor 100 may judge the stability of the first treatment instrument 31, etc., and measure the distance between the first measurement point 41 and the second measurement point 42. In this case, although the flowchart is omitted, step S230 can be omitted in Figure 20. Alternatively, although the flowchart is omitted, the processor 100 may perform the image stability judgment process (step S210) and the treatment instrument stability judgment process (step S250) in parallel in the measurement process (step S200), and then perform step S230. In this case, if the processor 100 determines YES in step S230, it should perform step S270, and if it determines NO in step S230, it should perform step S290.

[0110] A more detailed example of the image stability determination process (step S210) will be explained using the flowchart in Figure 21. The processor 100 performs a process to determine whether or not the endoscopic image contains features unsuitable for measurement (step S212). Features unsuitable for measurement in an endoscopic image include, for example, out of focus, insufficient overall brightness of the image, insufficient overall contrast of the image, and other events that make the image unsuitable for stereo matching.

[0111] If the processor 100 determines that the endoscopic image does not contain features unsuitable for measurement (NO in step S212), it increments the second count value by 1 (step S214). For example, if the processor 100 determines YES again in step S214 of the next imaging frame, it further increments the second count value by 1. In other words, if the endoscopic image is stable, the second count value accumulates with each increase in the number of imaging frames. Thus, the second count value has technical significance as an indicator of the period during which a state suitable for measurement continues. A second predetermined value may also be set as an upper limit for the second count value. This allows the user to determine that a state suitable for measuring the distance between the first measurement point 41 and the second measurement point 42 has continued for a sufficient amount of time when the second count value is equal to or greater than the second predetermined value.

[0112] On the other hand, if the processor 100 determines that the endoscopic image contains features unsuitable for measurement (YES in step S212), it sets the second count value to 0 (step S216). For example, suppose that the second count value had been continuously increasing because NO was determined in step S212, but then, due to certain circumstances, YES was determined in step S212. These certain circumstances include, for example, the appearance of the endoscopic image suddenly and drastically changing due to external factors such as mist, or a significant lapse in the operation of the user holding the endoscope 20. In such cases, it is considered appropriate to set the second count value to 0 to correspond to restarting the assessment of image stability from the beginning.

[0113] Step S216 is not limited to what is shown in Figure 21, and can be determined by the user as appropriate. For example, it may be a process that subtracts a predetermined number from the second count value. Alternatively, it may be a process that does neither add nor subtract from the second count value, that is, if the processor 100 determines NO in step S212, it may proceed to step S218 without performing step S214.

[0114] Then, after performing step S214 or step S216, the processor 100 determines the stability of the image according to the second count value (step S218). Step S218 is the same process as step S259 described above. For example, the processor 100 calculates the ratio of the accumulated second count value to a second predetermined value to determine the degree of stability of the endoscopic image. Then, in the subsequent step S290, the processor 100 generates display data corresponding to the determination result of step S218, and in the subsequent step S300, displays the image corresponding to the display data generated in step S290 on the second display DP2.

[0115] Figure 22 shows an example screen when the processing shown in Figures 20 and 21 is applied. In Figure 22, the screen shown at A70, the icons shown at A73, A74, and A75 are displayed on the second display DP2. The icon shown at A73 is a pie chart-shaped icon that shows the ratio of the period during which the endoscopic image is judged to be stable to the period during which the endoscopic image is considered desirable to be stable. In other words, it displays the ratio of the second count value accumulated in step S214 to the second predetermined value using a graph icon. The icon shown at A74 is a text icon related to the stability of the endoscopic image. The icon shown at A75 is a symbol icon that succinctly indicates the degree of stability of the endoscopic image. The situation shown in Figure 22 is that the screen stability is insufficient because the second count value added in step S214 of Figure 21 has not reached a predetermined ratio to the second predetermined value, and is therefore unsuitable for measuring the distance between the first measurement point 41 and the second measurement point 42. The pie chart icon shown in A73 visually indicates the ratio of the second count value to the second predetermined value, and whether the second count value has not reached the predetermined ratio to the second predetermined value. The icons shown in A74 and A75 clearly indicate that the endoscopic image is unstable.

[0116] Note that the screen example in Figure 22 is an example of a screen where step S230 of Figure 20 has been omitted. Therefore, steps S250 and S270 of Figure 20 are performed, and the distance information icon shown in A71 and the arrow icon shown in A72 of Figure 22 are displayed together. Note that the distance information icon shown in A71 is the same as the icon in A11 of Figure 10. Note that in Figure 22, the icons A13, A14, and A15 of Figure 10 are omitted from the illustration for the sake of simplicity, but these icons may also be displayed on the second display DP2.

[0117] Figure 23 shows another example screen when the processing shown in Figures 20 and 21 is applied. In Figure 23, the screen shown at A80, the icons shown at A83, A84, and A85 are displayed on the second display DP2. The situation shown in Figure 23 indicates that the stability of the endoscopic image is sufficient, as the second count value added in step S214 of Figure 21 has reached the second predetermined value, making it suitable for measuring the distance between the first measurement point 41 and the second measurement point 42. The pie chart-shaped icon shown at A83 indicates that the second count value has reached the second predetermined value. The icons shown at A84 and A85 clearly indicate that the endoscopic image is stable.

[0118] Note that the icon configurations shown in A73 and A75 in Figure 22 are not limited to those shown in Figure 22. Similarly, the icon configurations shown in A83 and A85 in Figure 23 are not limited to those shown in Figure 23. For example, the icon configurations shown in A73 in Figure 22 and A83 in Figure 23 may appropriately adopt the examples described in Figure 15. Similarly, the icon configurations shown in A75 in Figure 22 and A85 in Figure 23 may appropriately adopt the patterns described in Figure 12.

[0119] In this embodiment, the image processing device 10 calculates a determined distance value when the endoscopic image is stable for a predetermined period of time. The processor 100 also displays an indication on the display DP (second display DP2) that the endoscopic image is stable for the predetermined period of time. In this way, the user can easily understand that the endoscopic image is stable.

[0120] Furthermore, the image stability determination process (step S210) may be performed as shown in the flowchart of Figure 24. In Figure 24, the processor 100 performs a process (step S222) to determine whether the first treatment instrument 31 is projected horizontally or not. For example, the processor 100 performs a process to calculate a direction vector parallel to the longitudinal direction of the first treatment instrument 31 based on the treatment instrument mask image related to the first treatment instrument 31 specified in step S110, and a process to calculate the angle between the said direction vector and the X-axis direction. If the calculated angle is within a predetermined range, the processor 100 determines that the first treatment instrument 31 is projected horizontally, thereby enabling the process of step S222. The predetermined range may be appropriately changed depending on the stereo matching conditions.

[0121] If the processor 100 determines that the first treatment instrument 31 is not displayed horizontally (NO in step S222), it performs a process to determine that the image is stable (step S224) and terminates the flow. On the other hand, if the processor 100 determines that the first treatment instrument 31 is displayed horizontally (YES in step S222), it performs a process to determine that the image is unstable (step S226) and terminates the flow. In the subsequent step S290, the processor 100 performs a process to generate image data corresponding to step S224 or step S226. In the subsequent step S300, the processor 100 performs a process to display the image data generated in step S290 on the second display DP2. Note that the image stability determination process (step S210) may be a combination of the processes in Figure 21 and Figure 24.

[0122] Figure 25 shows an example screen when the processing example in Figure 24 is applied. Note that in Figure 25, the icons shown at A81, A82, A83, A84, and A85 in Figure 23, as well as the icons indicating the first measurement point 41 and the second measurement point 42, are omitted, but these may be displayed.

[0123] In Figure 25, the second display DP2 shows the screen shown at A90. In the screen shown at A90, the first treatment instrument 31 is projected parallel to the horizontal direction. In other words, the first treatment instrument 31 is projected parallel to the parallax direction. For this reason, the processor 100 determines YES in step S222 of Figure 24 and proceeds to step S226. As a result, in steps S290 and S300, the image shown at A97 is superimposed on the image shown at A90 and displayed. The image shown at A97 includes instructions to the user that the first treatment instrument 31 is projected horizontally to the screen and to tilt the first treatment instrument 31 relative to the horizontal direction. Note that tilting the first treatment instrument 31 relative to the horizontal direction means, for example, displacing the first treatment instrument 31 by the user operating the first treatment instrument 31, but is not limited to this; for example, the user operating the endoscope 20 may rotate the scope of the endoscope 20. For example, if the tip of the first treatment instrument 31 is already positioned at the desired location, it is more appropriate to rotate the scope of the endoscope 20 than to displace the first treatment instrument 31. Thus, in the image processing device 10 of this embodiment, when the first treatment instrument 31 is parallel to the parallax direction in the stereo view of the endoscope image, the processor 100 displays an instruction on the display DP (second display DP2) to tilt the first treatment instrument 31 with respect to the parallax direction. By doing so, the accuracy of stereo matching can be improved. Because the treatment instrument has little texture, the entire instrument may appear horizontally, which may reduce the accuracy of stereo matching. Texture here refers to patterns, designs, etc. that appear due to the brightness of each pixel in the image. In this respect, by applying the method of this embodiment, it is possible to prevent a decrease in the accuracy of stereo matching.

[0124] Furthermore, for example, the image shown in A98 of Figure 25 may be displayed instead of the image shown in A97. Alternatively, the images shown in A97 and A98 may be displayed together. The image shown in A98 is an object image indicating the direction in which to rotate the scope of the endoscope 20, in order to prompt the user to rotate the scope of the endoscope 20. In other words, in the image processing device 10 of this embodiment, when the first treatment instrument 31 is parallel to the parallax direction in the stereo view of the endoscope image, the processor 100 displays an object on the display DP (second display DP2) indicating the direction in which to tilt the first treatment instrument 31. In this way, the user can easily determine the operation necessary to further improve the accuracy of stereo image construction.

[0125] Figures 24 and 25 show examples of measuring distance using the first treatment tool 31. However, as mentioned above in Figure 14, when measuring distance using both the first treatment tool 31 and the second treatment tool 32, the processing shown in Figure 24 can also be applied to the second treatment tool 32 to display the images shown at A97 and A98 in Figure 25, and various modifications can be implemented.

[0126] Furthermore, for example, the measurement process (step S200) in this embodiment may be as shown in the flowchart of Figure 26. The processing example in Figure 26 differs from the processing example in Figure 8 in that a smoothing process (step S280) is performed after step S270.

[0127] The smoothing process (step S280) is, for example, a process that smooths the measured distance in the time direction. More specifically, for example, the processor 100 calculates the average value of the distance information measured in step S270 and the distance information from past frames stored in memory (not shown). The average value here is the arithmetic mean, which is, for example, the sum of the distance information stored in memory (not shown) divided by the number of frames. Then, the processor 100 generates display data of the distance information based on the average value calculated in step S290. As a result, although not shown in the diagram, an icon of the smoothed distance information is displayed on the second display DP2.

[0128] In step S280 of Figure 26, when calculating the average value, it is desirable that the number of past imaging frames required be the same as the first predetermined value. By doing so, more accurate distance information can be displayed on the second display DP2. The three-dimensional position information of the first measurement point 41 and the second measurement point 42 calculated in step S130 may also be smoothed in the time direction in the same manner.

[0129] In other words, smoothing the measured distance requires that the distance measurement be stable for a predetermined period. This predetermined period may be, for example, the first predetermined period described in Figure 9, or the second predetermined period described in Figure 13.

[0130] Alternatively, instead of step S280, a process may be performed to smooth the 3D position information of the first measurement point 41 over time. In this case, the process of smoothing the 3D position information of the first measurement point 41 over time is performed before step S270. The same applies to the second measurement point 42.

[0131] Alternatively, a method of smoothing in the spatial direction may be applied. More specifically, for example, the Z coordinate of the 3D position information of the first measurement point 41 may be smoothed in the spatial direction by performing the measurement process (step S200) as shown in the processing example in Figure 27. The process in Figure 27 differs from that in Figure 8 in that it further includes a Z coordinate correction process (step S260).

[0132] Figure 28 is a detailed flowchart of the Z-coordinate correction process (step S260). The processor 100 performs a process to set the region of interest (step S262). Specifically, as shown in B30 of Figure 29(A), the region of interest is set around the first measurement point 41. Then the processor 100 performs a process to determine whether or not there are past frames available for correction (step S264). More specifically, the processor 100 determines whether or not the data of the frame that was determined to be YES in step S254 of Figure 9 is stored in memory not shown.

[0133] If the processor 100 determines that there are past frames available for correction (YES in step S264), it performs a process to correct the Z coordinate for the past frame and the current frame (step S265). On the other hand, if the processor 100 determines that there are no past frames available for correction (NO in step S264), it performs a process to correct the Z coordinate for the current frame (step S266).

[0134] In step S266, correcting the Z coordinates specifically refers to calculating the mean or median of the Z coordinates of pixels related to the tip edge of the first treatment tool 31 within the region of interest of the stereo image for the current frame. Pixels related to the tip edge of the first treatment tool 31 within the region of interest are, for example, the pixels in the region shown as B31 in Figure 29(A). The user can decide whether to use the mean or the median as appropriate, but if, for example, the distribution of Z coordinate data tends to deviate from a normal distribution due to the occurrence of large amounts of noise, the median is more appropriate than the mean. For example, if the Z coordinate values ​​consist of five data points (1, 3, 5, 6, 20), the mean is 7 and the median is 5. However, if the data with Z=20 is treated as noise, the median is considered more appropriate than the mean.

[0135] To correct the Z coordinate in step S265, for example, one can perform the same calculation as in step S266 for each past frame and use the median value of the Z coordinate calculated for each frame.

[0136] Subsequently, the processor 100 performs a process (step S268) to calculate the three-dimensional position information of the first measurement point 41. For example, the process in step S268 can be realized by using a method of perspective projection of the three-dimensional position information of the first measurement point 41 onto a reference image plane. For example, the position shown in B41 in Figure 29(B) is the origin of the reference camera, the position shown in B42 is the three-dimensional position of the corrected first measurement point 41, and the position shown in B43 is the position indicating the Z coordinate of the corrected first measurement point 41. Then, in the reference image plane shown in B40, the position shown in B44 is the position corresponding to the first measurement point 41 in the reference image, and the point shown in B45 is the intersection of the Z axis of the reference camera and the reference image plane. The line-of-sight vector LV is a vector that points from the origin position shown in B41 to the position shown in B44, and further to the three-dimensional position of the first measurement point 41 shown in B42.

[0137] In this case, the position information shown at B44 is known from step S100, and the Z coordinate of the position shown at B45 is known because it is the corrected Z coordinate from step S265 or step S266. Furthermore, the triangle formed by points B41, B42, and B43 is similar to the triangle formed by points B41, B44, and B45. Therefore, by using camera parameters such as focal length, the scale of the line-of-sight vector LV can be changed, and the 3D position information of the point shown at B42, i.e., the corrected 3D position information of the first measurement point 41, can be obtained.

[0138] As described above, in the image processing apparatus 10 of this embodiment, the processor 100 obtains the three-dimensional position information of the first measurement point 41 using the median or average value of the depth coordinates in the surrounding area of ​​the measurement position coordinates of the first measurement point 41 in the endoscopic image, and the line-of-sight vector LV based on the measurement position coordinates. In this way, the position information of the first measurement point 41 can be obtained more accurately by stereo matching. Because the treatment instrument is made of metal and has poor texture, reflections and glare occur, resulting in large variations in the Z coordinate after stereo matching. However, by applying the method of this embodiment, the Z coordinate can be smoothed in space without smoothing the X and Y coordinates of the first measurement point 41, thereby improving the calculation accuracy of the first measurement point 41. As a result, the measured distance can be determined more accurately.

[0139] Furthermore, even when the tip of the second treatment tool 32 is used as the second measurement point 42, the above-described time-direction smoothing process or spatial-direction smoothing process can be applied.

[0140] Based on the above, in the image processing apparatus 10 of this embodiment, the processor 100 measures the distance by performing a smoothing process in the time direction or a smoothing process in the spatial direction on at least the first measurement point 41. By doing so, the accuracy of distance measurement can be further improved.

[0141] Furthermore, as shown in the example screen at A100 in Figure 30, for example, a position away from the tip of the first treatment tool 31 may be designated as the first measurement point 41. In the example screen at A100, the second measurement point 42 is shown as the point designated in step S120 in Figure 7.

[0142] For example, the processor 100 can realize the screen example shown in A100 of Figure 30 by performing a modified version of the first measurement point specification process (step S1110) shown in Figure 31. In Figure 31, the processor 100 functions as a measurement point selection unit 114 and performs a process (step S1112) to select the first treatment tool 31 from among the recognized treatment tools. Step S1112 in Figure 31 corresponds to steps S112 and S114 in Figure 6. The processor 100 then performs a process (step S1114) to detect the tip and base portions of the selected first treatment tool 31. The base portion of the first treatment tool 31 is the part closest to the base among the parts of the first treatment tool 31 that can be displayed on the second display DP2.

[0143] Subsequently, the processor 100 performs a process to calculate three-dimensional position information (step S1116). Step S1116 is the same process as the three-dimensional position information calculation process (step S130) shown in Figure 5. More specifically, for example, the processor 100 generates a treatment tool mask image related to the first treatment tool 31 in step S1112, and performs stereo matching based on the treatment tool mask image and stereo matching based on the image of the subject obtained by removing the portion related to the first treatment tool 31 from the stereo image in step S1116.

[0144] As the processor 100 performs steps S1112, S1114, and S1116, the three-dimensional structure of the first treatment tool 31 is constructed, for example, as shown in B51 of Figure 32. The area shown in B52 conceptually represents a part of the three-dimensional structure of the subject, excluding the portion related to the first treatment tool 31. Figure 32 does not specifically identify the structure of the first treatment tool 31, etc.

[0145] Returning to the flowchart in Figure 31, the explanation continues. The processor 100 then performs a process (step S1118) to calculate the first extension line PL1. Specifically, for example, the tip portion of the first treatment tool 31 detected in step S1114 is constructed three-dimensionally in step S1116 as shown in B53. Similarly, the base portion of the first treatment tool 31 detected in step S1114 is constructed three-dimensionally in step S1116 as shown in B54. Then, in step S1118, the first extension line PL1 is calculated based on the constructed tip portion and base portion. For example, the processor 100 performs a process to calculate a straight line passing through the centroid coordinates of the tip portion shown in B53 and the centroid coordinates of the base portion shown in B54. As a result, the first extension line PL1 is approximately the same as a straight line that passes through the tip of the first treatment tool 31 and is parallel to the longitudinal direction of the first treatment tool 31. This allows for accurate determination of the direction the tip of the first treatment tool 31 is pointing, and thus enables accurate positioning of the first measurement point 41 at the desired location.

[0146] Subsequently, the processor 100 performs a process (step S1119) to designate the intersection point of the first extension line PL1 and the subject as the first measurement point 41. As a result, the intersection point of the first extension line PL1, which was determined in step S1118, and the subject shown in B52 is determined, as shown in B55 of Figure 32. In other words, the point shown in B55 is designated as the first measurement point 41, and the first measurement point designation process (step S1110) shown in Figure 31 is completed.

[0147] Although a flowchart is not shown, for example, by performing the processing from step S1116 onward after further specifying the second measurement point 42 using the same process as in step S122 in Figure 7, between step S1114 and step S1116, the state will be the same as when the position specification calculation process (step S100) is completed.

[0148] Although the flowchart is not shown here, if it is desired to specify a second measurement point 42 at a location away from the tip of the second treatment tool 32, the processor 100 can select the first treatment tool 31 and the second treatment tool 32 in step S1112, and then perform steps S1114, S1116, S1118, and S1119 for the selected first treatment tool 31 and second treatment tool 32.

[0149] Furthermore, while Figures 31 and 32 show the first measurement point 41 being specified by using a 3D reconstruction process based on stereo images, the method of this embodiment is not limited to this, and the first measurement point 41 may be specified on a 2D image. In this case, for example, the first measurement point specification process (step S2110) shown in the flowchart of Figure 33 can be performed.

[0150] In Figure 33, the processor 100 functions as a measurement point selection unit 114 and, similar to step S1112 in Figure 31, performs the process of selecting the first treatment tool 31 from among the recognized treatment tools (step S2112). Then, similar to step S1114 in Figure 31, the processor 100 performs the process of detecting the tip and base portions of the selected first treatment tool 31 (step S2114).

[0151] Subsequently, the processor 100 performs a process to calculate the second extension line PL2 (step S2116). Specifically, for example, in the reference image shown at B60 in Figure 34, in step S2112, the processor 100 recognizes the area shown at B61 as the first treatment tool 31. Then, in step S2114, the processor 100 detects the tip portion shown at B62 and the base portion shown at B63. Then, in step S2116, the processor 100 calculates the second extension line PL2. For example, the processor 100 performs a process to calculate a straight line that passes through the centroid coordinates of the tip portion shown at B62 and the centroid coordinates of the base portion shown at B63. As a result, the second extension line PL2 is approximately the same straight line as the straight line that passes through the centroid of the tip portion of the first treatment tool 31 recognized in the reference image and is parallel to the longitudinal direction of the first treatment tool 31. Thus, the method for calculating the second extension line PL2 differs from the method for calculating the first extension line PL1 described above in that the processor 100 does not function as a 3D construction unit 112.

[0152] Let's return to the flowchart in Figure 33 and continue the explanation. The processor 100 then performs a process (step S2118) to designate a predetermined position on the second extension line PL2 as the first measurement point 41. Specifically, for example, as shown in B65 of Figure 34, the processor 100 performs a process to set the position where the second extension line PL2 is extended by the length shown in B64 from the centroid coordinate of the tip portion of B62 as the first measurement point 41.

[0153] Furthermore, similar to the reference image shown at B60 in Figure 34, the processing shown in Figure 33 is also performed on the reference image. Subsequently, as in Figure 4, the processor 100 performs the second measurement point designation process (step S120) and the 3D position information calculation process (step S130), thereby completing the position designation calculation process (step S100). As a result, stereo matching is performed based on the stereo image, and the 3D position information of the first measurement point 41 is obtained.

[0154] Furthermore, if you want to specify a second measurement point 42 at a position away from the tip of the second treatment tool 32 using the method shown in Figure 33, you can select the first treatment tool 31 and the second treatment tool 32 in step S2112, and then perform steps S2114, S2116, and S2118 for the selected first treatment tool 31 and second treatment tool 32.

[0155] Based on the above, in the image processing apparatus 10 of this embodiment, the processor 100 determines 3D position information by designating the point where a straight line based on the 3D position information of at least a part of the shaft portion of the first treatment instrument 31 intersects with the subject as the first measurement point 41. In this way, the first measurement point 41 can be specified for any position in an area that the tip of the first treatment instrument 31 does not directly reach. For example, it may be desired to specify a desired position on the surface of a given tissue as the first measurement point 41, but the tip of the first treatment instrument 31 may not be able to reach that desired position due to reasons such as insufficient length of the shaft portion of the first treatment instrument 31. In this regard, by applying the method of this embodiment, the position on the extension of the direction in which the tip of the first treatment instrument 31 is pointing can be treated as the first measurement point 41.

[0156] Furthermore, the processor 100 may calculate the three-dimensional position information of the first measurement point 41 based on a predetermined position on a straight line derived from the two-dimensional position information of at least a portion of the shaft of the first treatment tool 31 and the two-dimensional position information of the tip of the first treatment tool 31. By doing so, the same effects as described above can be obtained.

[0157] Furthermore, for example, if the processor 100 has measured the desired distance, it may enable tracking of the first measurement point 41 and the second measurement point 42 for a certain period of time. For example, the position specification calculation process (step S100) can be performed as shown in the flowchart of Figure 35.

[0158] In Figure 35, the processor 100 performs a process (step S102) to determine whether or not it is within a certain time after the start of tracking. If the processor 100 determines that it is not within a certain time after the start of tracking (NO in step S102), it performs a process (step S104) to determine whether or not the measured distance of the previous frame is within a specified range. More specifically, in step S104, for example, the processor 100 determines whether or not the distance measured in the previous measurement process (step S200) is within the range of the target value for measurement. In other words, the specified range in step S104 is the range of the target value when you want to measure a target distance. For example, in a procedure to remove a tumor, the removal range may be predetermined based on the results of pre-procedure examinations, etc., at a specified distance from the tumor. When measuring the width of the area to be removed in such a case, a target distance is set.

[0159] Furthermore, if the processor 100 determines that the measured distance of the previous frame is not within the specified range (NO in step S104), it performs the first measurement point specification process (step S110), the second measurement point specification process (step S120), and the 3D position information calculation process (step S130), and then terminates the flow. On the other hand, if the processor 100 determines that the measured distance of the previous frame is within the specified range (YES in step S104), it performs the process of tracking the first measurement point 41 and the second measurement point 42 in the previous frame (step S106). After executing step S106, the processor 100 continues to track the first measurement point 41 and the second measurement point 42 by image processing until a certain period of time has elapsed. Note that the image processing method related to tracking is publicly known, so its explanation is omitted.

[0160] On the other hand, if the processor 100 determines that it is within a certain time after the start of tracking (YES in step S102), it performs the aforementioned 3D position information calculation process (step S130) and terminates the flow. The specific time for this certain period can be determined by the user as appropriate. If the answer in step S102 is YES, it means that tracking of the first measurement point 41 and the second measurement point 42 is continuing. In this case, the processor 100 does not perform the first measurement point specification process (step S110) or the second measurement point specification process (step S120) again, but performs stereo matching on the stereo image related to the newly acquired frame in the 3D position information calculation process (step S130). After that, the processor 100 superimposes the stereo image after stereo matching and the image information of the first measurement point 41 and the second measurement point 42 related to step S106 performed in the previous frame and displays it on the second display DP2. If the processor 100 performs step S130 because it determined YES in step S102, it may omit the measurement process (step S200). This is because the measured distance does not change due to tracking. Furthermore, after the aforementioned period of time has elapsed, the distance information stored in the memory (not shown) may be erased. As a result, after a certain period of time has elapsed since the start of tracking, the processor 100 can determine NO in step S104.

[0161] Using Figures 36 and 37, we will explain an example of the screen when the process shown in Figure 35 is applied. Note that the example screens in Figures 36 and 37 show an example where the second measurement point 42 is specified as the tip of the second treatment tool 32, but this does not prevent the application of the process shown in Figure 35 when the second measurement point 42 is specified by step S122 in Figure 7.

[0162] For example, at the first timing, suppose the screen example shown in A120 is displayed on the second display DP2. In the screen example shown in A120, the processor 100 designates the tip of the first treatment instrument 31 as the first measurement point 41 and the tip of the second treatment instrument 32 as the second measurement point 42 through position designation calculation processing (step S100). Then, the processor 100 superimposes the distance information image shown in A121 and the arrow image shown in A122 onto the endoscopic image through measurement processing (step S200) and display update processing (step S300) and displays them on the second display DP2.

[0163] Furthermore, the first timing in the screen example shown in A120 is defined as the timing when the distance information related to the image shown in A121 is not within the range of the measurement target value. In this case, the processor 100 determines NO in step S104 of Figure 35 and processes in the order of first measurement point specification processing (step S110), second measurement point specification processing (step S120), and 3D position information calculation processing (step S130). In other words, the position specification calculation processing (step S100) shown in Figure 35 is substantially the same as the position specification calculation processing (step S100) in Figure 5 at the first timing shown in the screen example shown in A120.

[0164] Subsequently, the user operates at least one of the first treatment instrument 31 and the second treatment instrument 32 to change the distance between the first measurement point 41 and the second measurement point 42, and the second timing shown in the example screen A130 is reached. In the example screen A130, similar to the example screen A120, the processor 100 designates the tip of the first treatment instrument 31 as the first measurement point 41 and the tip of the second treatment instrument 32 as the second measurement point 42 through position designation calculation processing (step S100). The processor 100 also superimposes the distance information image shown in A131 and the arrow image shown in A132 onto the endoscope image through measurement processing (step S200) and display update processing (step S300) and displays them on the second display DP2.

[0165] In the example screen shown in A130, the second timing is defined as the timing when the distance information related to the image shown in A131 falls within the range of the measurement target value. In this case, the processor 100 determines YES in step S104 in Figure 35 and performs step S106. As a result, the example screen shown in A130 becomes the example screen shown in A140 in Figure 36.

[0166] In the example screen shown at A140 in Figure 36, when tracking is started, the processor 100 performs display processing on the second display DP2 so that the display mode of the icon indicating the first measurement point 41 and the display mode of the icon indicating the second measurement point 42 change, as shown at A141 and A142. This allows the user to recognize that step S106 has been executed and tracking has started. As mentioned above, just as the display of the icon indicating the first measurement point 41 is not mandatory, it is not necessary to change the display mode of the icon indicating the first measurement point 41. The same applies to the icon indicating the second measurement point 42.

[0167] In Figure 37, the display patterns of the first measurement point 41 and the second measurement point 42 are changed to allow the user to distinguish whether or not tracking has started. However, the system is not limited to this, and for example, the system may notify the user that tracking has started by a predetermined means without changing the display patterns of the first measurement point 41 and the second measurement point 42. The predetermined means may include, for example, displaying a message indicating that tracking has started on the second display DP2, displaying a mark including a predetermined color on the second display DP2, or outputting a predetermined sound. Although not shown in the figure, the processor 100 may also display the remaining time until tracking is completed on the second display DP2 after tracking has started.

[0168] Then, at the third timing, which is after the second timing but before a certain amount of time has elapsed since the second timing, the screen shown in A150 becomes as shown. The user determines that the desired distance has been measured and operates the first treatment tool 31 further for reasons such as performing a subsequent procedure. Therefore, the position of the tip of the first treatment tool 31 at the third timing is different from the position of the tip of the first treatment tool 31 at the second timing. Similarly, the position of the tip of the second treatment tool 32 at the third timing is different from the position of the tip of the second treatment tool 32 at the second timing.

[0169] In this case, the processor 100, based on its determination of YES in step S102 related to the frame, performs stereo matching based on the stereo image captured by the imager of the endoscope 20 in that frame. As a result, the positions of the first treatment instrument 31 and the second treatment instrument 32 in the screen example A150 are different from the positions of the first treatment instrument 31 and the second treatment instrument 32 in the screen example A140. Meanwhile, since the processor 100 has executed step S106 related to the previous frame, it is tracking the first measurement point 41 and the second measurement point 42 at the second timing. The tracked first measurement point 41 is displayed at the position shown in A151 at the third timing, and the tracked second measurement point 42 is displayed at the position shown in A152 at the third timing. The position shown in A151 is the same as the position shown in A141, and the position shown in A152 is the same as the position shown in A142. Furthermore, since the first measurement point designation process (step S110) is not performed at the third timing, the position of the tip of the first treatment tool 31 and the position shown in A151 will no longer coincide. Similarly, since the second measurement point designation process (step S120) is not performed at the third timing, the position of the tip of the second treatment tool 32 and the position shown in A152 will no longer coincide.

[0170] Figure 35 shows an example of a process for tracking the first measurement point 41 and the second measurement point 42 when the desired distance has been measured. However, the process is not limited to this, and for example, the user may be able to track the first measurement point 41 and the second measurement point 42 at any time. For example, although not shown in the figure, the image processing device 10, endoscope 20, etc., may be equipped with user-operable operation buttons, and step S104 in Figure 35 may be a process to determine whether the processor 100 has received an instruction signal based on the user's operation of the operation buttons. Thus, in the image processing device 10 of this embodiment, when a distance measurement value is obtained or an instruction signal is input by the user, the processor 100 tracks the first measurement point 41 and the second measurement point 42 at the time the measurement value was obtained (step S106), and displays the tracking positions of the first measurement point 41 and the second measurement point 42 on the display DP (second display DP2). In this way, the user can perform tasks other than measurement using the treatment tools used for measurement while keeping track of the desired measurement value. This can improve the convenience of the procedure.

[0171] Alternatively, for example, a first treatment tool 31 that has a predetermined texture (e.g., a pattern) applied to it may be used for treatment, and the three-dimensional position information calculation process (step S130) shown in Figure 5 may be performed to calculate the three-dimensional position information of the first measurement point 41.

[0172] The instrument used in the procedure according to this embodiment includes a metal part and a handle part. However, as shown in B70 of Figure 38, the difference in pixel values ​​of each pixel related to the metal part in the reference image and the reference image is small, which may reduce the accuracy of stereo matching. The same applies to the handle part. Therefore, by applying a predetermined texture to the metal part and the handle part of the first instrument 31, for example, as shown in B71 or B72, the accuracy of matching in stereo matching can be improved. The predetermined texture is not limited to the examples shown in B71 or B72, and various modifications are possible.

[0173] The specified texture can be achieved, for example, by directly painting the first treatment tool 31 with color.

[0174] Alternatively, the processor 100 may perform a process of irradiating the first treatment instrument 31 with a predetermined pattern of light from a light source device included in the tip of the endoscope 20, acquire a reference image and a reference image from the endoscope 20 while the predetermined pattern of light is irradiated onto the first treatment instrument 31, and perform stereo matching.

[0175] As mentioned above in Figure 14, if the position of the tip of the second treatment tool 32 is set as the second measurement point 42, a predetermined texture may be applied to both the first treatment tool 31 and the second treatment tool 32, but a certain effect can be expected even if only the first treatment tool 31 is applied to the predetermined texture. Therefore, the image processing apparatus 10 of this embodiment includes the first treatment tool 31, and the first treatment tool 31 is fitted with a texture to improve the measurement accuracy of the first measurement point 41. By doing so, the processor 100 can determine the parallax more accurately, and thus calculate the three-dimensional position information of the first measurement point 41 more accurately. As a result, the distance between the first measurement point 41 and the second measurement point 42 can be measured more accurately.

[0176] Although this embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible without substantially departing from the novelty and effects of this disclosure. Therefore, all such modifications are included within the scope of this disclosure. For example, any term that appears at least once in the specification or drawings together with a broader or synonymous term may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of this embodiment and its modifications are also included within the scope of this disclosure. In addition, the configuration and operation of the image processing apparatus, endoscope system, image processing method, program, etc., are not limited to those described in this embodiment, and various modifications are possible. [Explanation of Symbols]

[0177] 1…Endoscope system, 10…Image processing device, 11…First interface, 12…Second interface, 20…Endoscope, 31…First treatment instrument, 32…Second treatment instrument, 41…First measurement point, 42…Second measurement point, 100…Processor, 110…Position specification calculation unit, 112…3D construction unit, 114…Measurement point selection unit, 120…Distance calculation unit, 122…Measurement execution determination unit, DP…Display, DP1…First display, DP2…Second display, LV…Line of sight vector, P-A1, P-A2, P-A3, P-A4, P-B1, P-B2, P-B3, P-B4…Pattern, PL1…First extension line, PL2…Second extension line

Claims

1. Includes a processor that performs display processing for the display, The aforementioned processor, Using the endoscopic image of the subject acquired by the endoscope, the three-dimensional positional information of the first treatment instrument and a predetermined part within the endoscopic image is calculated. Based on the three-dimensional positional information of the first treatment tool and the predetermined portion, the distance between the first measurement point on the tip side of the first treatment tool and the second measurement point relating to the predetermined portion is measured. An image processing apparatus characterized by performing a display process on the display to indicate whether or not the measurement of the distance is in a stable state.

2. In the image processing apparatus of claim 1, The three-dimensional position information of the predetermined portion is The three-dimensional position information of the second treatment instrument, The aforementioned processor, An image processing apparatus characterized by measuring the distance between a first measurement point on the tip side of the first treatment tool and a second measurement point which is the position on the tip side of the second treatment tool, based on the three-dimensional position information of the first treatment tool and the predetermined part.

3. In the image processing apparatus of claim 1, The three-dimensional position information of the predetermined portion is The 3D position information of the second measurement point, which is the position of the subject corresponding to the position on the display as specified by the user, The aforementioned processor, An image processing apparatus characterized by measuring the distance between the first measurement point on the tip side of the first treatment tool and the second measurement point based on the three-dimensional position information of the first treatment tool and the predetermined part.

4. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized in that it determines that the measurement of the distance is in a stable state when the first measurement point and the second measurement point are measured stably over a predetermined period of time, or when the distance is measured stably over a predetermined period of time.

5. In the image processing apparatus of claim 4, The aforementioned processor, An image processing apparatus characterized in that it determines that the state is stable when the amount of movement of at least the first measurement point is within a first predetermined range within a first predetermined period.

6. In the image processing apparatus of claim 5, The aforementioned processor, An image processing apparatus characterized by displaying image information on the display that allows at least the amount of movement of the first measurement point to be within a first predetermined range.

7. In the image processing apparatus of claim 4, The aforementioned processor, An image processing apparatus characterized in that it determines that the state is stable when the amount of change in the distance is within a second predetermined range within a second predetermined period.

8. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized by changing the display mode of the measured distance value depending on whether the measurement of the distance is in the stable state.

9. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized by displaying the period of the stable state of the distance measurement in a graph.

10. In the image processing apparatus of claim 1, The aforementioned processor, If the endoscopic image remains stable over the predetermined period, the determined value of the distance is calculated. An image processing apparatus characterized by displaying on the display an indication that the endoscopic image is stable during the predetermined period.

11. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized in that, when the first treatment instrument is parallel to the parallax direction in the stereo view of the endoscopic image, an instruction to tilt the first treatment instrument with respect to the parallax direction is displayed on the display.

12. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized in that, when the first treatment instrument is parallel to the parallax direction in the stereo view of the endoscopic image, an object indicating the direction in which the first treatment instrument should be tilted is displayed on the display.

13. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized by measuring the distance by performing a smoothing process in the time direction or the smoothing process in the spatial direction on at least the first measurement point.

14. In the image processing apparatus of claim 13, The aforementioned processor, An image processing apparatus characterized by determining the three-dimensional position information of the first measurement point using the median or average value of the depth coordinates in the area surrounding the measurement position coordinates of the first measurement point within the endoscopic image, and a line-of-sight vector based on the measurement position coordinates.

15. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized in that it determines the three-dimensional position information by setting the point where a straight line based on the three-dimensional position information of at least a part of the shaft of the first treatment instrument and the three-dimensional position information of the tip of the first treatment instrument intersects with the subject as the first measurement point.

16. In the image processing apparatus of claim 1, The aforementioned processor, An image processing apparatus characterized by calculating the three-dimensional position information of the first measurement point based on a predetermined position on a straight line, which is determined by the two-dimensional position information of at least a part of the shaft of the first treatment tool and the two-dimensional position information of the tip of the first treatment tool.

17. In the image processing apparatus of claim 1, The aforementioned processor, An image processing device characterized in that, when the measured distance is obtained or when an instruction signal is input by the user, tracking is performed between the first measurement point and the second measurement point at the time the measured value was obtained, and the tracking positions of the first measurement point and the second measurement point are displayed on the display.

18. An image processing apparatus according to any one of claims 1 to 17, Endoscope and, An endoscopic system characterized by including the following.

19. The process of displaying the endoscopic image of the subject acquired by the endoscope on a display, Using the endoscopic image of the subject, a process is performed to calculate the three-dimensional positional information of the first treatment instrument and a predetermined part within the endoscopic image. A process for measuring the distance between a first measurement point on the tip side of the first treatment tool and a second measurement point relating to the predetermined part, based on the three-dimensional position information of the first treatment tool and the predetermined part, A process that displays whether the measurement of the aforementioned distance is in a stable state, An image processing method characterized by being performed by a computer.

20. The process of displaying the endoscopic image of the subject acquired by the endoscope on a display, Using the endoscopic image of the subject, a process is performed to calculate the three-dimensional positional information of the first treatment instrument and a predetermined part within the endoscopic image. A process for measuring the distance between a first measurement point on the tip side of the first treatment tool and a second measurement point relating to the predetermined part, based on the three-dimensional position information of the first treatment tool and the predetermined part, A process that displays whether the measurement of the aforementioned distance is in a stable state, A program characterized by being executed by a computer.

Images