IMAGE PROCESSING DEVICE, IMAGE PROCESSING PROGRAM, AND OPERATION METHOD OF ENDOSCOPIC SYSTEM

Abstract

Provided are an image processing device, a method, and a program that can suitably aid a treatment using a tool. This image processing device processes images captured by an endoscope through which a tool can protrude from a tip thereof, the image processing device comprising a processor. The processor performs processing for acquiring images captured by the endoscope, processing for displaying the acquired images in a display, processing for detecting a region of interest from the acquired images, processing for determining whether the region of interest is present in a location reachable by the tool protruding from the tip of the endoscope, and processing for issuing a notification if it is determined that the region of interest is present in a location reachable by the tool protruding from the tip of the endoscope.

Description

[Technical field]

[0001] The present invention relates to an image processing device, IMAGE PROCESSING PROGRAM AND OPERATION METHOD OF ENDOSCOPIC SYSTEM In particular, an image processing device for processing images captured by an endoscope capable of projecting an instrument from its tip; IMAGE PROCESSING PROGRAM AND OPERATION METHOD OF ENDOSCOPIC SYSTEM Regarding. [Background technology]

[0002] As a technology for supporting endoscopic examinations, a technology is known in which a region of interest, such as a lesion, is automatically detected from an image captured by an endoscope by image processing, and then notified.

[0003] In order to support examinations efficiently, Patent Document 1 proposes that the movement of an endoscope operator is determined from images captured by the endoscope, and images in which a predetermined movement such as treatment is being performed are excluded from detection targets for the region of interest. In other words, when a movement such as treatment is being performed, it is highly likely that the region of interest has already been found, so it is considered that there is little need to detect it by image processing, and it is excluded from detection targets, thereby improving efficiency. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. WO2017 / 002184 Summary of the Invention [Problem to be solved by the invention]

[0005] By the way, tissue sampling (biopsy) may be performed during endoscopic examination. Tissue sampling is performed through a forceps port provided at the tip of the endoscope. When sampling tissue, it is expected that the target lesion will be notified at the appropriate time to support the surgeon.

[0006] However, in Patent Document 1, detection of the target lesion is not performed during treatment, and therefore there is a drawback in that the surgeon cannot be appropriately supported.

[0007] The present invention has been made in view of the above circumstances, and provides an image processing device capable of appropriately supporting treatment using an instrument, IMAGE PROCESSING PROGRAM AND OPERATION METHOD OF ENDOSCOPIC SYSTEM The purpose is to provide. [Means for solving the problem]

[0008] (1) An image processing device that processes images taken with an endoscope capable of protruding an instrument from its tip, the image processing device having a processor that performs the following processes: acquiring the image taken with the endoscope; displaying the acquired image on a display; detecting a region of interest from the acquired image; determining whether or not the region of interest exists in a position that can be reached by an instrument protruding from the tip of the endoscope; and reporting when it is determined that the region of interest exists in a position that can be reached by an instrument protruding from the tip of the endoscope.

[0009] (2) An image processing device according to (1), in which a processor determines whether the distance from a reference point set in the image to the area of ​​interest is less than or equal to a first threshold value, and determines whether the area of ​​interest is in a position that can be reached by an instrument protruding from the tip of the endoscope.

[0010] (3) An image processing device as described in (2), wherein the reference point is the center of the image.

[0011] (4) An image processing device according to (1), in which the processor determines whether the distance from the extension of the instrument protruding from the tip of the endoscope to the area of ​​interest is less than or equal to a second threshold value, and determines whether the area of ​​interest is in a position that can be reached by the instrument protruding from the tip of the endoscope.

[0012] (5) An image processing device according to any one of (1) to (4), wherein the processor further performs processing to determine whether or not an obstacle exists between the area of ​​interest and the instrument from the acquired image, and the processor issues an alert when it determines that no obstacle exists and that the area of ​​interest exists in a position reachable by an instrument protruding from the tip of the endoscope.

[0013] (6) An image processing device according to any one of (1) to (4), wherein the processor further performs processing to detect an instrument from the acquired image, and the processor detects the instrument from the image and performs processing to notify when it is determined that an area of ​​interest exists in a position reachable by the instrument protruding from the tip of the endoscope.

[0014] (7) An image processing device according to any one of (1) to (6), in which the processor changes the display image displayed on the display and notifies the user.

[0015] (8) The image processing device of (7), wherein the processor changes the display of a display area of ​​the image set in the display image to provide notification.

[0016] (9) An image processing device according to (8), wherein the processor notifies the user by displaying a figure indicating the area of ​​interest superimposed on the image to be displayed in the display area.

[0017] (10) An image processing device as described in (7), wherein, when an area of ​​interest is detected, the processor further performs processing to display a figure indicating the area of ​​interest superimposed on an image to be displayed in the display area of ​​the image set in the display image, and the processor changes the display of the figure to notify the user.

[0018] (11) An image processing device according to (10), wherein the processor changes at least one of the color, shape, brightness, and line type of the graphic to change the display of the graphic.

[0019] (12) An image processing device according to (7), wherein the processor further performs processing to display information indicating the protruding direction of the instrument superimposed on the image to be displayed in the display area of ​​the image set in the display image, and the processor changes the display of the information indicating the protruding direction of the instrument to notify the user.

[0020] (13) An image processing device according to (12), in which the processor displays a straight line along the protruding direction of the instrument as information indicating the protruding direction of the instrument.

[0021] (14) An image processing device according to (13), wherein the processor changes at least one of the color, brightness, and line type of the line to change the display of the line.

[0022] (15) The image processing device of (7), wherein the processor changes the display of an area other than the display area of ​​the image set in the display image to provide notification.

[0023] (16) The image processing device of (15), wherein the processor notifies the user by displaying information in an area other than the display area of ​​the image.

[0024] (17) An image processing device as defined in (16), in which the processor displays a message or a graphic as information.

[0025] (18) Any one of the image processing devices (1) to (17) in which the processor outputs a sound to notify the user.

[0026] (19) Any one of the image processing devices (1) to (18) in which the region of interest is a lesion area.

[0027] (20) Any one of the image processing devices (1) to (19) in which the region of interest is an organ.

[0028] (21) An image processing method including the steps of acquiring an image taken with an endoscope capable of protruding an instrument from its tip, displaying the acquired image on a display, detecting a region of interest from the acquired image, determining whether or not the region of interest exists in a position that can be reached by an instrument protruding from the tip of the endoscope, and reporting when it is determined that the region of interest exists in a position that can be reached by an instrument protruding from the tip of the endoscope.

[0029] (22) An image processing program that enables a computer to perform the following functions: acquire an image taken with an endoscope capable of protruding an instrument from its tip; display the acquired image on a display; detect a region of interest from the acquired image; determine whether or not the region of interest exists in a position that can be reached by an instrument protruding from the tip of the endoscope; and report an indication when it is determined that the region of interest exists in a position that can be reached by an instrument protruding from the tip of the endoscope. Effect of the Invention

[0030] According to the present invention, treatment using an instrument can be appropriately supported. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a block diagram showing an example of a system configuration of an endoscope system to which the present disclosure is applied; [Diagram 2] FIG. 1 shows an example of an endoscope. [Diagram 3] FIG. 1 is a perspective view showing an example of a configuration of a tip end of an insertion portion of an endoscope; [Figure 4] Block diagram of one function realized by the image processing device [Diagram 5] FIG. 13 is a diagram showing an example of a display image when a treatment tool is protruded from a forceps port. [Figure 6] FIG. 13 is a diagram showing an example of a display image. [Figure 7] FIG. 13 is a diagram showing an example of a display image. [Figure 8] A flowchart showing the procedure of a process for displaying an endoscopic image on a display by an image processing device. [Figure 9] FIG. 13 is a diagram showing a first modified example of a display image. [Figure 10] FIG. 11 is a diagram showing a second modified example of a display image. [Figure 11] FIG. 1 is a block diagram showing an example of a system configuration of an ultrasound endoscope system to which the present disclosure is applied. [Figure 12] FIG. 1 shows an example of an ultrasonic endoscope. [Figure 13] FIG. 1 is a perspective view showing an example of the configuration of the tip of an insertion portion of an ultrasonic endoscope. [Figure 14] FIG. 11 is a block diagram of a function realized by an image processing device according to a second embodiment. [Figure 15] FIG. 13 is a diagram showing an example of a display image when a treatment tool is protruded from a treatment tool protrusion port; [Figure 16] FIG. 13 is a diagram showing an example of a display image. [Figure 17] FIG. 13 is a diagram showing an example of a display image. [Figure 18] A flowchart showing the procedure of a process for displaying an endoscopic image on a display by an image processing device. [Figure 19] FIG. 13 is a diagram showing a first modified example of a display image. [Figure 20] FIG. 11 is a diagram showing a second modified example of a display image. [Figure 21] FIG. 13 is a block diagram of a function implemented by an image processing device according to a third embodiment. [Figure 22] FIG. 13 is a diagram showing an example of a display image. [Diagram 23] FIG. 13 is a diagram showing an example of a display image. [Figure 24] A flowchart showing the procedure of a process for displaying an endoscopic image on a display by an image processing device. [Diagram 25] FIG. 13 is a block diagram of a function realized by an image processing device according to a fourth embodiment. [Figure 26] A flowchart showing the procedure of a process for displaying an endoscopic image on a display by an image processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0033] [First embodiment] [System Configuration] FIG. 1 is a block diagram showing an example of a system configuration of an endoscope system to which the present disclosure is applied.

[0034] As shown in the figure, the endoscope system 1 of this embodiment includes an endoscope 10, a light source device 100, a processor device 200, an image processing device 300, and a display 400.

[0035] FIG. 2 is a diagram illustrating an example of an endoscope.

[0036] The endoscope 10 is a flexible endoscope (electronic endoscope) that allows a treatment tool to be projected from a tip thereof. As shown in FIG. 2, the endoscope 10 is mainly composed of an insertion section 12, an operation section 14, and a connection section 16.

[0037] The insertion section 12 is a portion that is inserted into a body cavity. The insertion section 12 is made up of, in order from the distal end side, a distal end section 12A, a bending section 12B that can be bent, and a flexible section 12C that has flexibility.

[0038] FIG. 3 is a perspective view showing an example of the configuration of the tip of the insertion portion of an endoscope.

[0039] As shown in the figure, the tip portion 12A is provided with an observation window 20, an illumination window 22, a nozzle 24, a forceps port 26, etc. on its end surface.

[0040] The observation window 20 is a window for observation. An imaging unit is provided inside the observation window 20. The imaging unit includes an imaging optical system and an image sensor. The image sensor is, for example, a color CMOS (Complementary Metal-Oxide Semiconductor) image sensor having a predetermined color filter array (e.g., Bayer array, etc.), a color CCD (Charge Coupled Device) image sensor, etc.

[0041] The illumination window 22 is a window for illumination. Illumination light supplied from the light source device 100 is emitted from this illumination window 22. As shown in Fig. 3, the endoscope 10 of this embodiment is provided with illumination windows 22 in two places.

[0042] The nozzle 24 selectively sprays a liquid (for example, water) or a gas (for example, air) toward the observation window 20. When the observation window 20 becomes dirty, the liquid or gas is sprayed from the nozzle 24 to wash away the dirt.

[0043] The forceps port 26 is an outlet for a treatment tool 500 such as forceps. 38 The treatment tool 500 inserted from the forceps port 26 protrudes from the forceps port 26. The treatment tool 500 is an example of an instrument.

[0044] The bending portion 12B can be bent up, down, left, or right by operating an angle knob 30 provided on the operation portion 14. This allows the tip portion 12A to be oriented in a desired direction.

[0045] 2, the operation unit 14 is a portion that is held by the surgeon (user) to operate the endoscope 10. The operation unit 14 is provided with various operation members. For example, an angle knob 30 for performing bending operation of the bending portion 12B, an air / water supply button 32 for performing an air / water supply operation, a suction button 34 for performing a suction operation, a shutter button 36 for taking still images, etc. are provided.

[0046] 2, the operation section 14 is provided with a forceps insertion port 38 for inserting a treatment tool 500 such as forceps. The treatment tool 500 inserted from the forceps insertion port 38 passes through a forceps channel 40 provided in the insertion section 12 and protrudes from the forceps port 26 at the tip.

[0047] The connection part 16 is a part for connecting the endoscope 10 to the light source device 100 and the processor device 200. The connection part 16 is made of a flexible cord. The tip of the connection part 16 is provided with a connector 16A for connecting to the light source device 100 and a connector 16B for connecting to the processor device 200.

[0048] The light source device 100 includes a light source and supplies light from the light source as illumination light to the endoscope 10. The illumination light supplied to the endoscope 10 passes through a light guide (not shown) and is emitted from an illumination window 22 at the tip. For example, white light is used as the light from the light source.

[0049] The processor device 200 takes in an imaging signal output from the endoscope 10, performs a predetermined signal processing, and performs processing to generate an observation image (endoscopic image) by the endoscope 10. The processor device 200 also performs overall control of the entire system. The processor device 200 is composed of a computer equipped with, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The processor device 200 realizes a function of generating an endoscopic image and a function of overall control of the entire system by the CPU executing a predetermined program. The ROM stores various programs executed by the CPU, and data necessary for control, processing, etc. The RAM provides the CPU with a working memory space.

[0050] The image processing device 300 acquires an endoscopic image output from the processor device 200 and performs processing to display it on the display 400. The image processing device 300 also performs processing to support a predetermined treatment by an operator using a treatment tool 500 via the display 400. Specifically, when performing tissue sampling, predetermined information is displayed on the display 400 so that the operator can easily sample a target (lesion, etc.). This point will be described later. The image processing device 300 is composed of, for example, a computer equipped with a CPU, a ROM, a RAM, etc. The image processing device 300 functions as an image processing device by the CPU executing a predetermined program. The ROM stores various programs executed by the CPU and data necessary for various processes and controls. The RAM provides the CPU with a working memory space.

[0051] FIG. 4 is a block diagram of a function realized by the image processing device.

[0052] As shown in the figure, the image processing device 300 has the functions of an image acquisition unit 300A, an attention area detection unit 300B, a determination unit 300C, a display image generation unit 300D, and a display control unit 300E. These functions are realized by the CPU executing a predetermined program (image processing program).

[0053] The image acquiring section 300A acquires an endoscopic image from the processor device 200. The endoscopic image is a moving image. The image acquiring section 300A sequentially acquires images of each frame constituting the moving image.

[0054] The area of ​​interest detection unit 300B detects an area of ​​interest from the image of each frame acquired by the image acquisition unit 300A. The "area of ​​interest" here refers to the area to be treated. In the case of tissue sampling, it is the lesion that is the target. The area of ​​interest detection unit 300B detects an area of ​​interest from an endoscopic image, for example, by image recognition. The image recognition can be performed, for example, by using an image recognition model generated by machine learning (for example, deep learning, etc.). Other known methods can also be used.

[0055] The region of interest is detected by identifying its position within the image. The position is acquired, for example, as information on the pixel position of the center or center of gravity of the region of interest present in the endoscopic image.

[0056] When a region of interest is detected, the determination unit 300C determines from the endoscopic image whether or not the region of interest is located at a position that can be easily treated with the treatment tool 500. A position that can be easily treated with the treatment tool 500 is a position that can be reached by the treatment tool 500. Therefore, the determination unit 300C determines from the image whether or not the region of interest is located at a position that can be reached by the treatment tool 500. The determination is performed as follows.

[0057] FIG. 5 is a diagram showing an example of a display image (an image displayed on a display) when a treatment tool is projected from a forceps port.

[0058] As shown in the figure, the display image 410 is composed of a rectangular image having a predetermined aspect ratio. The endoscopic image 412 is displayed in a predetermined display area 414 set within the display image 410. In the example shown in the figure, an area in which the top and bottom of a circle are cut out is set as the display area 414 for the endoscopic image 412.

[0059] When the treatment tool 500 is protruded from the forceps opening 26, the treatment tool 500 is reflected in the endoscopic image 412. Since the positional relationship between the forceps opening 26 and the observation window 20 is fixed, the position where the treatment tool 500 is reflected is always constant. In addition, the protruding direction (the direction indicated by the arrow in FIG. 5) is also constant. Therefore, the range that the treatment tool 500 can reach in the endoscopic image can be obtained in advance. This range can be determined, for example, by a circle 416 with a radius r centered on a predetermined reference point P, as shown in FIG. 5. In general, when the treatment tool 500 is protruded from the forceps opening 26, its tip is located at or near the center of the endoscopic image 412. Therefore, the range that the treatment tool 500 can reach can be determined by setting the center of the endoscopic image 412 to the reference point P.

[0060] The determination unit 300C determines whether the attention area exists within the range of a circle 416 having a radius r centered on the center (reference point P) of the endoscopic image 412, and determines whether the attention area exists in a position reachable by the treatment tool 500. In other words, the determination unit 300C determines whether the attention area exists in a position reachable by the treatment tool 500 by determining whether the distance from the center (reference point P) of the endoscopic image 412 to the attention area is r or less (first threshold or less). r is an example of the first threshold. If the attention area exists within the range of the circle 416 having a radius r centered on the center (reference point P) of the endoscopic image 412 (if the distance from the center (reference point P) of the endoscopic image 412 to the attention area is r or less), it determines that the attention area exists in a position reachable by the treatment tool 500. On the other hand, if the area of ​​interest does not exist within the range of a circle 416 of radius r centered on the center (reference point P) of the endoscopic image 412 (if the distance from the center (reference point P) of the endoscopic image 412) to the area of ​​interest exceeds r), it is determined that the area of ​​interest does not exist in a position reachable by the treatment tool 500.

[0061] The display image generating section 300D generates a display image to be displayed on the display 400 from the endoscopic image acquired by the image acquiring section 300A. The display image generating section 300D generates a display image based on the judgment result of the judging section 300C. That is, when the judging section 300C judges that the attention area exists in a position reachable by the treatment tool 500, the display image is changed to notify the surgeon that the attention area exists in a position reachable by the treatment tool 500. Specifically, the display area of ​​the endoscopic image 412 is 414 Change the display.

[0062] Fig. 6 and Fig. 7 are diagrams showing examples of a display image. Fig. 6 shows an example of a case where the attention area does not exist in a position reachable by the treatment tool 500. Fig. 7 shows an example of a case where the attention area exists in a position reachable by the treatment tool 500.

[0063] When the attention area does not exist in a position that the treatment tool 500 can reach, only the endoscopic image 412 is displayed in the display area 414 of the endoscopic image 412 as shown in Fig. 6. In Fig. 6, a cross (X) mark 418 shown by a dashed line indicates the position of the attention area. In the example shown in Fig. 6, since the position of the attention area (the position of the mark 418) is outside the range of the circle 416, it is determined that the attention area does not exist in a position that the treatment tool 500 can reach. In Fig. 6, the circle 416 indicating the range that the treatment tool 500 can reach and the mark 418 indicating the position of the attention area are displayed for convenience of explanation, but these are not displayed in the actual display image 410.

[0064] On the other hand, when the attention area exists in a position reachable by the treatment tool 500, a cross (X) mark 420 indicating the position of the attention area is displayed superimposed on the endoscopic image 412 in the display area 414 of the endoscopic image 412 as shown in Fig. 7. Note that, for convenience of explanation, a circle 416 indicating the range reachable by the treatment tool 500 is displayed in Fig. 7, but this circle 416 is not displayed in the actual display image 410.

[0065] The timing of treatment can be appropriately notified to the surgeon by displaying the mark 420 indicating the position of the region of interest only when the region of interest is in a position reachable by the treatment tool 500. The mark 420 is an example of a figure indicating the region of interest.

[0066] The display control unit 300E causes the display 400 to display the display image generated by the display image generating unit 300D.

[0067] The display 400 is configured, for example, by a liquid crystal display, an organic EL display (Organic ElectroLuminescent, OEL), or the like.

[0068] [Operation] Here, an example will be described in which tissue sampling is performed using biopsy forceps.

[0069] 2, the biopsy forceps (treatment tool) 500 has an insertion section 510 and an operation section 512. The insertion section 510 is flexible and has a tip claw section 514 at its tip. The operation section 512 has a handle 512A and a slider 512B, and the tip claw section 514 opens and closes by sliding the slider 512B back and forth. For tissue sampling, the tip claw section 514 grasps and samples a target (lesion, etc.).

[0070] FIG. 8 is a flowchart showing a processing procedure (image processing method) for displaying an endoscopic image on a display by an image processing device.

[0071] First, an endoscopic image is acquired from the processor device 200 (step S1). The endoscopic image is acquired sequentially on a frame-by-frame basis.

[0072] Next, a region of interest is detected from the acquired endoscopic image (step S2). The region of interest here is a lesion that is a target for tissue sampling.

[0073] Next, based on the result of detection of the region of interest, it is determined whether or not the region of interest has been detected (step S3).

[0074] If it is determined that the region of interest has not been detected, a normal display image is generated (step S4). Then, the generated display image is displayed on the display 400 (step S7). Here, the normal display image is an endoscopic image in which no mark indicating the position of the region of interest is displayed (an image in which the display of the mark is turned off) (see FIG. 6).

[0075] On the other hand, if it is determined that the region of interest has been detected, it is further determined whether or not the region of interest exists in a position reachable by the biopsy forceps 500 (step S5).

[0076] If it is determined that the area of ​​interest does not exist in a position reachable by the biopsy forceps 500, a normal display image is generated (step S4), as in the case where the area of ​​interest is not detected, and the generated display image is displayed on the display 400 (step S7).

[0077] On the other hand, if it is determined that the region of interest is present in a position reachable by the biopsy forceps 500, a display image displaying a mark indicating the position of the region of interest is generated (step S6). Then, the generated display image is displayed on the display 400 (step S7). The display image displaying the mark indicating the position of the region of interest is an image in which a mark 420 indicating the position of the region of interest is superimposed on an endoscopic image 412 (an image with the mark display turned on), as shown in FIG.

[0078] Thereafter, it is determined whether or not imaging by the endoscope 10 has ended (step S8). When imaging has ended, the display process ends. The end of imaging is determined, for example, by whether or not the image of the next frame has been input. If imaging has not ended, the process returns to step S1, and the above series of processes are performed.

[0079] In this way, according to the endoscope system 1 of the present embodiment, when the region of interest is located at a position reachable by the biopsy forceps 500, a mark indicating the position of the region of interest is displayed in the displayed image. This allows the surgeon to be notified of the target lesion at an appropriate time during tissue sampling, and appropriately supports the surgeon in performing the treatment.

[0080] [Variations] [Modification of notification] In the above embodiment, a mark indicating the position of the region of interest is displayed superimposed on the endoscopic image to notify the user that the region of interest is located in a position that can be reached by the treatment tool, but the method of notification is not limited to this. Modified examples of the method of notification will be described below.

[0081] (1) First modified example of the notification method 9 is a diagram showing a first modified example of a display image. A display image 410A shown in FIG. 9(A) is a display image of a treatment tool. 500 A display image 410B shown in Fig. 4(B) is a display image when the attention area is located in a position reachable by the treatment tool. A display image 410B shown in Fig. 4(B) is a display image when the attention area is located in a position reachable by the treatment tool.

[0082] As shown in FIGS. 9(A) and (B), in this example, when an attention area is detected from endoscopic images 412A and 412B, marks 420A and 420B indicating the position are displayed superimposed on endoscopic images 412A and 412B.

[0083] However, 500 The display form of the marks 420A and 420B differs depending on whether or not the attention area is located in a position reachable by the user.

[0084] As shown in FIG. 9(A), 500 When the region of interest is not located in a position that can be reached by the treatment tool, the mark 420A is displayed in a thin line. 500 When the attention area is located at a position that can be reached by the treatment tool, the mark 420B is displayed in a thick line as shown in FIG. 500 The marks 420A and 420B are displayed with different degrees of emphasis depending on whether the region of interest is located in a position reachable by the treatment tool or not. 500 When the region of interest is located in a position that can be reached by the treatment tool, the degree of emphasis of the mark 420B is intensified. 500If the region of interest is not located in a position that is reachable by the target object, the emphasis of the mark 420A is weakened.

[0085] In this way, by changing the display form of the mark depending on whether or not the area of ​​interest is located in a position that can be reached by a treatment tool, it is also possible to notify that the area of ​​interest is located in a position that can be reached by a treatment tool.

[0086] In this example, the mark display form is switched by changing the line thickness of the mark, but the method of switching the mark display form is not limited to this. The mark display form can be switched by changing at least one of the color, shape, brightness, and line type of the mark. In this case, it is preferable to switch the mark display form so that the degree of emphasis is increased when the attention area is located at a position reachable by the treatment tool.

[0087] In addition, when the region of interest is located in a position that can be reached by the treatment tool, the mark may be made to flash to notify the user.

[0088] The shape of the mark indicating the position of the attention area is not limited to a cross (X) and various shapes can be used. Also, a figure surrounding the attention area (for example, a rectangular frame) can be displayed as a figure indicating the attention area.

[0089] (2) Second modified example of the notification method In the above embodiment, the notification is given by changing the display of the display area of ​​the endoscopic image set in the display image. The notification may also be given by changing the display of an area other than the display area of ​​the endoscopic image.

[0090] 10 is a diagram showing a second modified example of a display image. In the figure, a dashed cross (X) mark 418 indicates the position of the attention area. This mark 418 is displayed for the convenience of explanation and is not actually displayed in the display image.

[0091] The display image 410A shown in FIG. 500 A display image 410B shown in FIG. 10B is a display image when the attention area is not located in a position that can be reached by the treatment tool. 500 4 is a display image when the attention area is located at a position reachable by the user.

[0092] As shown in FIG. 10(B), 500 When the region of interest is located in a position that can be reached by the treatment tool, a message 422 is displayed in an area outside the display area of ​​the endoscopic image 412B. In this example, the message 422 displays the word "Push!" 500 If the attention area is not located in a position that can be reached by the user, the message is not displayed, as shown in FIG. 10(A).

[0093] In this manner, by displaying the predetermined message 422, it is possible to notify that the region of interest is located in a position that can be reached by the treatment tool.

[0094] In this example, the message is displayed in an area other than the display area of ​​the endoscopic image, but the display position of the message is not limited to this. The message may be displayed within the display area of ​​the endoscopic image.

[0095] In this example, a message consisting of characters is displayed to notify the user that the region of interest is located at a position that the treatment tool can reach, but an icon or the like may be displayed to notify the user that the region of interest is located at a position that the treatment tool can reach. The message and icon are examples of information.

[0096] (3) Third variation of the notification method It is also possible to configure the device to output a sound to notify the user that the region of interest is located at a position that can be reached by the treatment tool. In this case, a speaker is separately provided, and a predetermined sound is output from the speaker to notify the user that the region of interest is located at a position that can be reached by the treatment tool.

[0097] The notification by voice can be used in combination with the notification by display on the display 400.

[0098] [Modification of Determination of Whether or Not a Region of Interest Exists in a Position Reachable by a Treatment Tool] The reference point P is set at a position where the treatment is easy to perform, taking into consideration the position where the treatment tool protrudes, the type of treatment tool, the content of the treatment, etc. Similarly, the threshold value r is also set, taking into consideration the type of treatment tool, the content of the treatment, etc.

[0099] Furthermore, the method of determining whether or not the region of interest is located at a position that can be reached by the treatment tool is not limited to the method described in the above embodiment. For example, the distance to the region of interest may be measured, and whether or not the measured distance is equal to or less than a threshold may be determined to determine whether or not the region of interest is located at a position that can be reached by the treatment tool. The distance to the region of interest is measured from an endoscopic image, for example, using a known image measurement method.

[0100] In addition, when making a judgment based on a reference point set in an endoscopic image, the threshold value r (first threshold value) can be constant regardless of the type of treatment tool and the type of treatment, or can be set individually depending on the type of treatment tool and the type of treatment. For example, when making a constant regardless of the type of treatment tool and the type of treatment, 1 / 10 of the width of the endoscopic image can be set as the threshold value. In addition, when changing the threshold value depending on the type of treatment tool, for example, it can be set as follows. For example, when performing a treatment using forceps, 1 / 10 of the width of the endoscopic image is set as the threshold value. In addition, when performing a treatment using a local injection needle (local injection), 1 / 20 of the width of the endoscopic image is set as the threshold value. Since local injection is performed at the pinpoint of the root of the lesion, the threshold value is set to a narrower range. In addition, when performing a treatment using a snare, 1 / 2 of the width of the endoscopic image is set as the threshold value. Since the snare can be widely spread to capture the lesion, the threshold value is set to a wider range.

[0101] In addition, the configuration may be such that the necessity of the notification is determined by taking into account the operation status of the endoscope and / or the treatment tool. For example, when a treatment is performed using a snare, the necessity of the notification can be determined by taking into account information on the opening details of the snare. That is, the necessity of the notification is determined by determining whether the snare is opened to a size that allows the region of interest to be sampled. In this case, the opening details of the snare can be detected, for example, from an image. In addition, for example, when a treatment is performed using a puncture needle, if the protruding angle of the puncture needle is adjustable, the necessity of the notification can be determined by taking into account information on the protruding angle.

[0102] [Second embodiment] Here, an example in which the present disclosure is applied to an ultrasound endoscope system will be described.

[0103] [System Configuration] FIG. 11 is a block diagram showing an example of a system configuration of an ultrasound endoscope system to which the present disclosure is applied.

[0104] As shown in the figure, an ultrasonic endoscope system 1000 of this embodiment includes an ultrasonic endoscope 1010, a light source device 1100, an endoscope processor device 1200, an ultrasonic processor device 1500, an image processing device 1300, and a display 1400.

[0105] FIG. 12 is a diagram showing an example of an ultrasonic endoscope.

[0106] The ultrasonic endoscope 1010 shown in the figure is a convex type ultrasonic endoscope, and is mainly composed of an insertion section 1012, an operation section 1014, and a connection section 1016.

[0107] The insertion section 1012 is composed of, in order from the distal end side, a distal end section 1012A, a bending section 1012B that can be bent, and a flexible section 1012C that has flexibility.

[0108] FIG. 13 is a perspective view showing an example of the configuration of the tip of the insertion portion of an ultrasonic endoscope.

[0109] As shown in the figure, a treatment tool protruding portion 1020, an endoscopic observation portion 1030, and an ultrasonic probe 1040 are provided on a distal end portion 1012A.

[0110] The treatment tool protruding portion 1020 is provided with a treatment tool protruding port 1022 from which the treatment tool protrudes, a stand 1024 for adjusting the protruding direction of the treatment tool, etc. The stand 1024 swings in response to the operation of a stand lever 1056 provided in the operation portion 1014, and changes the protruding angle of the treatment tool.

[0111] The endoscope observation section 1030 includes an observation window 1032, an illumination window 1034, and a nozzle 1036. An imaging section is provided inside the observation window 1032. The imaging section includes an imaging optical system and an image sensor.

[0112] The ultrasonic probe 1040 includes therein a plurality of piezoelectric elements for transmitting and receiving ultrasonic waves, an acoustic lens, and the like.

[0113] 12, various operating members are provided on the operating section 1014. For example, an angle knob 1050 for bending the bending section 1012B, a suction button 1052 for performing a suction operation, an air / water supply button 1054 for performing an air / water supply operation, and a standing lever 1056 for operating the standing base 1024 to stand up are provided.

[0114] 12, the operation section 1014 is provided with a treatment tool insertion port 1060 for inserting a treatment tool. The treatment tool inserted from the treatment tool insertion port 1060 passes through a treatment tool channel (not shown) provided in the insertion section 1012 and protrudes from a treatment tool protruding port 1022 at the tip.

[0115] The connection unit 1016 is a portion for connecting the ultrasonic endoscope 1010 to the light source device 1100, the endoscope processor device 1200, and the ultrasonic processor device 1500. The connection unit 1016 is formed of a flexible cord. The tip of the connection unit 1016 is provided with a connector 1016A for connecting to the light source device 1100, a connector 1016B for connecting to the endoscope processor device 1200, and a connector 1016C for connecting to the ultrasonic processor device 1500.

[0116] The light source device 1100 includes a light source, and supplies light from the light source as illumination light to the ultrasonic endoscope 1010. The illumination light supplied to the ultrasonic endoscope 1010 passes through a light guide (not shown) and is emitted from an illumination window 1034 at the tip. For example, white light is used as the light from the light source.

[0117] The endoscope processor device 1200 takes in an imaging signal output from the imaging section of the ultrasonic endoscope 1010, performs predetermined signal processing, and generates an observation image (endoscopic image) by the endoscopic observation section 1030. The endoscope processor device 1200 also controls the entire system. The endoscope processor device 1200 is composed of a computer equipped with a CPU, ROM, RAM, etc. The endoscope processor device 1200 realizes functions such as generating an endoscopic image and controlling the entire system by the CPU executing a predetermined program. The ROM stores various programs executed by the CPU, and data necessary for control, processing, etc. The RAM provides the CPU with a working memory space.

[0118] The ultrasonic processor 1500 is an ultrasonic probe of the ultrasonic endoscope 1010. 1040The ultrasonic processor device 1500 is configured, for example, with a computer equipped with a CPU, ROM, RAM, etc. The ultrasonic processor device 1500 realizes functions such as generating ultrasonic images by executing a predetermined program with the CPU. The ROM stores various programs executed by the CPU, and data necessary for control, processing, etc. The RAM provides the CPU with a working memory space.

[0119] The image processing device 1300 acquires an endoscopic image output from the endoscope processor device 1200 and an ultrasonic image output from the ultrasonic processor device 1500, and performs processing to display them on the display 1400. The image processing device 1300 also performs processing to support a predetermined treatment by an operator using a treatment tool via the display 1400. The image processing device 1300 is composed of, for example, a computer equipped with a CPU, a ROM, a RAM, and the like. The image processing device 1300 functions as an image processing device by the CPU executing a predetermined program. The ROM stores various programs executed by the CPU, and data necessary for various processes and controls. The RAM provides the CPU with a working memory space.

[0120] Regarding the treatment support performed by the image processing device 1300, the treatment support based on an endoscopic image is the same as in the above-described first embodiment. Here, the treatment support based on an ultrasonic image will be described.

[0121] FIG. 14 is a block diagram of a function realized by the image processing device of this embodiment.

[0122] As shown in the figure, the image processing device 1300 has the functions of an image acquisition unit 1300A, an attention area detection unit 1300B, a determination unit 1300C, a display image generation unit 1300D, and a display control unit 1300E. These functions are realized by the CPU executing a predetermined program (image processing program).

[0123] The image acquiring unit 1300A acquires an ultrasonic image from the ultrasonic processor device 1500. Note that the ultrasonic image is a moving image. The image acquiring unit 1300A sequentially acquires images of each frame constituting the moving image.

[0124] The region of interest detection unit 1300B detects a region of interest from the image of each frame acquired by the image acquisition unit 1300A. The "region of interest" here refers to the region to be treated. In the case of tissue sampling, it is the lesion that is the target. The region of interest detection unit 1300B detects a region of interest from an endoscopic image, for example, by image recognition. The image recognition can be performed, for example, by using an image recognition model generated by machine learning (for example, deep learning, etc.). Other known methods can also be used.

[0125] The region of interest is detected by identifying its position within the image. The position is acquired, for example, as information on the pixel position of the center or center of gravity of the region of interest present in the endoscopic image.

[0126] When a region of interest is detected, the determining section 1300C determines whether or not the region of interest is present in a position reachable by the treatment tool. The determination is performed as follows.

[0127] 15 is a diagram showing an example of a display image (an image displayed on a display) when a treatment tool is projected from a treatment tool projection port, which shows an example of tissue sampling by Endoscopic Ultrasound-Fine Needle Aspiration (EUS-FNA).

[0128] As shown in the figure, a display image 1410 on a display 1400 is composed of a rectangular image having a predetermined aspect ratio. An ultrasound image 1412 is displayed in a predetermined display area 1414 set within the display image 1410. In the example shown in the figure, a fan-shaped area is set as the display area 1414 for the ultrasound image 1412.

[0129] When the treatment tool (puncture needle) 1600 is protruded from the treatment tool outlet 1022, the treatment tool 1600 is reflected in the ultrasound image 1412. Since the positional relationship between the treatment tool outlet 1022 and the ultrasound probe 1040 is fixed, the position at which the treatment tool 1600 is reflected is always constant. Furthermore, the protruding direction is also constant. In FIG. 15, a straight line L shown by a dashed line is an extension line that extends along the protruding direction of the treatment tool 1600 protruding from the treatment tool outlet 1022. When the treatment tool 1600 is protruded from the treatment tool outlet 1022, the treatment tool 1600 moves forward and backward along the straight line L in the ultrasound image 1412. The determination unit 1300C determines whether or not the distance D from this straight line L to the region of interest is equal to or less than a threshold (equal to or less than a second threshold) and determines whether or not the treatment tool 1600 It is determined whether or not the area of ​​interest exists in a position reachable by the treatment tool 1600. The distance D from the line L to the area of ​​interest is measured as the distance of a perpendicular line extended from the area of ​​interest toward the line L. If the distance D is equal to or less than the threshold, it is determined that the area of ​​interest exists in a position reachable by the treatment tool 1600. On the other hand, if the distance D exceeds the threshold, it is determined that the area of ​​interest does not exist in a position reachable by the treatment tool 1600.

[0130] The display image generating unit 1300D generates a display image to be displayed on the display 1400 from the ultrasound image acquired by the image acquiring unit 1300A. The display image generating unit 1300D generates a display image based on the determination result of the determining unit 1300C. That is, when the determining unit 1300C determines that the region of interest exists in a position reachable by the treatment tool 1600, the display image is changed to notify the operator that the region of interest exists in a position reachable by the treatment tool 1600. Specifically, the display area of ​​the ultrasound image 1412 is 1414 Change the display.

[0131] Fig. 16 and Fig. 17 are diagrams showing examples of a display image. Fig. 16 shows an example of a case where the attention area does not exist in a position reachable by the treatment tool 1600. Fig. 17 shows an example of a case where the attention area exists in a position reachable by the treatment tool 1600.

[0132] When there is no region of interest in a position reachable by the treatment tool 1600, only the ultrasound image 1412 is displayed in the display area 1414 of the ultrasound image 1412, as shown in Fig. 16. In Fig. 16, a cross (X) mark 1418 shown by a dashed line indicates the position of the region of interest. For convenience of explanation, a straight line L (an extension line of the treatment tool) extending along the protruding direction of the treatment tool 1600 and the mark 1418 indicating the position of the region of interest are displayed in Fig. 16, but these are not actually displayed in the display image 1410.

[0133] On the other hand, when the region of interest exists in a position that can be reached by the treatment tool 1600, a cross (X) mark 1420 indicating the position of the region of interest is displayed in the display area 1414 of the ultrasound image 1412, superimposed on the ultrasound image 1412, as shown in Fig. 17. Note that, for convenience of explanation, a straight line L (an extension line of the treatment tool) extending along the protruding direction of the treatment tool 1600 is displayed in Fig. 17, but this straight line L does not appear on the actual display image. 1410 It is not displayed in.

[0134] Only when the region of interest is present in a position reachable by the treatment tool 1600, the mark 1420 indicating the position of the region of interest is displayed, thereby making it possible to appropriately notify the operator of the timing of treatment.

[0135] The display control unit 1300E causes the display 400 to display the display image generated by the display image generating unit 1300D.

[0136] [Operation] Here, an example will be described in which tissue is harvested by ultrasonic endoscopic puncture aspiration. In tissue harvesting by ultrasonic endoscopic puncture aspiration, a puncture needle 1600 is used as a treatment tool. The puncture needle 1600 is inserted from a treatment tool insertion port 1060 provided in an operation unit 1014 of an ultrasonic endoscope 1010. The puncture needle 1600 inserted from the treatment tool insertion port 1060 protrudes from a treatment tool ejection port 1022 provided at the tip of an insertion unit 1012. The protruding puncture needle 1600 is used to pierce a target and harvest tissue.

[0137] FIG. 18 is a flowchart showing a processing procedure for displaying an endoscopic image on a display by an image processing device.

[0138] First, an ultrasound image is acquired from the ultrasound processor 1500 (step S11). Next, a region of interest is detected from the acquired ultrasound image (step S12). The region of interest here is a lesion that is a target for tissue sampling. Next, based on the detection result of the region of interest, it is determined whether or not the region of interest has been detected (step S13). If it is determined that the region of interest has not been detected, a normal display image is generated (step S14). Then, the generated display image is displayed on the display 1400 (step S17). Here, the normal display image is an image in which a mark indicating the position of the region of interest is not displayed in the ultrasound image (an image in which the display of the mark is turned off) (see FIG. 16). On the other hand, if it is determined that the region of interest has been detected, it is further determined whether or not the region of interest exists in a position that the puncture needle 1600 can reach (step S15). If it is determined that the region of interest does not exist in a position that the puncture needle 1600 can reach, a normal display image is generated (step S14) as in the case where the region of interest has not been detected, and the generated display image is displayed on the display 1400 (step S17). On the other hand, if it is determined that the region of interest exists in a position reachable by the puncture needle 1600, a display image displaying a mark indicating the position of the region of interest is generated (step S16). Then, the generated display image is displayed on the display 1400 (step S17). The display image displaying the mark indicating the position of the region of interest is an image in which a mark 1420 indicating the position of the region of interest is superimposed on an ultrasound image 1412 (an image with the mark display turned on), as shown in FIG. 17. Then, it is determined whether or not ultrasonic imaging by the ultrasound endoscope 1010 has ended (step S18). Then, when imaging ends, the display process ends. The end of imaging is determined, for example, by whether or not an image of the next frame has been input. If imaging has not ended, the process returns to step S11, and the above series of processes are performed.

[0139] In this way, according to the ultrasound endoscope system 1000 of the present embodiment, a mark indicating the position of the region of interest is displayed in the displayed image when the region of interest is located at a position reachable by the puncture needle 1600. This allows the surgeon to be notified of the target at the appropriate time during tissue sampling, thereby providing appropriate support to the surgeon.

[0140] [Variations] [Modification of notification] In the above embodiment, a mark indicating the position of the region of interest is displayed superimposed on the endoscopic image to notify the user that the region of interest is located in a position that can be reached by the treatment tool, but the method of notification is not limited to this. Modified examples of the method of notification will be described below.

[0141] (1) First modified example of the notification method 19 is a diagram showing a first modified example of a display image, which shows a display image in a case where the attention area is located at a position reachable by a treatment tool.

[0142] As shown in the figure, in this example, a treatment tool 1 600 When the region of interest is located at a position reachable by the ultrasound image 1412, a mark 1420 indicating the position of the region of interest and a guide line GL indicating the protruding direction of the treatment tool 1600 are displayed superimposed on the ultrasound image 1412. The guide line GL is information indicating the protruding direction of the treatment tool 1600, and when the treatment tool 1600 is protruding from the treatment tool ejection port 1022, the treatment tool 1600 moves along the guide line GL. The guide line GL coincides with an extension line of the treatment tool 1600 protruding from the treatment tool ejection port 1022.

[0143] By displaying the guide lines GL in this manner, the protruding direction of the treatment tool 1600 can be clarified, and better support can be achieved.

[0144] In this example, the mark 1420 and the guide line GL are displayed when the attention area is located at a position reachable by the treatment tool, but it is also possible to configure the display to display only the guide line GL.

[0145] Also, the mark 1420 indicating the position of the attention area may be always displayed, and when the attention area is located at a position reachable by the treatment tool, the guide line GL may be displayed. In this case, the display form of the mark 1420 may be changed depending on whether the attention area is located at a position reachable by the treatment tool or not. For example, when the attention area is located at a position reachable by the treatment tool, the color, shape, brightness, line type, etc. of the mark 1420 may be switched to change the degree of emphasis. In other words, when the attention area is located at a position reachable by the treatment tool, the degree of emphasis of the mark 1420 is increased.

[0146] Also, the guide line GL may be configured to be always displayed, and when the attention area is located at a position reachable by the treatment tool, the mark 1420 may be displayed. In this case, too, the display form of the guide line GL may be changed depending on whether the attention area is located at a position reachable by the treatment tool or not. For example, when the attention area is located at a position reachable by the treatment tool, the color, brightness, line type, etc. of the guide line GL may be switched to change the degree of emphasis. In other words, when the attention area is located at a position reachable by the treatment tool, the degree of emphasis of the guide line GL is increased.

[0147] Furthermore, the mark 1420 and the guide line GL may be constantly displayed, and when the attention area is positioned at a position reachable by the treatment tool, the display form of the mark 1420 and the guide line GL may be changed.

[0148] Also, the guide lines GL may be configured to be always displayed regardless of whether or not a region of interest has been detected.

[0149] The shape of the mark 1420 indicating the position of the region of interest is not limited to a cross (X), and various other shapes can be used.

[0150] (2) Second modified example of the notification method In the above embodiment, the notification is made by changing the display of the display area of ​​the ultrasound image. However, the notification may be made by changing the display of an area other than the display area of ​​the ultrasound image.

[0151] FIG. 20 is a diagram showing a second modified example of a display image.

[0152] As shown in the figure, in this example, when the region of interest is located at a position that can be reached by the treatment tool, the display region of the ultrasound image 1412 is 1414 A message 1422 is displayed in the outer area. In this example, the message 1422 displays the text "Push!".

[0153] In this way, by displaying the predetermined message 1422, it is possible to notify that the region of interest is located in a position that can be reached by the treatment tool.

[0154] In this example, when the attention area is located at a position reachable by the treatment tool, only the message 1422 is displayed, but it is also possible to simultaneously display a mark indicating the position of the attention area and / or a guide line indicating the protrusion direction. Also, the mark and / or guide line can be configured to be displayed at all times. When the mark and / or guide line are always displayed, it is also possible to switch the display form depending on whether the attention area is located at a position reachable by the treatment tool or not.

[0155] In addition, in this example, the message is displayed in an area other than the display area of ​​the ultrasound image, but the display position of the message is not limited to this. The message may be displayed within the display area of ​​the ultrasound image.

[0156] Also, instead of a message, an icon or the like may be displayed.

[0157] (3) Third variation of the notification method It is also possible to configure the device to output a sound to notify the user when the region of interest is located at a position that can be reached by the treatment tool. In this case, a speaker is separately provided, and a predetermined sound is output from the speaker to notify the user that the region of interest is located at a position that can be reached by the treatment tool. The sound notification can also be used in combination with the notification by display on the display 400.

[0158] [About the threshold (second threshold)] The threshold value (second threshold value) used to determine whether or not the area of ​​interest is located in a position reachable by the treatment tool can be constant regardless of the type of treatment tool and the type of treatment, etc., or can be set individually depending on the type of treatment tool and the type of treatment, etc.

[0159] [Third embodiment] In the first and second embodiments, when the region of interest is located at a position that can be reached by the treatment tool, a mark or the like indicating the position of the region of interest is displayed to notify the operator. In the present embodiment, it is further configured to determine whether or not an obstacle exists between the region of interest and the treatment tool, and to notify the operator when no obstacle exists. Hereinafter, a case where this process is performed in an ultrasonic endoscope system will be described.

[0160] FIG. 21 is a block diagram of a function realized by the image processing device of this embodiment.

[0161] As shown in the figure, an image processing device 1300 of this embodiment further includes the function of an obstacle detection unit 1300F in addition to the components of the image processing device of the second embodiment.

[0162] The obstacle detection unit 1300F detects an obstacle from the ultrasound image acquired by the image acquisition unit 1300A. The "obstacle" here is an object that becomes an obstacle when the treatment tool 1600 is protruded from the treatment tool ejection port 1022. For example, a large blood vessel corresponds to this. The obstacle detection unit 1300F detects an obstacle from the ultrasound image, for example, by image recognition. The image recognition can be performed, for example, by using an image recognition model generated by machine learning (for example, deep learning, etc.). In addition, it can be performed by using other known methods.

[0163] The obstacle is detected by identifying its position in the image. The position is acquired, for example, as information on the pixel position of the center or center of gravity of the obstacle present in the ultrasound image.

[0164] The determination unit 1300C determines whether or not a notification is required based on the detection results of the attention area detection unit 1300B and the obstacle detection unit 1300F. Specifically, it is determined that a notification is required when there is no obstacle between the attention area and the tip of the treatment tool and the attention area is located at a position that the treatment tool can reach, and otherwise it is determined that a notification is not required. Therefore, even if the attention area is located at a position that the treatment tool can reach, if there is an obstacle between the tip of the treatment tool and the attention area, it is determined that a notification is not required.

[0165] 22 and 23 are diagrams showing an example of a display image. In FIG. 22, a cross (X) mark 1418 shown by a dashed line indicates the position of the region of interest. Also, in FIG. 22 and FIG. 23, reference numeral 1424 indicates an obstacle (here, an artery). FIG. 22 shows an example of a case where an obstacle exists between the region of interest and the treatment tool 1600. Also, FIG. 23 shows an example of a case where no obstacle exists between the region of interest and the treatment tool 1600 and the treatment tool 1600 13 shows an example in which the region of interest is located at a position reachable by the

[0166] As shown in FIG. 22, an obstacle is present between the attention area and the treatment tool 1600. 1424 If present, treatment tool 1600Even if the region of interest is located in a position that can be reached by the ultrasound image, 1412 In the display area 1414, only the ultrasound image 1412 is displayed. Note that, for convenience of explanation, in Fig. 22, a straight line extending along the protruding direction of the treatment tool 1600 and a mark 1418 indicating the region of interest are displayed, but the straight line and the mark 1418 are not displayed in the actual display image.

[0167] On the other hand, as shown in FIG. 23, if there is an obstacle between the attention area and the treatment tool 1600, 1424 does not exist and 1600 If the region of interest is present in a position reachable by the ultrasound image 1412, a mark 1420 indicating the position of the guide line GL and the region of interest is displayed superimposed on the ultrasound image 1412.

[0168] FIG. 24 is a flowchart showing a processing procedure for displaying an endoscopic image on a display by the image processing device.

[0169] First, an ultrasound image is acquired from the ultrasound processor device 1500 (step S21). Next, a region of interest is detected from the acquired ultrasound image (step S22). Also, an obstacle is detected from the acquired ultrasound image (step S23). The obstacle is, for example, an artery. Next, based on the detection result of the region of interest, it is determined whether or not the region of interest has been detected (step S24). If it is determined that the region of interest has not been detected, a normal display image is generated (step S25). Then, the generated display image is displayed on the display 1400 (step S29). On the other hand, if it is determined that the region of interest has been detected, it is further determined whether or not the region of interest exists in a position that the puncture needle 1600 can reach (step S26). If it is determined that the region of interest does not exist in a position that the puncture needle 1600 can reach, a normal display image is generated (step S25) as in the case where the region of interest has not been detected, and the generated display image is displayed on the display 1400 (step S29). On the other hand, if it is determined that the region of interest exists in a position that the puncture needle 1600 can reach, it is determined whether or not an obstacle exists (step S27). That is, it is determined whether or not an obstacle exists between the tip of the puncture needle 1600 and the region of interest. If it is determined that an obstacle exists, a normal display image is generated (step S25), similar to the case where the region of interest has not been detected and the case where it is determined that the region of interest does not exist in a position reachable by the puncture needle 1600, and the generated display image is displayed on the display 1400 (step S29). On the other hand, if it is determined that an obstacle does not exist, a display image displaying a mark indicating the position of the region of interest is generated (step S28). Then, the generated display image is displayed on the display 1400 (step S29). Thereafter, it is determined whether or not ultrasonic imaging by the ultrasonic endoscope 1010 has ended (step S30). Then, when imaging ends, the display process ends.

[0170] In this way, according to the ultrasound endoscope system 1000 of the present embodiment, a notification is given only when treatment is truly possible, thereby providing more appropriate support to the surgeon.

[0171] In this embodiment, the guide line GL and the mark 1420 are displayed when notifying, but the form of notification is not limited to this. For example, it is also possible to display either the guide line GL or the mark 1420. It is also possible to always display the guide line GL and display the mark 1420 to notify. Conversely, it is also possible to always display the mark 1420 indicating the position of the attention area and display the guide line GL to notify. In addition, it is possible to notify by combining notification by voice, displaying information in an area other than the display area, etc.

[0172] In addition, in this embodiment, the processing using an ultrasonic endoscope has been described, but a configuration is also possible in which the presence or absence of an obstacle is determined in a similar manner in a normal endoscope, and the notification processing is performed only when no obstacle is present.

[0173] [Fourth embodiment] In this embodiment, it is determined whether the treatment tool is in a standby state from images (endoscopic images and ultrasound images), and notification processing is performed only when the treatment tool is in a standby state. A standby state refers to a state in which a treatment operation is possible. Below, an example of implementing this processing in an ultrasound endoscope system will be described.

[0174] FIG. 25 is a block diagram of a function realized by the image processing device of this embodiment.

[0175] As shown in the figure, an image processing device 1300 of this embodiment further includes a function of a treatment tool detection unit 1300G in addition to the components of the image processing device of the third embodiment.

[0176] The treatment tool detection unit 1300G detects a treatment tool (for example, a puncture needle) from the ultrasound image acquired by the image acquisition unit 1300A. The treatment tool detection unit 1300G detects the treatment tool from within the ultrasound image, for example, by image recognition. The image recognition can be performed, for example, by using an image recognition model generated by machine learning (for example, deep learning, etc.). Other known methods can also be used.

[0177] The determination section 1300C determines whether or not a notification process is required based on the detection result of the treatment tool detection section 1300G. Specifically, when the treatment tool 1600 is detected, it determines that a notification process is required.

[0178] FIG. 26 is a flowchart showing a processing procedure for displaying an endoscopic image on a display by an image processing device.

[0179] First, an ultrasonic image is acquired from the ultrasonic processor device 1500 (step S41). Then, a treatment tool is detected from the acquired ultrasonic image (step S42). Then, based on the detection result of the treatment tool, it is determined whether or not the treatment tool is detected (step S42). That is, it is determined whether or not the treatment tool is reflected in the image. If it is determined that the treatment tool is not detected, a normal display image is generated (step S44). Then, the generated display image is displayed on the display 1400 (step S51). On the other hand, if it is determined that the treatment tool is detected, a region of interest is detected from the acquired ultrasonic image (step S45). Also, an obstacle (for example, an artery) is detected from the acquired ultrasonic image (step S46). Then, based on the detection result of the region of interest, it is determined whether or not the region of interest is detected (step S47). If it is determined that the region of interest is not detected, a normal display image is generated (step S44). Then, the generated display image is displayed on the display 1400 (step S51). On the other hand, if it is determined that the attention area is detected, it is further determined whether or not the attention area exists in a position reachable by the puncture needle 1600 (step S48). If it is determined that the attention area does not exist in a position reachable by the puncture needle 1600, a normal display image is generated (step S44) in the same manner as when the attention area is not detected, and the generated display image is displayed on the display 1400 (step S51). On the other hand, if it is determined that the attention area exists in a position reachable by the puncture needle 1600, it is determined whether or not there is an obstacle (step S49). If it is determined that there is an obstacle, a normal display image is generated (step S44) in the same manner as when the attention area is not detected and when it is determined that there is no attention area in a position reachable by the puncture needle 1600, and the generated display image is displayed on the display 1400 (step S51). On the other hand, if it is determined that there is no obstacle, a display image is generated in which a mark indicating the position of the attention area is displayed (step S50). Then, the generated display image is displayed on the display 1400 (step S51). Thereafter, it is determined whether or not ultrasonic imaging by the ultrasonic endoscope 1010 has been completed (step S52). Then, when imaging has been completed, the display process is terminated.

[0180] In this way, according to the ultrasound endoscope system 1000 of the present embodiment, notification processing is performed only in the standby state, thereby making it possible to more appropriately support the surgeon.

[0181] In this embodiment, an example has been described in which the present invention is implemented in an ultrasound endoscopic system, but a typical endoscopic system can also be configured to similarly detect the standby state of a treatment tool and perform notification processing only when the treatment tool is in the standby state.

[0182] [Other embodiments] [Applicable measures] In the above embodiment, the case of tissue sampling has been described as an example, but the application of the present disclosure is not limited thereto. It can be applied to treatments in which various instruments are projected from the tip of an endoscope (including an ultrasonic endoscope). For example, in addition to the tissue sampling of the above embodiment, it can also be applied to treatments such as the insertion of a guidewire in biliary drainage under an ultrasonic endoscope.

[0183] [Area of ​​interest] The region of interest is not limited to a lesion, and may be a specific organ. For example, in the case of a normal endoscope, examples of the lesion that is the region of interest include colon polyps, stomach cancer, esophageal cancer, Barrett's esophagus, and ulcerative colitis. In the case of an ultrasound endoscope, examples of the lesion that is the region of interest include pancreatic cancer. In the case of an ultrasound endoscope, examples of the organ that is the region of interest include Ultrasound endoscope An example is the common bile duct in inferior biliary drainage.

[0184] [Hardware configuration of image processing device] The functions of the image processing device can be realized by various processors. The various processors include a CPU, which is a general-purpose processor that executes a program and functions as various processing units, an FPGA (Field Programmable Gate Array), and the like. m Programmable logic devices (PLCs), such as programmable gate arrays (PGAs), are processors whose circuit configuration can be changed after fabrication. mThese include dedicated electrical circuits, such as processors with circuit configurations designed specifically to execute specific processes, such as Programmable Logic Devices (PLDs) and Application Specific Integrated Circuits (ASICs).

[0185] A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Also, multiple processing units may be composed of one processor. As an example of multiple processing units being composed of one processor, first, as represented by a computer such as a client or a server, one processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units in one IC (Integrated Circuit) chip. In this way, the various processing units are composed of one or more of the above various processors as a hardware structure.

[0186] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0187] The functions of the image processing device can also be incorporated in a processor device constituting an endoscope system, or an ultrasonic processor device constituting an ultrasonic endoscope system.

[0188] [About lighting] The illumination light is selected from various wavelength bands such as white light, light in one or more specific wavelength bands, or a combination of these, depending on the purpose of observation. A "specific wavelength band" is a band narrower than the wavelength band of white light. Specific examples of specific wavelength bands are shown below.

[0189] A first example of the specific wavelength band is, for example, a blue band or a green band in the visible range. The first example wavelength band includes a wavelength band of 390 nm to 450 nm or a wavelength band of 530 nm to 550 nm, and the first example light has a peak wavelength within the wavelength band of 390 nm to 450 nm or the wavelength band of 530 nm to 550 nm.

[0190] A second example of the specific wavelength band is, for example, the red band of the visible range. The second example wavelength band includes a wavelength band of 585 nm to 615 nm or a wavelength band of 610 nm to 730 nm, and the light of the second example has a peak wavelength within the wavelength band of 585 nm to 615 nm or within the wavelength band of 610 nm to 730 nm.

[0191] A third example of the specific wavelength band includes a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and the light of the third example has a peak wavelength in the wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. Wavelength band , a wavelength band of 440±10 nm, a wavelength band of 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the third example light has a peak wavelength within the above wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm or less.

[0192] A fourth example of the specific wavelength band is a wavelength band of excitation light that is used for observing fluorescence emitted by fluorescent substances in living organisms (fluorescence observation) and excites the fluorescent substances, for example, from 390 nm to 470 nm.

[0193] A fifth example of the specific wavelength band is an infrared light wavelength band, which includes a wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm, and the fifth example light has a peak wavelength within the wavelength band of 790 nm to 820 nm or within the wavelength band of 905 nm to 970 nm.

[0194] [About switching the lighting] The type of light source may be a laser light source, a xenon light source, or an LED (Light-Emitting Diode) light source, or an appropriate combination of these. The type of light source, wavelength, presence or absence of a filter, etc. are preferably determined according to the type of subject, the purpose of observation, etc., and during observation, it is preferable to combine and / or switch the wavelength of illumination light according to the type of subject, the purpose of observation, etc. When switching the wavelength, for example, the wavelength of the irradiated light may be switched by rotating a disk-shaped filter (rotary color filter) that is disposed in front of the light source and has a filter that transmits or blocks light of a specific wavelength.

[0195] [About the imaging unit] The image sensor provided in the imaging section of the endoscope is not limited to a color image sensor in which a color filter is provided for each pixel, and may be a monochrome image sensor. When a monochrome image sensor is used, the wavelength of the illumination light can be switched sequentially to capture images in a surface sequential (color sequential) manner. For example, the wavelength of the emitted illumination light may be switched sequentially among purple, blue, green, and red, or white light may be irradiated and the wavelength of the emitted illumination light may be switched using a rotary color filter (red, green, blue, etc.). Also, one or more narrowband lights may be irradiated and the wavelength of the emitted illumination light may be switched using a rotary color filter. The narrowband light may be infrared light having two or more different wavelengths.

[0196] [Example of special light image generation] The processor device may generate an image having information of a specific wavelength band (so-called special light image) based on an image captured using white light (so-called normal light image). The processor device can acquire a signal of a specific wavelength band by performing a calculation based on color information of red (Red, R), green (Green, G) and blue (Blue, B) or cyan (Cyan, C), magenta (Magenta, M) and yellow (Yellow, Y) contained in the normal light image.

[0197] [About the program that enables a computer to function as an image processing device] It is possible to record a program that causes a computer to realize the functions of the image processing device described in the above embodiment on a computer-readable medium that is a non-transitory information storage medium such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible information storage medium, and provide the program through this information storage medium. Also, instead of providing the program by storing it on such a tangible non-transitory information storage medium, it is also possible to provide a program signal as a download service using an electric communication line such as the Internet.

[0198] It is also possible to provide some or all of the functions of the image processing apparatus described in the above embodiment as an application server, and to provide a service that provides processing functions via a telecommunications line.

[0199] [Combinations of embodiments and modified examples] The components described in the above embodiment and the components described in the modified examples may be used in appropriate combination, and some of the components may be replaced with each other. [Explanation of symbols]

[0200] 1 Endoscope system 10 Endoscopy 12 Endoscope insertion section 12A Tip of the insertion part 12B Curved part of insertion section 12C Soft part of insertion part 14 Endoscope control section 16 Endoscope connection part 16A Connection Connector 16B Connection Connector 20 Observation window 22 Lighting window 24 Nozzles 26 forceps mouth 3 0 Angle Knob 32 Air / water supply button 34 Suction button 36 Shutter Button 38 Forceps insertion port 40 Forceps Channel 100 Light source device 200 Processor Unit 300 Image processing device 300A Image acquisition unit 300B Area of ​​interest detection unit 300C judgment section 300D display image generation section 300E Display control unit 400 Display 410 Display images 410A Display image 410B Display image 412 Endoscopic Images 412A Endoscopic Image 412B Endoscopic images 414 Display area of ​​endoscopic image 416 Circle showing the range that the treatment tool can reach 418 Mark indicating the position of the area of ​​interest 420 Marking the location of the area of ​​interest 420A Mark indicating the location of the area of ​​interest 420B Mark indicating the location of the area of ​​interest 422 Message 500 Treatment tools (biopsy forceps, etc.) 510 Insertion part of treatment tool 512 Operating part of treatment tool 514 Tip claw of treatment tool 512A Control Handle 512B Control slider 1000 Ultrasound Endoscope System 1010 Ultrasound endoscope 1012 Insertion section of ultrasonic endoscope 1012A Tip of the insertion part 1012B Curved part of insertion section 1012C Insertion section soft part 1014 Operation unit of ultrasonic endoscope 1016 Ultrasound endoscope connection part 1016A Connection Connector 1016B Connector 1016C Connection Connector 1020 Treatment tool protrusion 1022 Treatment tool outlet 1024 Standing platform 1030 Endoscopic observation section 1032 Observation window 1034 Lighting window 1036 Nozzle 1040 Ultrasonic probe 1050 Angle knob 1052 Suction button 1054 Air / water supply button 1056 Standing Lever 1060 Treatment tool insertion port 1100 Light source device 1200 Endoscope processor 1300 Image Processing Device 1300A Image Acquisition Unit 1300B Area of ​​interest detection section 1300C Judgment section 1300D Display image generation section 1300E Display control unit 1300F Obstacle detection unit 1300G Treatment tool detector 1400 Display 1410 Display images 1412 Ultrasound Images 1412B Ultrasound Imaging 1414 Display area of ​​ultrasound image 1418 Mark indicating the location of the area of ​​interest 1420 Mark indicating the location of the area of ​​interest 1422 Messages 1424 Obstructions (arteries, etc.) 1500 Ultrasonic processor 1600 Treatment tools (puncture needles, etc.) GL Guide Line P reference point S1 to S8: Procedure for processing the display of an endoscopic image on a display by an image processing device S11 to S18: Procedure for processing the display of an endoscopic image on a display by an image processing device S21 to S30: Procedure for processing the display of an endoscopic image on a display by an image processing device S41 to S52: Procedure for processing the display of an endoscopic image on a display by an image processing device

Claims

1. An image processing device that processes an image captured by an endoscope capable of performing treatment using an instrument protruding from a tip thereof, A processor is provided. The processor, A process of acquiring an image captured by the endoscope; A process of displaying the acquired image on a display; A process of detecting an area of ​​interest to be treated from the acquired image; A process of determining whether the region of interest is present in a position that can be treated by the instrument protruding from the tip of the endoscope; a process of notifying that the region of interest is present in a position treatable by the instrument when it is determined that the region of interest is present in a position treatable by the instrument protruded from the tip of the endoscope; To carry out Image processing device.

2. the processor determines whether or not a distance from a reference point set on the image to the region of interest is equal to or less than a first threshold value, and determines whether or not the region of interest is present in a position that can be treated by the instrument protruded from the tip of the endoscope; The image processing device according to claim 1 .

3. the reference point being the center of the image; The image processing device according to claim 2 .

4. the processor determines whether or not a distance from an extension line of the instrument protruding from the tip of the endoscope to the region of interest is equal to or less than a second threshold value, and determines whether or not the region of interest is present in a position that can be treated by the instrument protruding from the tip of the endoscope. The image processing device according to claim 1 .

5. The processor further performs processing to determine whether an obstacle exists between the region of interest and the instrument from the acquired image; When the processor determines that the obstacle does not exist and that the region of interest exists in a position that can be treated with the instrument protruded from the tip of the endoscope, the processor notifies the user that the region of interest exists in a position that can be treated with the instrument. The image processing device according to claim 1 .

6. The processor further performs processing to detect the instrument from the acquired image. the processor detects the instrument from the image, and when it is determined that the region of interest is present at a position that can be treated with the instrument protruding from the tip of the endoscope, performs a process of notifying that the region of interest is present at a position that can be treated with the instrument. The image processing device according to claim 1 .

7. The processor changes the display image displayed on the display to indicate that the region of interest is present in a position treatable by the instrument. The image processing device according to claim 1 .

8. The processor changes the display of a display area of ​​the image set in the display image to notify that the area of ​​interest is present in a position treatable by the instrument. The image processing device according to claim 7.

9. The processor displays a graphic indicating the region of interest superimposed on the image to be displayed in the display area, thereby notifying the user that the region of interest is present in a position treatable by the instrument. The image processing device according to claim 8.

10. The processor further performs a process of displaying a figure indicating the attention area on the image to be displayed in a display area of ​​the image set in the display image when the attention area is detected, The processor changes the display of the graphic to indicate that the region of interest is in a position treatable by the instrument. The image processing device according to claim 7.

11. the processor modifies the display of the shape by changing at least one of a color, a shape, a brightness, and a line style of the shape; The image processing device according to claim 10.

12. The processor further performs a process of displaying information indicating a protruding direction of the tool superimposed on the image to be displayed in a display area of ​​the image set in the display image, The processor changes a display of information indicating a protruding direction of the instrument to notify that the region of interest is present in a position treatable by the instrument. The image processing device according to claim 7.

13. The processor displays a straight line along the protruding direction of the tool as information indicating the protruding direction of the tool. The image processing device according to claim 12.

14. the processor modifies the display of the line by changing at least one of a color, brightness, and line style of the line; The image processing device according to claim 13.

15. The processor changes the display of a region other than the display region of the image set in the display image to notify that the region of interest is present in a position treatable by the instrument. The image processing device according to claim 7.

16. The processor displays information in an area other than a display area of ​​the image to notify the user that the area of ​​interest is present in a position treatable by the instrument. The image processing device according to claim 15.

17. The processor displays a message or a graphic as the information. The image processing device according to claim 16.

18. The processor outputs a sound to notify the user that the region of interest is in a position treatable by the instrument. The image processing device according to any one of claims 1 to 17.

19. The region of interest is a lesion.

19. An image processing device according to any one of claims 1 to 18.

20. The region of interest is an organ.

20. An image processing device according to any one of claims 1 to 19.

21. A method for operating an endoscope system including an endoscope capable of performing treatment using an instrument protruding from a distal end thereof, and an image processing device for processing images captured by the endoscope, comprising the steps of: a step of the image processing device acquiring an image captured by the endoscope; a step of causing the image processing device to display the image acquired from the endoscope on a display; detecting a region of interest to be treated from the image acquired by the endoscope by the image processing device; a step of determining whether or not the region of interest is present in a position that can be treated by the instrument protruding from the tip of the endoscope by the image processing device; a step of notifying that the region of interest is present in a position treatable by the instrument when the image processing device determines that the region of interest is present in a position treatable by the instrument protruded from the tip of the endoscope; Including, A method for operating an endoscope system.

22. A function for acquiring images taken by an endoscope capable of performing treatment using a tool protruding from the tip; A function of displaying the acquired image on a display; A function of detecting an area of ​​interest to be treated from the acquired image; a function of determining whether the region of interest is present in a position that can be treated by the instrument protruding from the tip of the endoscope; and a function of notifying that the region of interest is present in a position treatable by the instrument when it is determined that the region of interest is present in the position treatable by the instrument protruded from the tip of the endoscope; To realize this on a computer, Image processing program.

Images