Method for generating auxiliary information, device for generating auxiliary information, and program

Abstract

This method for generating auxiliary information: sequentially acquires a plurality of images obtained by an imaging unit of an endoscope at a plurality of positions in the digestive tract of a subject; detects a characteristic area(s) in the digestive tract on the basis of the plurality of images; calculates the number of the characteristic areas in each of a plurality of sections in the digestive tract; generates auxiliary information according to the number of the characteristic areas; and controls the mode of the auxiliary information according to the number of the characteristic areas when the auxiliary information is displayed on a display unit.

Description

Auxiliary information generating method, auxiliary information generating device, and program

[0001] The present invention relates to an auxiliary information generating method, an auxiliary information generating device, and a program.

[0002] In recent years, methods for identifying characteristic features in an image using image recognition technology and displaying the image and the result of the identification have been used in a variety of situations. However, as the number or types of characteristic features detected increase, the display screen becomes filled with information related to the result of the identification, reducing the visibility of the image.

[0003] Patent Literature 1 discloses a technique for superimposing a graphics object on an image of real space, in which multiple pieces of augmented reality information that are close to each other are displayed as a single aggregate graphics object.

[0004] Japanese Patent Application Publication No. 2010-238098

[0005] The technology disclosed in Patent Document 1 displays augmented reality information related to a currently displayed image, but does not emphasize augmented reality information related to images displayed in the past.

[0006] The present invention aims to provide an auxiliary information generation method, an auxiliary information generation device, and a program that can avoid the cumbersome notification of auxiliary information obtained from multiple images, including images acquired in the past.

[0007] According to a first aspect of the present invention, an auxiliary information generating method includes: sequentially acquiring a plurality of images obtained by an imaging unit of an endoscope at a plurality of positions in the gastrointestinal tract of a subject; detecting characteristic regions in the gastrointestinal tract based on the plurality of images; calculating the number of characteristic regions in each of a plurality of sections in the gastrointestinal tract; generating auxiliary information according to the number of characteristic regions, and controlling the appearance of the auxiliary information according to the number of characteristic regions when the auxiliary information is displayed on a display unit.

[0008] According to a second aspect of the present invention, in the first aspect, the auxiliary information generation method may determine whether the imaging unit is performing imaging in the depth direction of the digestive tract, and may output the auxiliary information to the display unit when it is determined that the imaging unit is performing imaging in the depth direction.

[0009] According to a third aspect of the present invention, in the second aspect, the auxiliary information generation method may generate the auxiliary information for each of a plurality of image areas obtained by dividing the image obtained by the imaging unit in a direction corresponding to the circumferential direction of the digestive tract.

[0010] According to a fourth aspect of the present invention, in the third aspect, the auxiliary information generation method may control the size of an icon indicating the auxiliary information for each image area of ​​the plurality of image areas in accordance with the number of feature areas.

[0011] According to a fifth aspect of the present invention, in the second aspect, the image obtained by the imaging unit may be divided in a direction corresponding to the radial direction of the digestive tract, and the auxiliary information for each of the multiple image areas divided in a direction corresponding to the circumferential direction of the digestive tract may be generated.

[0012] According to a sixth aspect of the present invention, in the first aspect, the auxiliary information generation method may generate the auxiliary information to be displayed in a first aspect when the number of feature regions is equal to or greater than a predetermined number, and may generate the auxiliary information to be displayed in a second aspect different from the first aspect when the number of feature regions is less than the predetermined number.

[0013] According to a seventh aspect of the present invention, in the sixth aspect, the auxiliary information displayed in the second aspect may indicate the position of the characteristic region.

[0014] According to an eighth aspect of the present invention, an auxiliary information generation device includes an image acquisition unit, a region detection unit, a region calculation unit, and an auxiliary information generation unit. The image acquisition unit sequentially acquires a plurality of images obtained by an imaging unit of an endoscope at a plurality of positions in the gastrointestinal tract of a subject. The region detection unit detects characteristic regions in the gastrointestinal tract based on the plurality of images. The region calculation unit generates auxiliary information according to the number of characteristic regions, and controls the appearance of the auxiliary information when the auxiliary information is displayed on a display unit according to the number of characteristic regions.

[0015] According to a ninth aspect of the present invention, there is provided a program for causing a computer to execute the following: sequentially acquiring a plurality of images obtained by an imaging unit of an endoscope at a plurality of positions in the gastrointestinal tract of a subject; detecting characteristic regions in the gastrointestinal tract based on the plurality of images; calculating the number of characteristic regions in each of a plurality of sections in the gastrointestinal tract; generating auxiliary information according to the number of characteristic regions, and controlling the appearance of the auxiliary information according to the number of characteristic regions when the auxiliary information is displayed on a display unit.

[0016] According to the above aspects, the auxiliary information generating method, auxiliary information generating device, and program can avoid cumbersome notification of auxiliary information obtained from multiple images including images acquired in the past.

[0017] 1 is a block diagram showing an example of the configuration of an endoscopic system according to a first embodiment of the present invention. FIG. 2 is a diagram showing an example of a plurality of sections according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of an imaging direction according to the first embodiment of the present invention. FIG. 4 is a diagram showing an example of an imaging direction according to the first embodiment of the present invention. FIG. 5 is a flowchart showing an example of a processing procedure performed by an auxiliary information generating device according to the first embodiment of the present invention. FIG. 6 is a diagram showing an example of a method for calculating the number of unobserved areas according to the first embodiment of the present invention. FIG. 7 is a flowchart showing an example of a processing procedure performed by an auxiliary information generating device according to the first embodiment of the present invention. FIG. 8 is a diagram showing an example of an unobserved area detected in an image according to the first embodiment of the present invention. FIG. 9 is a diagram showing an example of an image and auxiliary information displayed on a display unit according to the first embodiment of the present invention. FIG. 10 is a flowchart showing an example of a processing procedure performed by an auxiliary information generating device according to the first embodiment of the present invention. FIG. 11 is a diagram showing an example of an unobserved area detected in an image according to the first embodiment of the present invention. FIG. 12 is a diagram showing an example of an image and auxiliary information displayed on a display unit according to the first embodiment of the present invention. FIG. 10 is a diagram showing a third example of a display screen of the display unit in the second embodiment of the present invention. FIG. 11 is a diagram showing a fourth example of a display screen of the display unit in the second embodiment of the present invention. FIG. 12 is a diagram showing a fifth example of a display screen of the display unit in the second embodiment of the present invention. FIG. 13 is a diagram showing a sixth example of a display screen of the display unit in the second embodiment of the present invention. FIG. 14 is a flowchart showing an example of a processing procedure executed by the auxiliary information generating device according to the second embodiment of the present invention. FIG. 15 is a flowchart showing an example of a processing procedure executed by the auxiliary information generating device according to the second embodiment of the present invention.

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. An example of an endoscope system including an auxiliary information generating device will be described below. In the following description, "plurality of XX" means two or more XXs.

[0019] 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in FIG. 1 includes an endoscope 10 and an auxiliary information generating device 20.

[0020] The endoscope 10 is inserted into the digestive tract of a subject. For example, the digestive tract is the large intestine. The endoscope 10 has an imaging unit 11. The imaging unit 11 is disposed at the tip of the endoscope 10. The imaging unit 11 is an image sensor and generates multiple images at multiple positions in the digestive tract of the subject. The imaging unit 11 outputs the multiple images to the auxiliary information generation device 20.

[0021] The auxiliary information generating device 20 includes an image acquiring unit 21 , a storage unit 22 , a region detecting unit 23 , a region calculating unit 24 , a shooting direction determining unit 25 , an auxiliary information generating unit 26 , and a display unit 27 .

[0022] The image acquisition unit 21 sequentially acquires the plurality of images output from the imaging unit 11. The image acquisition unit 21 outputs the acquired plurality of images to the storage unit 22.

[0023] The storage unit 22 is a memory that stores a plurality of images output from the image acquisition unit 21. The plurality of images stored in the storage unit 22 are sequentially output to the area detection unit 23, the imaging direction determination unit 25, and the display unit 27.

[0024] The region detection unit 23 detects characteristic regions in each of multiple sections in the digestive tract based on multiple images output from the storage unit 22. In the following example, an example will be described in which the region detection unit 23 detects unobserved regions as characteristic regions. There are cases in which a region in the digestive tract is not shown in the image, or the visibility of the region in the image is poor. Alternatively, there are cases in which a region in the digestive tract is shown in the image, but the doctor does not perform detailed observation or treatment on that region. The region detection unit 23 detects these regions as unobserved regions. The region detection unit 23 outputs region position information indicating the position of the unobserved region in the image to the region calculation unit 24.

[0025] FIG. 2 shows an example of multiple sections. The digestive tract is divided into sections SE1 to SE5. For example, each section is set based on the time at which images of the area within the section were captured, and includes areas captured within a predetermined length of time. Each section may have a predetermined length and include an area within that length. Each section may also be set for a specific part of the digestive tract.

[0026] The region calculation unit 24 calculates the number of unobserved regions in each of the multiple sections in the digestive tract, and outputs region number information indicating the number of unobserved regions in each section to the auxiliary information generation unit 26.

[0027] The imaging direction determination unit 25 determines the imaging direction of the imaging unit 11 based on the multiple images output from the storage unit 22. Figures 3A and 3B show examples of imaging directions.

[0028] In Fig. 3A, the imaging unit 11 captures images in a depth direction D1 (longitudinal direction) along the inner wall IW of the digestive tract. In Fig. 3B, the imaging unit 11 captures images in a direction D2 toward the inner wall IW. The doctor bends the endoscope 10 to observe or treat a specific region. At this time, the imaging unit 11 captures images in the direction D2. The imaging direction determination unit 25 may determine the imaging direction based on the state of a bending mechanism for bending the endoscope 10. The imaging direction determination unit 25 outputs imaging direction information indicating the imaging direction to the region calculation unit 24 and the auxiliary information generation unit 26.

[0029] When the multiple images change and a continuously dark image feature of a lumen hole is detected near the center of the image, the imaging direction determination unit 25 may determine that the imaging direction is direction D1. Alternatively, when the feature gradually disappears and an image feature specific to a uniform lumen wall surface is detected, the imaging direction determination unit 25 may determine that the imaging direction has switched to direction D2. Alternatively, the imaging direction determination unit 25 may determine direction D1 and direction D2 by using information detected by a special sensor or a sensor in the operation unit.

[0030] The imaging direction determination unit 25 also determines the traveling direction of the endoscope 10 based on the multiple images output from the storage unit 22. When the doctor is moving the endoscope 10 toward the examination target in the digestive tract, the traveling direction of the endoscope 10 is the depth direction. When the doctor is moving the endoscope 10 in the direction to be removed from the digestive tract, the traveling direction of the endoscope 10 is the backward direction. The imaging direction determination unit 25 outputs traveling direction information indicating the traveling direction of the endoscope 10 to the area detection unit 23.

[0031] The auxiliary information generating unit 26 generates auxiliary information according to the area number information output from the area calculating unit 24, and outputs the auxiliary information to the display unit 27. The auxiliary information is displayed in a format according to the number of unobserved areas. The auxiliary information generating unit 26 controls the format of the auxiliary information.

[0032] The display unit 27 is a liquid crystal monitor or the like. The display unit 27 sequentially displays the multiple images output from the storage unit 22. The display unit 27 also displays the auxiliary information output from the auxiliary information generation unit 26 together with the images.

[0033] At least one of the image acquisition unit 21, the area detection unit 23, the area calculation unit 24, the shooting direction determination unit 25, and the auxiliary information generation unit 26 may be configured by a processor such as a CPU (Central Processing Unit).

[0034] A computer may load a program and execute the loaded program, which includes instructions that define the operation of at least one of the image acquisition unit 21, the area detection unit 23, the area calculation unit 24, the shooting direction determination unit 25, and the auxiliary information generation unit 26. In other words, at least one function of the image acquisition unit 21, the area detection unit 23, the area calculation unit 24, the shooting direction determination unit 25, and the auxiliary information generation unit 26 may be realized by software.

[0035] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope system 1 via a transmission medium or by transmission waves in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on the computer with the difference program.

[0036] An example of the operation of the auxiliary information generating device 20 will be described with reference to Fig. 4. Fig. 4 shows an example of the procedure of the process executed by the auxiliary information generating device 20.

[0037] (Step S100) The image acquisition unit 21 acquires an image output from the imaging unit 11. The image acquisition unit 21 outputs the image to the storage unit 22. After being output to the storage unit 22, the image output from the image acquisition unit 21 is output to the area detection unit 23, the shooting direction determination unit 25, and the display unit 27. The display unit 27 displays the image output from the storage unit 22.

[0038] (Step S101) The imaging direction determination unit 25 determines whether the traveling direction of the endoscope 10 is the depth direction based on the image output from the storage unit 22. If the traveling direction of the endoscope 10 is the depth direction, the imaging direction determination unit 25 outputs traveling direction information indicating the depth direction to the area detection unit 23. At this time, step S102 is executed. If the traveling direction of the endoscope 10 is not the depth direction, step S108 is executed.

[0039] (Step S102) The shooting direction determination unit 25 determines whether the shooting position in the depth direction has changed based on the image output from the storage unit 22. When the amount of change between the image of the current frame and the image of the frame immediately before the current frame is large, the shooting direction determination unit 25 determines that the shooting position in the depth direction has changed. When the amount of change is small, the shooting direction determination unit 25 determines that the shooting position in the depth direction has not changed. When the shooting position has changed, step S103 is executed. When the shooting position has not changed, step S100 is executed.

[0040] (Step S103) When the traveling direction information output from the shooting direction determination unit 25 indicates the depth direction, the area detection unit 23 detects an unobserved area in the section corresponding to the current shooting position, based on the image output from the storage unit 22. The area detection unit 23 outputs area position information indicating the position of the unobserved area in the image to the area calculation unit 24.

[0041] For example, the region detection unit 23 estimates the camera position and performs three-dimensional (3D) reconstruction by using a technique called Visual SLAM (Simultaneous Localization and Mapping). A 3D model is restored through the 3D reconstruction. Positions on the 3D model and positions on the image are associated with each other.

[0042] If a part of the digestive tract is not visible in the image or the image is not suitable for 3D reconstruction, the part of the 3D model will not be restored. The region detection unit 23 detects that part as an unobserved region.

[0043] The region detection unit 23 may process the image output from the storage unit 22 to detect a lesion. If a lesion is captured in an image generated when the imaging direction is the depth direction D1 shown in Fig. 3A and the lesion is not captured in an image generated when the imaging direction is the direction D2 shown in Fig. 3B, the region detection unit 23 may detect the lesion as an unobserved region. As will be described later, the region detection unit 23 may detect the unobserved region by using artificial intelligence (AI).

[0044] (Step S104) The shooting direction determination unit 25 determines whether the shooting direction of the imaging unit 11 is the depth direction, based on the image output from the storage unit 22. If the shooting direction of the imaging unit 11 is the depth direction, the shooting direction determination unit 25 outputs shooting direction information indicating the depth direction to the area calculation unit 24 and the auxiliary information generation unit 26. At this time, step S105 is executed. If the shooting direction of the imaging unit 11 is not the depth direction, step S100 is executed.

[0045] (Step S105) When the shooting direction information output from the shooting direction determination unit 25 indicates the depth direction, the region calculation unit 24 calculates the number of unobserved regions. For example, the region calculation unit 24 calculates the number of unobserved regions in each of a plurality of image regions obtained by dividing the image output from the storage unit 22 in the clockwise direction.

[0046] For example, the image is divided into four image regions symmetrical about the center of the image. The four image regions are an upper right image region, an upper left image region, a lower left image region, and a lower right image region. The region calculation unit 24 calculates the number of unobserved regions in each image region based on the region position information output from the region detection unit 23. The region calculation unit 24 outputs region number information indicating the number of unobserved regions in each image region to the auxiliary information generation unit 26.

[0047] 5 shows an example of a method for calculating the number of unobserved regions. The endoscope 10 moves backward. The imaging unit 11 generates an image IMG1 at an imaging time t1, an image IMG2 at an imaging time t2, and an image IMG3 at an imaging time t3.

[0048] The region detection unit 23 uses image IMG1 to detect unobserved regions R1 and R2 that appear in image IMG1. The region detection unit 23 uses image IMG2 to detect unobserved regions R1 and R2 that appear in image IMG2. The region detection unit 23 uses image IMG3 to detect unobserved region R3 that appears in image IMG3. Unobserved regions R1 and R2 are not shown in image IMG3. When the endoscope 10 passes through the same section of the digestive tract from imaging time t1 to t3, unobserved regions R1 to R3 are included in that section. Therefore, the number of unobserved regions in that section is three.

[0049] (Step S106) When the shooting direction information output from the shooting direction determination unit 25 indicates the depth direction, the auxiliary information generation unit 26 generates auxiliary information and outputs the auxiliary information to the display unit 27. Details of step S106 will be described later.

[0050] (Step S107) The display unit 27 displays the auxiliary information output from the auxiliary information generation unit 26. Examples of the auxiliary information will be described later.

[0051] (Step S108) The imaging direction determination unit 25 determines whether the traveling direction of the endoscope 10 is the backward direction based on the image output from the storage unit 22. When the traveling direction of the endoscope 10 is the backward direction, the imaging direction determination unit 25 outputs traveling direction information indicating the backward direction to the area detection unit 23. At this time, step S109 is executed. When the traveling direction of the endoscope 10 is not the backward direction, step S100 is executed.

[0052] (Step S109) The auxiliary information generating unit outputs insertion auxiliary information for assisting the insertion of the endoscope 10 to the display unit 27. The display unit 27 displays the insertion auxiliary information output from the auxiliary information generating unit .

[0053] A first example of the operation of the auxiliary information generating device 20 in step S106 will be described with reference to Fig. 6. Fig. 6 shows an example of the procedure of the process executed by the auxiliary information generating device 20 in step S106.

[0054] (Step S200) The auxiliary information generating unit 26 sets a variable n to 0, which is an initial value.

[0055] (Step S201) The auxiliary information generating unit 26 increments the variable n by one.

[0056] (Step S202) The auxiliary information generation unit 26 references the area count information output from the area calculation unit 24 and determines whether the number of unobserved areas in the nth area is 1 or more. The first area is the upper right image area. The second area is the upper left image area. The third area is the lower left image area. The fourth area is the lower right image area. When the number of unobserved areas in the nth area is 1 or more, step S203 is executed. When the number of unobserved areas in the nth area is 0, step S206 is executed.

[0057] (Step S203) The auxiliary information generation unit 26 determines whether the number of unobserved areas in the nth area is greater than 3. When the number of unobserved areas in the nth area is greater than 3, step S204 is executed. When the number of unobserved areas in the nth area is 3 or less, step S205 is executed. The threshold value of 3 in step S203 is an example. The threshold value is not limited to 3.

[0058] (Step S204) The auxiliary information generating unit 26 generates a circle having an area that is 30% of the area of ​​the n-th region on the display screen of the display unit 27 as auxiliary information.

[0059] (Step S205) The auxiliary information generating unit 26 generates a circle having an area that is 10% of the area of ​​the n-th region on the display screen of the display unit 27 as auxiliary information.

[0060] (Step S206) The auxiliary information generating unit 26 determines whether or not the variable n is 4. If the variable n is 4, step S107 shown in Fig. 4 is executed. If the variable n is not 4, step S201 is executed.

[0061] In steps S204 and S205, the auxiliary information generation unit 26 controls the size of the icon indicating the auxiliary information for each image region according to the number of unobserved regions. When the number of unobserved regions is equal to or greater than a predetermined number, the auxiliary information generation unit 26 generates auxiliary information to be displayed in a first manner. In the above example, the first manner is a circle having an area that is 30% of the area of ​​the nth region of the display screen. When the number of unobserved regions is less than the predetermined number, the auxiliary information generation unit 26 generates auxiliary information to be displayed in a second manner different from the first manner. In the above example, the second manner is a circle having an area that is 10% of the area of ​​the nth region of the display screen.

[0062] 7A shows an example of unobserved regions detected in an image. The region detection unit 23 detects six unobserved regions R10 in the first region of the image IMG10 output from the storage unit 22. The region detection unit 23 also detects two unobserved regions R11 in the third region of the image IMG10. The region calculation unit 24 calculates the number of unobserved regions in each of the multiple image regions obtained by dividing the image IMG10 in a clockwise direction D10. The clockwise direction D10 is the circumferential direction of a circle whose center coincides with the center of the image IMG10.

[0063] 7A , the multiple sections in the digestive tract correspond to four regions obtained by dividing image IMG10 in a clockwise direction D10. When the imaging direction of the imaging unit 11 is the depth direction, the clockwise direction D10 corresponds to the circumferential direction of the digestive tract. The circumferential direction of the digestive tract is the clockwise direction in a cross section of the digestive tract perpendicular to the depth direction.

[0064] Users observe images with an emphasis on the shading or color of relatively flat surfaces, such as the inside of the digestive tract, the reproducibility of the image's gradation, or the clarity of the image, and observe the subject's health condition or the presence or absence of a lesion, so as not to miss even the slightest changes. When numerous shapes with outlines, such as the circles or frames representing the unobserved regions R10 and R11, are displayed side by side, users may find them cumbersome and cluttered. Of course, even when these are displayed on a sub-screen separate from the screen displaying the subject, the display of miscellaneous information in a relatively small area on the sub-screen can complicate what is essentially a simple guide and make it difficult to understand. When detailed shapes are displayed in a small area, it becomes difficult for users to quickly determine the meaning of their overlapping.

[0065] 7B shows an example of an image and auxiliary information displayed on display unit 27. Display unit 27 displays image IMG11. Icons IC10 and IC11 indicating auxiliary information are superimposed on image IMG11.

[0066] Icon IC10 is a circle with an area that is 30% of the area of ​​the first region of image IMG11. Icon IC10 indicates the number of unobserved regions in the first region. Icon IC10 is superimposed on the first region. Icon IC11 is a circle with an area that is 10% of the area of ​​the third region of image IMG11. Icon IC11 indicates the number of unobserved regions in the third region. Icon IC11 is superimposed on the third region.

[0067] The icons IC10 and IC11 are displayed as simplified circles, but may also be displayed as polygons, arrows, etc. Alternatively, frames representing the icons IC10 and IC11 may be displayed.

[0068] For example, if auxiliary information indicating the location or number of unobserved regions is displayed as many detailed icons or frames, the user may find it cluttered. Therefore, the information is displayed as simplified and summarized icons or frames. As described above, users observe images while prioritizing the shading or color of a relatively flat observation surface, the reproducibility of the image's gradation, or the clarity of the image, and observe the subject's health condition or the presence or absence of a lesion, etc., so as not to miss even the slightest changes. For such users, simplifying the auxiliary information may be preferable to ensure both the visibility of the observed region and the auxiliary information (icons IC10 and IC11) in an image displaying the observed region. The auxiliary information generating device 20 switches the form of such auxiliary information depending on the number of characteristic regions, thereby minimizing clutter or discomfort so as to enable the user to view the image of the actual observed region as much as possible.

[0069] The doctor can recognize the presence of an unobserved region based on icons IC10 and IC11. Because icon IC10 is larger than icon IC11, the doctor can determine that the unobserved region existing in the upper right region of the field of view of the endoscope 10 is larger than the unobserved region existing in the lower left region of the field of view of the endoscope 10.

[0070] A second example of the operation of the auxiliary information generating device 20 in step S106 will be described using Fig. 8. Fig. 8 shows an example of the procedure of the process executed by the auxiliary information generating device 20 in step S106. Processes that are the same as those shown in Fig. 6 will not be described. Processes that differ from those shown in Fig. 6 will be described.

[0071] (Step S210) The auxiliary information generating unit 26 sets the far region as the processing target. The far region is the region inside an ellipse passing through the midpoint between the center of the image and the corner of the image output from the storage unit 22. After step S210 is executed, step S200 is executed.

[0072] (Step S211) When the variable n is 4 in step S206, the auxiliary information generation unit 26 determines whether the near region is set as the processing target. The near region is the region outside an ellipse passing through the midpoint between the center of the image output from the storage unit 22 and the corner of the image. In other words, the near region is the region outside the far region. When the near region is set as the processing target, step S107 shown in FIG. 4 is executed. When the far region is set as the processing target, step S212 is executed.

[0073] (Step S212) The auxiliary information generating unit 26 sets the near region as a processing target. After step S212 is executed, step S200 is executed.

[0074] In the example shown in FIG. 8, the image output from the storage unit 22 is divided into four image regions in the far region and four image regions in the near region.

[0075] 9A shows an example of unobserved regions detected in an image. In image IMG12 output from memory unit 22, the far region is the region inside line L10, and the near region is the region outside line L10. Region detection unit 23 detects four unobserved regions R12 in the first region in the far region of image IMG12. Region detection unit 23 also detects two unobserved regions R13 in the first region and two unobserved regions R14 in the third region in the near region of image IMG12.

[0076] The region calculation unit 24 calculates the number of unobserved regions in each of a plurality of image regions obtained by dividing the image IMG10 output from the storage unit 22 in a direction D11 and a clockwise direction D12. The direction D11 is the direction from the center of the image IMG12 toward the outer periphery of the image IMG12. The clockwise direction D12 is the circumferential direction of a circle whose center coincides with the center of the image IMG12.

[0077] In the example shown in Figure 9A, each section in the digestive tract corresponds to eight regions obtained by dividing the image IMG12 in a direction D11 and a clockwise direction D12. When the imaging direction of the imaging unit 11 is the depth direction, the direction D11 corresponds to the radial direction of the digestive tract, and the clockwise direction D12 corresponds to the circumferential direction of the digestive tract. The radial direction of the digestive tract is parallel to a line that passes through the center of a cross section of the digestive tract perpendicular to the depth direction and is perpendicular to the depth direction. The circumferential direction of the digestive tract is the clockwise direction in the cross section of the digestive tract perpendicular to the depth direction.

[0078] When a large number of shapes with outlines are lined up, such as the circles or frames that indicate the unobserved regions R12 to R14, the user may find them cumbersome and cluttered. Even when these are displayed on a sub-screen that is different from the screen that displays the object, if miscellaneous information is lined up in a relatively small area on the sub-screen, the originally simple guide may become complicated and difficult to understand.

[0079] 9B shows an example of an image and auxiliary information displayed on display unit 27. Display unit 27 displays image IMG13. Icons IC12, IC13, and IC14 indicating auxiliary information are superimposed on image IMG13.

[0080] Icon IC12 is a circle with an area that is 30% of the area of ​​the first region in the far region of image IMG13. Icon IC12 indicates the number of unobserved regions in the first region in the far region. Icon IC12 is superimposed on the first region in the far region.

[0081] Icon IC13 is a circle having an area that is 10% of the area of ​​the first region in the near region of image IMG13. Icon IC13 indicates the number of unobserved regions in the first region in the near region. Icon IC13 is superimposed on the first region in the near region.

[0082] Icon IC14 is a circle having an area that is 10% of the area of ​​the third region in the near region of image IMG13. Icon IC14 indicates the number of unobserved regions in the third region in the near region. Icon IC14 is superimposed on the third region in the near region.

[0083] For example, if auxiliary information indicating the location or number of unobserved regions is displayed as many small icons or frames, the user will find it cluttered. Therefore, the information is displayed as simplified and summarized icons or frames. In order to ensure both the visibility of the observed region and the auxiliary information (icons IC12 to IC14) in an image displaying the observed region, it may be preferable to simplify the auxiliary information. The auxiliary information generating device 20 switches the form of such auxiliary information depending on the number of characteristic regions, thereby minimizing clutter or discomfort so that the image of the original observed region can be observed as much as possible.

[0084] A third example of the operation of the auxiliary information generating device 20 in step S106 will be described using Fig. 10. Fig. 10 shows an example of the procedure of the process executed by the auxiliary information generating device 20 in step S106. Processes that are the same as those shown in Fig. 8 will not be described. Processes that differ from those shown in Fig. 8 will be described.

[0085] Step S204 shown in FIG. 8 is changed to step S204a, and step S205 shown in FIG. 8 is changed to step S205a.

[0086] (Step S204a) The auxiliary information generating unit 26 sets the color of the n-th region of the auxiliary display unit to a first color. The auxiliary display unit is a part of the display screen of the display unit 27. The first color corresponds to the auxiliary information. For example, the first color is yellow, but is not limited to this.

[0087] (Step S205a) The auxiliary information generator 26 sets the color of the n-th region of the auxiliary display unit to a second color different from the first color. The second color corresponds to the auxiliary information. For example, the second color is red, but is not limited to this.

[0088] 11 shows an example of the display screen of display unit 27. Image IMG14 is displayed on display screen SC10. Display screen SC10 has auxiliary display unit SUB10.

[0089] Display region DR10 of auxiliary display unit SUB10 is displayed in a first color. Display region DR10 corresponds to a first region in the far region. Display regions DR11 and DR12 of auxiliary display unit SUB10 are displayed in a second color. Display region DR11 corresponds to a first region in the near region, and display region DR12 corresponds to a third region in the near region.

[0090] The region detection unit 23 sequentially acquires multiple images (image frames) obtained by the imaging unit 11 of the endoscope 10 and uses the image frames or groups of image frames to detect characteristic regions at multiple positions within the subject's digestive tract, for example. The region detection unit 23 may detect characteristic regions by using an inference model (AI) obtained by deep learning or the like using feature information appearing in the images as training data. Alternatively, the region detection unit 23 may use 3D model construction technology to stitch together multiple images and determine that there is an unobserved region (a hole in the 3D model) if there are no image frames to use as material for stitching. The region detection unit 23 may also be implemented by simply using logic that determines feature information based on rules, image processing calculations, or the like.

[0091] In each embodiment of the present invention, the region detection unit 23 detects the characteristic regions by using an image or a group of images acquired in each of multiple sections of the digestive tract. The auxiliary information generation unit 26 determines the number of characteristic regions and generates auxiliary information according to the number of characteristic regions. The auxiliary information is displayed on the image on the display unit 27, or in a region (display region DR11) different from the region where the image is displayed on the display unit 27. When this auxiliary information is displayed on the display unit 27, the auxiliary information generation unit 26 controls the mode of the auxiliary information according to the number of characteristic regions. Therefore, depending on the situation, the auxiliary information generation unit 26 may need to use the result of recording the history of information detected by the region detection unit 23 as the number of characteristic regions. In this case, the region detection unit 23 may be equipped with a function for recording information.

[0092] The above feature regions are assumed to be regions with features that are likely to be overlooked and regions where oversight actually occurred (holes in the 3D model). Specific examples of regions with features that are likely to be overlooked include the following: Example 1: Regions with residue, foam, or feces attached Example 2: Regions where the mucosa is contracted Example 3: Regions where the lumen is curved Example 4: Regions where there are many or deep folds

[0093] As described above, the auxiliary information generating unit 26 generates auxiliary information according to the number of unobserved areas. When the auxiliary information is displayed on the display unit 27, the auxiliary information generating unit 26 controls the mode of the auxiliary information according to the number of unobserved areas. Therefore, the auxiliary information generating unit 26 can avoid cumbersome notifications of the auxiliary information and a decrease in the visibility of the image.

[0094] Second Embodiment A second embodiment of the present invention will be described. In the second embodiment, an endoscope system 1 shown in Fig. 1 is used. An auxiliary information generating device 20 executes the processing shown in Fig. 4.

[0095] A first example of the operation of the auxiliary information generating device 20 in step S106 will be described with reference to Fig. 12. Fig. 12 shows an example of the procedure of the process executed by the auxiliary information generating device 20 in step S106.

[0096] (Step S220) The auxiliary information generation unit 26 references the area count information output from the area calculation unit 24 and determines whether the number of unobserved areas is equal to or greater than a predetermined number. If the number of unobserved areas is equal to or greater than the predetermined number, step S221 is executed. If the number of unobserved areas is less than the predetermined number, step S222 is executed.

[0097] (Step S221) The auxiliary information generator 26 generates auxiliary information according to the coverage rate. The coverage rate indicates the ratio of the area of ​​the digestive tract that does not include unobserved areas to the area of ​​the entire area. Alternatively, the auxiliary information generator 26 generates a warning notice as auxiliary information. The warning notice is an icon that notifies the user that there are more than a predetermined number of unobserved areas.

[0098] For example, the region detection unit 23 performs 3D reconstruction in step S103 shown in FIG. 4 to generate a 3D point cloud that constitutes a 3D model. The region detection unit 23 interpolates the 3D point cloud of a hole region. The hole region corresponds to an unobserved region. The auxiliary information generation unit 26 calculates the coverage rate CO (%) according to the following formula (1): CO=100*(1-N1 / N2) (1)

[0099] N1 in equation (1) represents the number of interpolated 3D point clouds, and N2 in equation (1) represents the number of all 3D point clouds including the interpolated 3D point clouds.

[0100] The region detection unit 23 may generate mesh (polygon) data from the 3D point cloud. The region detection unit 23 may interpolate meshes of hole regions in a plurality of meshes. The auxiliary information generation unit 26 may calculate the coverage rate CO (%) according to the following formula (2) or formula (3): CO=100*(1-A1 / A2) (2) CO=100*(1-N3 / N4) (3)

[0101] A1 in formula (2) indicates the area of ​​the interpolated mesh, A2 in formula (2) indicates the area of ​​the entire mesh including the interpolated mesh, N3 in formula (3) indicates the number of interpolated meshes, and N4 in formula (3) indicates the number of entire meshes including the interpolated mesh.

[0102] The auxiliary information generating unit 26 may generate auxiliary information according to the number of hole regions, instead of the coverage rate.

[0103] As described above, users observe images by focusing on the shading or color of a relatively flat observation surface, the reproducibility of the gradation expression of the image, or the clarity of the image, and observe the health condition of the object or the presence or absence of a lesion, etc., so as not to miss even the slightest changes. When a large number of shapes with outlines are lined up, such as the circles or frames indicating the unobserved regions R10 and R11, the user may find them cumbersome and cluttered.

[0104] (Step S222) The auxiliary information generating unit 26 generates auxiliary information according to the position of the unobserved region in the image.

[0105] 13A shows a first example of a display screen of display unit 27. An image IMG20 and an icon IC20 are displayed on display screen SC20.

[0106] Icon IC20 constitutes the auxiliary information generated in step S221 and indicates the coverage rate for each of multiple sections in the digestive tract. Icon IC20 is configured in a bar shape and is divided into multiple regions in the vertical direction. For example, the lower region of icon IC20 indicates the coverage rate for the front section of the digestive tract shown in image IMG20. The upper region of icon IC20 indicates the coverage rate for the back section of the digestive tract shown in image IMG20. Each region of icon IC20 is displayed in a color corresponding to the coverage rate for the corresponding section.

[0107] 13B shows a second example of the display screen of display unit 27. An image IMG20 and a message MS20 are displayed on display screen SC21.

[0108] The message MS20 constitutes the auxiliary information generated in step S221 and indicates the coverage rate. The coverage rate in the message MS20 indicates the coverage rate of one of the multiple sections in the digestive tract. The coverage rates of the multiple sections may be displayed on the display screen SC21.

[0109] 13C shows a third example of the display screen of display unit 27. Image IMG20 is displayed on display screen SC22. Display screen SC22 has auxiliary display unit SUB20.

[0110] Auxiliary display unit SUB20 displays a plurality of lines corresponding to the positions of the folds in image IMG20. The plurality of lines constitutes the auxiliary information generated in step S221. A plurality of sections in the digestive tract are separated by the folds. Each of the plurality of lines is displayed in a color corresponding to the coverage rate of the respective section.

[0111] 14A shows a fourth example of the display screen of display unit 27. Image IMG20 is displayed on display screen SC23. Display screen SC23 has auxiliary display unit SUB21.

[0112] The auxiliary display unit SUB21 displays a schematic diagram of the digestive tract constituting the auxiliary information generated in step S221. Each of the multiple sections in the digestive tract corresponds to a part such as the rectum. Each of the multiple sections is displayed in a color corresponding to the coverage rate of the respective section.

[0113] 14B shows a fifth example of a display screen of display unit 27. Image IMG20 is displayed on display screen SC24.

[0114] An attention notification AN20 is superimposed on the image IMG20. The attention notification AN20 indicates that there are a predetermined number of unobserved areas or more.

[0115] 15B shows a sixth example of a display screen of display unit 27. Image IMG20 is displayed on display screen SC25.

[0116] Position information PI20 is superimposed on image IMG20. The position information PI20 constitutes auxiliary information generated in step S222 and indicates the position of the unobserved area. The position information PI20 is displayed at a position on image IMG20 that corresponds to the position of the unobserved area. The position information PI20 may be displayed as an arrow indicating the position of the unobserved area.

[0117] A second example of the operation of the auxiliary information generating device 20 in step S106 will be described with reference to Fig. 15. Fig. 15 shows an example of the procedure of the process executed by the auxiliary information generating device 20 in step S106.

[0118] (Step S230) Step S105 shown in Fig. 4 is included in step S230. The region calculation unit 24 calculates the number of unobserved regions in the section corresponding to the current shooting position (current section). The number of unobserved regions in the current section does not include the number of unobserved regions in one or more sections (past sections) through which the endoscope 10 has already passed. The region calculation unit 24 outputs region number information indicating the number of unobserved regions to the auxiliary information generation unit 26.

[0119] (Step S231) The auxiliary information generation unit 26 references the area count information output from the area calculation unit 24 and determines whether the number of unobserved areas is equal to or greater than a predetermined number. If the number of unobserved areas is equal to or greater than the predetermined number, step S232 is executed. If the number of unobserved areas is less than the predetermined number, step S234 is executed.

[0120] (Step S232) The auxiliary information generating unit 26 calculates a coverage rate according to the number of unobserved regions. The method for calculating the coverage rate is the same as the method for calculating the coverage rate in step S221 shown in FIG.

[0121] (Step S233) The auxiliary information generating unit 26 generates auxiliary information according to the coverage rate.

[0122] (Step S234) The auxiliary information generating unit 26 generates auxiliary information indicating the position of the unobserved area in the image.

[0123] A third example of the operation of the auxiliary information generating device 20 in step S106 will be described using Figure 16. Figure 16 shows an example of the procedure of the process executed by the auxiliary information generating device 20 in step S106. Processes that are the same as those shown in Figure 15 will not be described. Processes that differ from the process shown in Figure 15 will be described.

[0124] Step S230 shown in FIG. 15 is changed to step S230a.

[0125] (Step S230a) The region calculation unit 24 calculates the number of unobserved regions in one or more sections (past sections) through which the endoscope 10 has already passed. The number of unobserved regions in the past sections does not include the number of unobserved regions in the section (current section) corresponding to the current imaging position. The region calculation unit 24 outputs region number information indicating the number of unobserved regions to the auxiliary information generation unit 26.

[0126] Once steps S232 and S233 are executed, the execution of steps S231 and S234 may be stopped. That is, after step S230a is executed, step S232 may be executed without executing step S231.

[0127] As described above, the auxiliary information generation unit 26 generates auxiliary information according to the number of unobserved areas. When the number of unobserved areas is large, the auxiliary information generation unit 26 generates auxiliary information indicating the coverage rate, allowing the user to roughly grasp the existence of unobserved areas. When the number of unobserved areas is small, the auxiliary information generation unit 26 generates auxiliary information indicating the positions of the unobserved areas, allowing the user to grasp the precise positions of the unobserved areas.

[0128] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims.

[0129] According to each embodiment of the present invention, the auxiliary information generation method, auxiliary information generation device, and program can avoid the cumbersome notification of auxiliary information obtained from multiple images, including images acquired in the past. Here, a gastrointestinal endoscope is used as an example, but it goes without saying that each embodiment of the present invention can be used with any measuring or inspection device that performs observation, measurement, and inspection by sequentially changing its position in the depth direction.

[0130] REFERENCE SIGNS LIST 1 Endoscope system 10 Endoscope 11 Imaging unit 20 Auxiliary information generating device 21 Image acquisition unit 22 Storage unit 23 Area detection unit 24 Area calculation unit 25 Imaging direction determination unit 26 Auxiliary information generating unit 27 Display unit

Claims

1. sequentially acquiring a plurality of images obtained by an imaging unit of an endoscope at a plurality of positions in the gastrointestinal tract of a subject; detecting characteristic regions in the digestive tract based on the plurality of images; Calculating the number of the characteristic regions in each of a plurality of sections in the digestive tract; generating auxiliary information according to the number of the feature regions; determining whether the imaging unit is performing imaging in a depth direction of the digestive tract; When it is determined that the imaging unit is performing imaging in the depth direction and the auxiliary information is displayed on the display unit, a mode of the auxiliary information is controlled in accordance with the number of the characteristic areas. Auxiliary information generation method.

2. The image obtained by the imaging unit is divided in a direction corresponding to the circumferential direction of the digestive tract into a plurality of image regions, and the auxiliary information for each image region is generated. The auxiliary information generating method according to claim 1 .

3. A size of an icon indicating the auxiliary information for each of the plurality of image regions is controlled in accordance with the number of the characteristic regions. The auxiliary information generating method according to claim 2 .

4. Dividing the image obtained by the imaging unit in a direction corresponding to the radial direction of the digestive tract, and generating the auxiliary information for each of a plurality of image regions obtained by dividing the image in a direction corresponding to the circumferential direction of the digestive tract. The auxiliary information generating method according to claim 1 .

5. generating the auxiliary information to be displayed in a first manner when the number of the feature regions is equal to or greater than a predetermined number; When the number of the characteristic regions is less than the predetermined number, the auxiliary information is generated to be displayed in a second manner different from the first manner. The auxiliary information generating method according to claim 1 .

6. The auxiliary information displayed in the second mode indicates the position of the characteristic region. The auxiliary information generating method according to claim 5 .

7. an image acquisition unit that sequentially acquires a plurality of images obtained by an imaging unit of the endoscope at a plurality of positions in the digestive tract of the subject; a region detection unit that detects a characteristic region in the digestive tract based on the plurality of images; a region calculation unit that calculates the number of the characteristic regions in each of a plurality of sections in the digestive tract; an auxiliary information generating unit that generates auxiliary information in accordance with the number of the feature regions; an imaging direction determination unit that determines whether the imaging unit is performing imaging in a depth direction of the digestive tract; Equipped with When it is determined that the imaging unit is performing imaging in the depth direction and the auxiliary information is displayed on the display unit, the auxiliary information generation unit controls the manner of the auxiliary information in accordance with the number of the characteristic regions. Auxiliary information generation device.

8. sequentially acquiring a plurality of images obtained by an imaging unit of an endoscope at a plurality of positions in the gastrointestinal tract of a subject; detecting a characteristic region in the digestive tract based on the plurality of images; calculating the number of the characteristic regions in each of a plurality of sections in the digestive tract; generating auxiliary information according to the number of the feature regions; determining whether the imaging unit is performing imaging in a depth direction of the digestive tract; controlling a mode of the auxiliary information in accordance with the number of the feature regions when it is determined that the imaging unit is capturing an image in the depth direction and the auxiliary information is displayed on a display unit; A program that causes a computer to execute the following.

9. Acquiring images obtained by an imaging unit at the tip of an endoscope at each position along the lumen within the lumen, Detecting a characteristic area at each position and an imaging direction of the imaging unit inside the lumen based on the image; Calculating the number of feature regions at each of the positions; generating auxiliary information according to the number of the feature regions; When the imaging direction is determined to be the depth direction of the lumen and the auxiliary information is displayed on a display unit, a manner of the auxiliary information is controlled in accordance with the number of the characteristic regions at each position. Auxiliary information generation method.

10. A region calculation unit that detects characteristic regions at each position along the lumen within the lumen based on images obtained by an imaging unit at the tip of the endoscope, and calculates the number of characteristic regions at each position; an imaging direction determination unit that determines an imaging direction of the imaging unit inside the lumen; an auxiliary information generating unit that generates the auxiliary information in accordance with the number of the feature regions at each position when the imaging direction is determined to be a depth direction of the lumen and auxiliary information is displayed on a display unit; An auxiliary information generating device having:

11. A step of detecting characteristic regions at each position along the lumen within the lumen based on images obtained by an imaging unit at the tip of the endoscope, and calculating the number of characteristic regions at each position; determining an imaging direction of the imaging unit inside the lumen; generating the auxiliary information according to the number of the feature regions at each position when the imaging direction is determined to be a depth direction of the lumen and auxiliary information is displayed on a display unit; A program that causes a computer to execute the following.