Image processing device and endoscope system

Abstract

The invention provides an image processing device and an endoscope system capable of ascertaining observation conditions based on an endoscope. An image processing device that processes a plurality of images captured in time series by an endoscope and that is provided with a processor. The processor acquires a plurality of images. The processor processes the image and identifies a region of interest that appears in the image from among a plurality of regions of interest in the body. A first region of interest among a plurality of regions of interest is recognized from a first image among a plurality of images, and a second region of interest among the plurality of regions of interest is recognized from a second image that is chronologically later than the first image. The processor displays, on the display device, information indicating that a region between the first region of interest and the second region of interest is observed.

Description

Technical Field

[0001] The present invention relates to an image processing device and an endoscope system, and more particularly to an image processing device for processing a plurality of images captured in time series by an endoscope and an endoscope system having the image processing device. Background Art

[0002] As a technology to assist endoscopic observation, there is known a technology that automatically recognizes a region of interest such as a lesion based on an image taken by an endoscope and notifies it (for example, Patent Documents 1 and 2). In particular, in recent years, due to the development of artificial intelligence (AI) technology, high-precision image recognition has become possible.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2020 / 110214

[0006] Patent Document 2: International Publication No. 2020 / 152758 Summary of the invention

[0007] Technical issues to be solved by the invention

[0008] However, even if image recognition is excellent, lesions or the like present in a part not photographed cannot be detected.

[0009] One embodiment of the technology according to the present invention provides an image processing device and an endoscope system capable of grasping an observation situation using an endoscope.

[0010] Means for solving technical problems

[0011] (1) An image processing device that processes a plurality of images captured by an endoscope in a time series, the image processing device comprising a processor that performs the following processing: acquiring a plurality of images; processing the images to identify a region of interest projected in the images from a plurality of regions of interest in a body; and displaying on a display device information indicating that a region between the first region of interest and the second region of interest is observed when a first region of interest among the plurality of regions of interest is identified from a first image among the plurality of images and a second region of interest among the plurality of regions of interest is identified from a second image that is later in time than the first image.

[0012] (2) An image processing device according to (1), wherein the processor performs the following processing: displays information of multiple specific regions of interest selected from multiple regions of interest in the body as first information on a display device; and displays information indicating an observed area between the first region of interest and the second region of interest as second information on the display device.

[0013] (3) The image processing device according to (2), wherein the second information is composed of information associating the first region of interest with the second region of interest in the first information.

[0014] (4) The image processing device according to (2) or (3), wherein the second information includes information indicating a moving direction of observation.

[0015] (5) The image processing device according to (3), wherein the first information is composed of information in which marks indicating the target region are arranged in a specific layout, and the second information is composed of information in which the marks are associated.

[0016] (6) The image processing device according to (5), wherein the first information includes a schematic diagram of the organ to be observed, and is composed of information in which a marker is arranged at a position of each region of interest on the schematic diagram.

[0017] (7) The image processing device according to (5) or (6), wherein the second information is composed of line segments connecting the markers.

[0018] (8) The image processing device according to (7), wherein the line segment includes an arrow indicating a moving direction of observation.

[0019] (9) The image processing device according to any one of (5) to (8), wherein the processor switches a display mode of the mark of the recognized interest region from a first mode to a second mode.

[0020] (10) The image processing device according to any one of (2) to (9), wherein the processor calculates the observation evaluation value based on an image between the first image and the second image in chronological order, and the second information includes information on the evaluation value.

[0021] (11) The image processing device according to (10), wherein the evaluation value is calculated based on the number of images satisfying the first criterion and / or the number of images between the first image and the second image.

[0022] (12) The image processing device according to (11), wherein the first reference is set based on at least one of a blur state of the image, a shake state of the image, and brightness of the image.

[0023] (13) An image processing device according to any one of (10) to (12), wherein the evaluation value is calculated as a ratio of images that satisfy the first criterion or a ratio of images that do not satisfy the first criterion.

[0024] (14) The image processing device according to (13), wherein the processor performs the following processing: extracting images that are similar to each other from images between the first image and the second image; and calculating the evaluation value by excluding one of the extracted similar images.

[0025] (15) The image processing device according to any one of (10) to (14), wherein the processor displays the second information in a manner corresponding to the evaluation value.

[0026] (16) An image processing device according to any one of (10) to (15), wherein, when a first region of interest is identified from a third image that is later than the second image in chronological order and a second region of interest is identified from a fourth image that is later than the third image in chronological order, the processor updates the evaluation value and updates the display of the second information.

[0027] (17) An image processing device according to any one of (1) to (16), wherein the processor performs the following processing: displaying a plurality of images on a display device in chronological order; accepting selection of a first image from images displayed on the display device; and accepting selection of a second image from images displayed on the display device after the first image.

[0028] (18) The image processing device according to (17), wherein the processor records the images selected as the first image and the second image as still images.

[0029] (19) The image processing device according to any one of (1) to (18), wherein the processor processes each of the plurality of images and identifies the target region reflected in each of the images.

[0030] (20) An image processing device according to any one of (1) to (19), wherein, when a region of interest is identified from an image, the processor further determines whether the image in which the region of interest is identified satisfies a second criterion, and confirms the identification of the region of interest if the second criterion is satisfied.

[0031] (21) The image processing device according to (20), wherein the content of the second reference is set for each region of interest.

[0032] (22) The image processing device according to any one of (2) to (8), wherein the processor ends display of the second information after a time T has elapsed since display of the second information was started.

[0033] (23) The image processing device according to (22), wherein the processor performs the following processing: recording a display history of the second information; and when redisplaying the second information, simultaneously displaying the second information displayed in the past based on the history.

[0034] (24) The image processing device according to (23), wherein when observation is completed and / or when display of the history is instructed, the processor displays the second information displayed in the past based on the history.

[0035] (25) An image processing device according to any one of (2) to (8), wherein the processor ends display of the second information when the first region of interest is recognized from a third image that is chronologically later than the second image.

[0036] (26) The image processing device according to any one of (1) to (25), wherein the plurality of regions of interest are a plurality of parts of an organ to be observed.

[0037] (27) An endoscope system comprising: an endoscope; a display device; and the image processing device according to any one of (1) to (26), which processes an image captured by the endoscope.

[0038] Effects of the Invention

[0039] According to the present invention, it is possible to grasp the observation status using an endoscope. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a block diagram showing an example of the system configuration of an endoscope system.

[0041] Figure 2 This is a block diagram of the main functions of a processor device.

[0042] Figure 3 This is a block diagram showing an example of the hardware configuration of the image processing device.

[0043] Figure 4 This is a block diagram of the main functions of the image processing device.

[0044] Figure 5 This is a block diagram of the main functions of the image processing unit.

[0045] Figure 6 A diagram showing an example of a screen display of a display device.

[0046] Figure 7 This is a diagram showing an example of transition of the display of the observation status display window.

[0047] Figure 8This is a flowchart showing the procedure of processing when information indicating the observation status is displayed on the display device.

[0048] Fig. 9 This is a block diagram of main functions of the observation status determination unit of the image processing device.

[0049] Fig.10 This is a conceptual diagram of the determination of the photographic area and the calculation of the evaluation value performed by the observation condition determination unit.

[0050] Fig.11 It is a figure which shows an example of the display of the observation status display window.

[0051] Fig.12 This is a diagram showing another example of display of information indicating the observation status.

[0052] Fig.13 1 is a diagram showing an example of transition of the display of the observation status display window W.

[0053] Fig.14 This is a diagram showing another example of display of information indicating the observation status.

[0054] Fig.15 This is a diagram showing another example of display of information indicating the observation status.

[0055] Fig.16 1 is a diagram showing an example of the display of the observation status display window after the part is recognized.

[0056] Fig.17 1 is a diagram showing an example of transition of the display of the observation status display window W.

[0057] Fig.18 This is a diagram showing another example of display of information indicating the observation status.

[0058] Fig.19 This is a diagram showing another example of display of information indicating the observation status.

[0059] Fig. 20 1 is a diagram showing an example of the display of the observation status display window after the part is recognized.

[0060] Fig.21 This is a diagram showing another example of marking.

[0061] Fig. 22 It is a diagram showing an example of emphasized display of evaluation values.

[0062] Fig.23 FIG. 1 is a diagram showing an example of a case where an arrow is highlighted and displayed.

[0063] Fig.24This is a flowchart showing the procedure of processing when the evaluation value is automatically recalculated when the already recognized part is recognized again.

[0064] Fig.25 This is a conceptual diagram of the process of releasing the determination of “recognized”.

[0065] Fig.26 This is a block diagram of an observation status determination unit having a function of calculating an evaluation value by excluding similar images.

[0066] Fig. 27 This is a conceptual diagram of when the evaluation value is calculated in real time.

[0067] Fig.28 This is a diagram showing an example of displaying evaluation values ​​in real time.

[0068] Fig.29 This is a diagram showing an example of transition of the display of the observation status display window.

[0069] Fig.30 This is a diagram showing an example of the display of the observation status display window when observing the large intestine. DETAILED DESCRIPTION

[0070] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0071] [First embodiment]

[0072] Here, the present invention is applied to an endoscope system for observing (including observation for inspection) an upper digestive organ in the body, particularly the stomach. The stomach is an example of an organ to be observed, particularly a lumen organ.

[0073] [Structure of the endoscope system]

[0074] Figure 1 This is a block diagram showing an example of the system configuration of an endoscope system.

[0075] like Figure 1 As shown, the endoscope system 1 of this embodiment includes an endoscope 10, a light source device 20, a processor device 30, an input device 40, a display device 50, and an image processing device 100. The endoscope 10 is connected to the light source device 20 and the processor device 30. The light source device 20, the input device 40, and the image processing device 100 are connected to the processor device 30. The display device 50 is connected to the image processing device 100.

[0076] The endoscope system 1 of the present embodiment is configured as a system that can perform observation using special light (special light observation) in addition to observation based on ordinary white light (white light observation). Special light observation includes narrow-band light observation. Narrow-band light observation includes BLI observation (blue laser imaging technology (Blue laser imaging) observation), NBI observation (narrow band imaging technology (Narrow band imaging) observation), LCI observation (linked color imaging technology (Linked Color Imaging) observation), etc. In addition, special light observation itself is a well-known technology, so the description of its details is omitted.

[0077] [Endoscope]

[0078] The endoscope 10 of the present embodiment is an electronic endoscope (flexible endoscope), in particular, an electronic endoscope for upper digestive organs. The electronic endoscope includes an operating portion, an insertion portion, and a connection portion, and the subject is photographed by an image sensor assembled to the front end of the insertion portion. The image sensor uses an image sensor (for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, etc.) with a prescribed color filter arrangement (for example, a Bayer arrangement). The operating portion includes an angle button, an air and water supply button, a suction button, a mode switching button, a release button, and a forceps port. The mode switching button is a button for switching the observation mode. For example, the mode for white light observation, the mode for LCI observation, and the mode for BLI observation are switched. The release button is a button for instructing the photography of a still image. In addition, the endoscope itself is well known, so the detailed description thereof is omitted. The endoscope 10 is connected to the light source device 20 and the processor device 30 via the connection portion.

[0079] [Light source device]

[0080] The light source device 20 generates illumination light supplied to the endoscope 10. As described above, the endoscope system 1 of the present embodiment is configured as a system capable of performing special light observation in addition to normal white light observation. Therefore, the light source device 20 has a function of generating light (e.g., narrow-band light) corresponding to special light observation in addition to normal white light. In addition, as described above, since special light observation itself is a well-known technology, the description of the generation of the illumination light is omitted. The switching of the light source type is performed, for example, using a mode switching button provided on the operating unit of the endoscope 10.

[0081] [Processor device]

[0082] The processor device 30 centrally controls the operation of the entire endoscope system and includes a processor, a main storage device, an auxiliary storage device, an input / output interface, and the like as its hardware configuration.

[0083] Figure 2 This is a block diagram of the main functions of a processor device.

[0084] like Figure 2 As shown, the processor device 30 has functions such as an endoscope control unit 31, a light source control unit 32, an image processing unit 33, an input control unit 34, and an output control unit 35. Each function is realized by executing a predetermined program by the processor. Various programs executed by the processor and various data required for control are stored in the auxiliary storage device.

[0085] The endoscope control unit 31 controls the endoscope 10. The control of the endoscope 10 includes the drive control of the image sensor, the control of air and water supply, the control of suction, and the like.

[0086] The light source control unit 32 controls the light source device 20. The control of the light source device 20 includes light emission control of the light source, switching control of the light source type, and the like.

[0087] The image processing unit 33 performs various signal processing on a signal (image signal) output from the image sensor of the endoscope 10 to generate an image.

[0088] The input control unit 34 performs processing for receiving input of operations from the input device 40 and the operating unit of the endoscope 10 and input of various information.

[0089] The output control unit 35 controls the output of information from the processor device 30 to the image processing device 100. The information output from the processor device 30 to the image processing device 100 includes, in addition to the image (endoscopic image) captured by the endoscope, information input via the input device 40 and various operation information. Among the various operation information, in addition to the operation information of the input device 40, operation information of the operation unit of the endoscope 10 is also included. The operation information includes a photography instruction of a still image. As described above, the photography instruction of the still image is performed using a release button provided on the operation unit of the endoscope 10. In addition, the photography instruction of the still image can also be set to a structure that is performed via a foot switch, a sound input device, a touch panel, etc. The images captured by the endoscope in time series are output to the image processing device 100 in sequence.

[0090] [Input device]

[0091] The input device 40 and the display device 50 together constitute a user interface in the endoscope system 1. The input device 40 is constituted by, for example, a keyboard, a mouse, a foot switch, etc. In addition, the input device 40 can be configured to include a touch panel, a voice input device, a visual input device, etc.

[0092] [Display device]

[0093] The display device 50 is used to display various information in addition to the endoscopic image. The display device 50 is composed of, for example, a liquid crystal display (LCD), an organic electroluminescence display (OELD), etc. In addition, the display device 50 can also be composed of a projector, a head-mounted display, etc.

[0094] [Image processing device]

[0095] The image processing device 100 processes the images output from the processor device 30. Specifically, the image processing device 100 performs a process of sequentially displaying the images sequentially output from the processor device 30 on the display device 50. Furthermore, the image processing device 100 sequentially processes the images sequentially output from the processor device 30 and performs a process of detecting a lesion from the image. Furthermore, the image processing device 100 performs a process of identifying the detected lesion. Furthermore, the image processing device 100 performs a process of displaying the detection result and the identification result of the lesion on the display device 50. Furthermore, the image processing device 100 performs a process of capturing and recording a still image and / or a moving image according to an instruction from a user. Furthermore, when a still image is captured, the image processing device 100 performs a process of identifying the part of an organ reflected in the captured still image. Furthermore, the image processing device 100 performs a process of displaying information indicating the observation status of the organ by the endoscope 10 on the display device 50 based on the identification result of the part of the organ. In the present embodiment, the plurality of parts of the organ to be identified are an example of a plurality of regions of interest in the body.

[0096] Figure 3 This is a block diagram showing an example of the hardware configuration of the image processing device.

[0097] The image processing device 100 is composed of a so-called computer, and as its hardware structure, it has a processor 101, a main storage device (main memory) 102, an auxiliary storage device (memory) 103, an input / output interface 104, etc. The image processing device 100 is connected to the processor device 30 and the display device 50 via the input / output interface 104. The auxiliary storage device 103 is composed of, for example, a hard disk drive (HDD), a flash memory including an SSD (Solid State Drive), etc. The auxiliary storage device 103 stores various data required for programs and controls executed by the processor 101. In addition, images (still images and dynamic images) taken by the endoscope are recorded in the auxiliary storage device 103. The detection results of the lesion, the identification results, the identification results of the site, and the determination results of the observation status are also recorded in the auxiliary storage device 103.

[0098] Figure 4 This is a block diagram of the main functions of the image processing device.

[0099] like Figure 4 As shown, the image processing device 100 has functions such as an image acquisition unit 111, a command acquisition unit 112, an image processing unit 113, a recording control unit 114, and a display control unit 115. The functions of each unit are realized by the processor 101 executing a predetermined program (image processing program).

[0100] The image acquisition unit 111 performs a process of sequentially acquiring images sequentially output from the processor device 30. As described above, images captured in time series by the endoscope 10 are sequentially output from the processor device 30. Therefore, the image acquisition unit 111 acquires images captured in time series by the endoscope 10 in time order.

[0101] The command acquisition unit 112 acquires command information. The command information includes information on instructions for taking still images.

[0102] The image processing unit 113 processes the image acquired by the image acquisition unit 111 . Figure 5 This is a block diagram of the main functions of the image processing unit. Figure 5 As shown, the image processing unit 113 of the image processing device 100 of this embodiment has functions such as a lesion detection unit 113A, an identification unit 113B, a part recognition unit 113C, and an observation status determination unit 113D.

[0103] The lesion detection unit 113A processes the input image and detects the lesion (for example, polyp, etc.) reflected in the image. The lesion includes, in addition to the part that is determined to be the lesion, a part that may be a lesion (benign tumor or atypical hyperplasia, etc.) and a part that has characteristics that may be directly or indirectly related to the lesion (redness, etc.). The lesion detection unit 113A is composed of a learned model that is trained to detect the lesion from the image. The detection of the lesion using the learned model itself is a well-known technology, so the detailed description thereof is omitted. As an example, it is composed of a model using a convolutional neural network (CNN). In addition, the detection of the lesion can include the discrimination of the category of the detected lesion.

[0104] The identification unit 113B performs identification processing on the lesion detected by the lesion detection unit 113A. As an example, in this embodiment, the lesion such as polyps detected by the lesion detection unit 113A is subjected to neoplastic (NEOPLASTIC) or non-neoplastic (HYPERPLASTIC) identification processing. The identification unit 113B is composed of a learning model trained to identify lesions from images.

[0105] The part recognition unit 113C processes the input image and performs a process of identifying the part of the organ projected in the image. In the present embodiment, since the stomach is taken as the observation object, the part recognition unit 113C performs a process of identifying the part of the stomach. The part to be recognized is, for example, a part that serves as an anatomical landmark. As an example, in the present embodiment, the parts that serve as landmarks of the stomach, such as the cardia, fornix, greater curvature of the body of the stomach, gastric angle, vestibule, and pylorus, are recognized. In the case where the part cannot be recognized from the input image, the part recognition unit 113C outputs "unrecognizable" as the recognition result. In addition, in the case where the recognized part is a part other than the part set as the recognition object, the part recognition unit 113C outputs "other".

[0106] The part recognition unit 113C is composed of a learned model trained to recognize the part of an organ from an image. In the present embodiment, it is composed of a learned model trained to recognize the part of the stomach from an image. As an example, in the present embodiment, the part recognition unit 113C is composed of a CNN.

[0107] As described above, in the present embodiment, the part recognition process is performed on the captured still image. Therefore, the captured still image is input to the part recognition unit 113C.

[0108] The observation status determination unit 113D determines the observation status of the organ by the endoscope 10 based on the recognition result of the organ by the part recognition unit 113C. Specifically, the observation status is determined in the following order. When the still image is photographed and the part photographed by the part recognition unit 113C is recognized, the information of the recognized part (the information of the recognition result of the part) is added to the observation status determination unit 113D. The observation status determination unit 113D obtains the information of the recognized part and compares it with the information of the part recognized last time. In the case where the part recognized this time is different from the part recognized last time, the observation status determination unit 113D determines that the region of the organ between the part recognized last time and the part recognized this time is observed. For example, in the case where the part recognized last time (the part recognized based on the image obtained by the still image photography last time) is the "greater curvature of the gastric body" and the part recognized this time (the part recognized based on the image obtained by the still image photography this time) is the "vestibule", it is determined that the region between the greater curvature of the gastric body and the vestibule is observed.

[0109] In this way, the observation situation determination unit 113D determines the region of the organ observed by comparison with the region recognized last time. Since the determination is made by comparison with the region recognized last time, the observation situation determination unit 113D stores at least the information of the region recognized last time.

[0110] In this embodiment, the part recognized last time is an example of the first region of interest (first part), and the part recognized this time is an example of the second region of interest (second part). Furthermore, the image in which the first region of interest is recognized (the image obtained by the previous still image shooting) is an example of the first image, and the image in which the second region of interest is recognized (the image obtained by the current still image shooting) is an example of the second image.

[0111] In addition, when a still image is taken again and the part is recognized, the part recognized this time (the second focus area) becomes the part recognized last time (the first focus area) due to its relationship with the newly recognized part. In this way, the observed areas are successively determined in comparison with the previous recognition result.

[0112] The recording control unit 114 performs processing to capture a still image in accordance with a still image shooting instruction and to record it in the auxiliary storage device 103. With respect to the still image, the image displayed on the display device 50 at the time of receiving the still image shooting instruction is recorded. Thus, the user can record a desired image as a still image during observation. The recording control unit 114 acquires an image of a frame displayed on the display device 50 in accordance with the still image shooting instruction and records it in the auxiliary storage device 103.

[0113] Furthermore, when the part recognition processing is performed in the part recognition unit 113C, the recording control unit 114 performs a process of associating the information of the recognition result (the information of the recognized part) with the still image and recording it in the auxiliary storage device 103. The method of establishing the association is not particularly limited. As long as the correspondence between the still image and the information of the recognition result of the part based on the still image can be recorded in a form that knows the correspondence. Therefore, for example, the correspondence between the two can also be managed by a management file generated separately. Furthermore, for example, the information of the recognition result of the part can be recorded as attached information (so-called metadata) of the still image.

[0114] When the observation status determination unit 113D determines the observation status, the recording control unit 114 performs processing to record the information of the determination result in the auxiliary storage device 103. The information of the determination result is recorded in chronological order. Thus, the history of the observed area can be recorded in chronological order.

[0115] The display control unit 115 controls the display of the display device 50. The display control unit 115 performs a process of sequentially displaying the images sequentially output from the processor device 30 on the display device 50. As described above, the images captured by the endoscope 10 in time series are sequentially output from the processor device 30 in time series. Therefore, the images captured by the endoscope 10 in time series are displayed in time series on the display device 50. In addition, the display control unit 115 performs a process of displaying the observation status of the organ by the endoscope 10 on the display device 50.

[0116] Figure 6 A diagram showing an example of a screen display of a display device. Figure 6 An example is shown in which the display device 50 is a so-called wide-screen monitor (a monitor with a horizontally long screen).

[0117] like Figure 6 As shown, a main display area A1 and a sub display area A2 are set on the screen 52. The main display area A1 and the sub display area A2 are set by dividing the screen 52 into two parts in the lateral direction (horizontal direction). Figure 6 The large area on the left is the main display area A1, and the small area on the right is the sub-display area A2.

[0118] The main display area A1 is an area that mainly displays images captured by the endoscope 10. An observation image display area A1a is set in the main display area A1, and an image Im captured by the endoscope 10 is displayed in real time in the observation image display area A1a. As an example, the observation image display area A1a is composed of areas cut out at the top and bottom of a circle.

[0119] In addition, when the lesion detection auxiliary function is turned on (ON), the detection result of the lesion is displayed superimposed on the image Im. The detection result of the lesion is displayed in the form of a frame (so-called boundary frame) B surrounding the detected lesion. In the lesion detection unit 113A, when the type of the lesion is determined, instead of the information indicating the position of the lesion, or in addition to the information indicating the position of the lesion, the type of the determined lesion is also displayed. The type of the lesion is displayed, for example, in the vicinity of the detected lesion. It is also possible to display the type of the lesion in the secondary display area A2.

[0120] In the main display area A1, a strip-shaped information display area A1b is set below the observation image display area A1a. The information display area A1b is used to display various information. For example, when the identification auxiliary function is turned on, the identification result is displayed in the information display area A1b. Figure 6 An example in which the identification result is "NEOPLASTIC" is shown.

[0121] The sub-display area A2 is an area for displaying various information related to the observation. For example, information Ip of the subject (patient) and still images Is taken during the observation are displayed. The still images Is are displayed, for example, in the order in which they are taken from the top to the bottom of the screen 52. When more still images than the number that can be displayed are taken, the image Im is eliminated in the old order to switch the display. Alternatively, the display size of each image is reduced to display all images.

[0122] Furthermore, when the observation status determination function is turned on, information indicating the observation status is displayed in the secondary display area A2. Figure 6 As shown in FIG. 1 , information indicating the observation status is displayed in a predetermined observation status display window W. The observation status display window W is composed of a rectangular display area and is displayed at a predetermined position in the sub-display area A2. Figure 6 2 shows an example in which the observation status display window W is displayed near the lower end of the auxiliary display area A2. In the observation status display window W, information on the part identified by the part identification unit 113C is displayed as information indicating the observation status. The information on the identified part is displayed in the predetermined part display frames Fl1, Fl2, ..., respectively. Whenever a new part is identified, a part display frame Fl1, Fl2, ... is added.

[0123] Figure 7 This is a diagram showing an example of transition of the display of the observation status display window.

[0124] Figure 7(A) shows an example of the initial display of the observation status display window W. That is, it shows an example of the display of the observation status display window W when the part recognition has not been performed once since the start of observation (when the still image is not taken). In this case, the observation status display window W is blank. That is, nothing is displayed.

[0125] Figure 7 (B) shows an example of the display of the observation status display window W when the part is recognized for the first time. That is, it shows an example of the display of the observation status display window W when a still image is photographed for the first time and a part is recognized from the photographed image. Figure 7 As shown in (B) of FIG. 1 , a part display frame F11 is displayed at a predetermined position (near the upper end) in the window, and information on the recognized part is displayed in the part display frame F11 . Figure 7 (B) shows an example in which the “greater curvature of the gastric body” is identified.

[0126] Figure 7 (C) shows an example of the display of the observation status display window W when a part is newly recognized. That is, it shows an example of the display of the observation status display window W when a still image is further taken and a part is recognized from the photographed image. Figure 7 As shown in (C) , a part display frame F12 is newly added below the previously displayed part display frame F11 , and information on the newly recognized part is displayed in the newly added part display frame F12 . Figure 7 (C) shows an example in which the "antrum" is newly identified.

[0127] And, if Figure 7 As shown in (C), when a new part is identified, the previously displayed part display frame Fl1 and the newly added part display frame Fl2 are connected by an arrow Ar1. The arrow Ar1 is displayed in the direction from the previously displayed part display frame Fl1 toward the newly added part display frame Fl2. Through this arrow Ar1, it is shown that the endoscope 10 moves from the part displayed in the part display frame Fl1 toward the part displayed in the part display frame Fl2. That is, it is shown that the area between the part displayed in the part display frame Fl1 and the part displayed in the part display frame Fl2 is observed. In addition, it is shown that the situation of observing from the part displayed in the part display frame Fl1 toward the part displayed in the part display frame Fl2 is observed. That is, the direction of movement of observation is shown. Figure 7 In the example shown in (C), the endoscope 10 is shown to move from the "greater curvature" of the gastric body toward the "antrum" and observe the area of ​​the stomach therebetween.

[0128] Figure 7 (D) shows an example of the display of the observation status display window W when the part is further identified. That is, it shows an example of the display of the observation status display window W when a still image is further taken and the part is identified from the photographed image. Figure 7 As shown in (D) of FIG. 1 , a part display frame F13 is newly added below the part display frame F12 previously displayed, and information on the newly recognized part is displayed in the newly added part display frame F13 . Figure 7 (D) shows an example in which the "gastric angle" is newly recognized.

[0129] And, if Figure 7 As shown in (D), when a new part is identified, the previously displayed part display frame Fl2 and the newly added part display frame Fl3 are connected by an arrow Ar2. The arrow Ar2 is displayed in the direction from the previously displayed part display frame Fl2 toward the newly added part display frame Fl3. Through this arrow Ar2, it is shown that the endoscope 10 moves from the part displayed in the part display frame Fl2 toward the part displayed in the part display frame Fl3. That is, it is shown that the area between the part displayed in the part display frame Fl2 and the part displayed in the part display frame Fl3 is observed. In addition, it is shown that the situation of observing from the part displayed in the part display frame Fl2 toward the part displayed in the part display frame Fl3 is observed. That is, the direction of movement of observation is shown. Figure 7 In the example shown in (D), the endoscope 10 is shown to move from the "antrum" toward the "gastric angle" and observe the area of ​​the stomach therebetween.

[0130] In this way, each time a new part is identified, part display frames F11, F12, ... are added to the observation status display window W. Then, the information of the newly identified part is displayed in the newly added part display frame. In addition, the previously displayed part display frame and the newly added part display frame are connected by an arrow. Thus, the observed part, the observed area, and the observation direction can be grasped based on the display of the observation status display window W.

[0131] In addition, Figure 6 and Figure 7In the example shown, a maximum of four part display frames can be displayed in the observation status display window W. When more than four parts are identified, the display size of the observation status display window W is enlarged. More specifically, it is enlarged by extending upward. Thus, the information of the newly identified parts can be displayed one by one. In addition, it is also possible to set a structure in which only the information of the most recently identified parts is displayed without changing the size of the observation status display window W. In other words, it is also possible to set a structure in which only the information of the most recently identified n parts is displayed without changing the display number n of the part display frames. Furthermore, it is also possible to set a structure in which the display range can be changed by scrolling without changing the size of the observation status display window W.

[0132] like Figure 6 As shown, the observation status display window W is displayed in the sub-display area A2 with priority over other displays. That is, when the display overlaps with other information, it is displayed at the top.

[0133] In the present embodiment, arrows Ar1, Ar2, ... connecting part display frames F11, F12, ... are an example of information indicating that a region between a first region of interest (first part) and a second region of interest (second part) is observed.

[0134] [Function of endoscope system]

[0135] As a basic operation of the endoscope system 1 , the image processing device 100 displays an image captured by the endoscope 10 on the display device 50 in real time.

[0136] In addition to this basic operation, the image processing device 100 also displays various auxiliary information on the display device 50. In the present embodiment, as auxiliary information, information on the detection result of the lesion part, information on the identification result of the lesion part, and information indicating the observation status are displayed. Each auxiliary information is displayed when the corresponding function is turned on. For example, when the detection auxiliary function of the lesion part is turned on, as the detection result of the lesion part, the detected lesion part is surrounded by a frame and displayed. And, when the identification auxiliary function is turned on, the identification result is displayed in the identification result display area A3. And, when the display function of the observation status is turned on, as the information indicating the observation status, the observation status display window W is displayed on the screen 52.

[0137] Furthermore, the image processing device 100 records the image of the frame displayed on the display device 50 as a still image according to the user's still image photography instruction. When the lesion detection auxiliary function is turned on, the information of the lesion detected from the still image is recorded in association with the still image taken. Furthermore, when the identification auxiliary function is turned on, the information of the identification result of the lesion is recorded in association with the still image taken. Furthermore, when the observation status display function is turned on, the information of the recognized part and the information of the identification result of the observation status are recorded in association with the still image taken.

[0138] The turning on / off of various auxiliary functions is performed, for example, on a setting screen. Alternatively, the turning on / off of various auxiliary functions may be performed by an operation unit of the processor device 30 or an operation unit of the endoscope 10 .

[0139] [Display of information indicating observation status]

[0140] Figure 8 This is a flowchart showing the procedure of processing when information indicating the observation status is displayed on the display device.

[0141] As described above, when the function of displaying the observation status is turned on, information indicating the observation status is displayed on the display device 50 .

[0142] When the observation status display function is turned on, the processor 101 of the image processing device 100 displays the observation status display window W on the screen 52 of the display device 50 (step S1). The observation status display window W is displayed at a predetermined position (sub-display area A2) of the screen 52 (refer to Figure 6 ). The initial observation status display window W is blank (refer to Figure 7 (A)).

[0143] When observation starts, the processor 101 determines whether a still image is taken (step S2). When a still image is taken, the still image is subjected to a part recognition process (step S3). That is, the part of the organ (the stomach in this embodiment) shown in the still image is recognized.

[0144] After the recognition process, the processor 101 determines whether a part is recognized (step S4). If a part is recognized, the processor 101 performs a process of determining the observation status (step S5). That is, it is determined whether there is a recognized part, and if there is a recognized part, it is determined that the area between the part recognized last time and the part recognized this time has been observed.

[0145] After that, the processor 101 updates the display of the observation status display window W based on the part recognition result and the observation status determination result (step S6). When the part is recognized for the first time, the part display frame is displayed in the observation status display window W, and the information of the recognized part is displayed in the part display frame (refer to Figure 7 (B)). If the identified part exists, the part display frame is added to the observation status display window W. Then, the information of the identified part is displayed in the newly added part display frame (refer to Figure 7 (C) and (D)). Furthermore, the part display frame of the part recognized last time and the part display frame of the part recognized this time are connected by an arrow, and information indicating the region where the organ between the part recognized last time and the part recognized this time is recognized is displayed (reference Figure 7 (C) and (D)).

[0146] After the display of the observation status display window W is updated, the processor 101 determines whether the observation is completed (step S7). In addition, when it is determined in step S2 that a still image is not captured, and when it is determined in step S4 that a part is not recognized, it is also determined whether the observation is completed. If it is determined that the observation is completed, the processing is terminated. If it is determined that the observation is not completed, it is determined whether a still image is captured (step S2).

[0147] In addition, when the display function of the observation status is turned off, the processing is also terminated. In this case, the observation status display window W is deleted from the screen 52.

[0148] As described above, according to the endoscope system 1 and the image processing device 100 of this embodiment, each time a still image is captured, the part of the organ shown in the captured still image is identified. Then, the information of the identified part is displayed on the display device 50. Thus, the observed part can be easily grasped.

[0149] Furthermore, the information of the part recognized last time and the information of the part recognized newly are connected and displayed by arrows, thereby making it easy to grasp the observed area in the organ to be observed.

[0150] [Second embodiment]

[0151] According to the endoscope system 1 of the first embodiment, the user can grasp the observed area based on the display of the observation status display window W. However, it is impossible to determine whether the area determined to be observed is correctly observed (photographed). If it is not observed correctly, it may not be detected with high accuracy when the lesion is automatically detected from the image. Therefore, it is more preferable to not only grasp the observed area, but also to confirm whether the area determined to be observed is correctly observed.

[0152] In this embodiment, an endoscope system having a function of evaluating an observation state is described. In addition, the basic structure of the system is substantially the same as the endoscope system 1 of the first embodiment described above. Therefore, only the main differences from the endoscope system 1 of the first embodiment described above, especially the differences in the image processing device 100, are described below.

[0153] [Image processing device]

[0154] The image processing device 100 of this embodiment also has a function of evaluating observation of an organ by the endoscope 10. The evaluation of observation is performed in the observation status determination unit 113D.

[0155] Fig. 9 This is a block diagram of main functions of the observation status determination unit of the image processing device.

[0156] like Fig. 9 As shown, the observation situation determination unit 113D has the functions of an observation area determination unit 113D1 , a photographic evaluation unit 113D2 , and an evaluation value calculation unit 113D3 . These functions are realized by the processor 101 executing a predetermined program.

[0157] The observation area determination unit 113D1 determines the observed area (observation area) based on the identification result of the part by the part identification unit 113C. Specifically, the observation area is determined in the following order. When a still image is taken and the part photographed by the part identification unit 113C is identified, the information of the identified part (the information of the identification result of the part) is added to the observation area determination unit 113D1. The observation area determination unit 113D1 obtains the information of the identified part and compares it with the information of the part identified last time. In the case where the part identified this time is different from the part identified last time, the observation area determination unit 113D1 determines that the area of ​​the organ between the part identified last time and the part identified this time is the observed area.

[0158] The photographic evaluation unit 113D2 evaluates the image captured by the endoscope 10. In particular, the image is evaluated from the perspective of image recognition. That is, as described above, in the endoscope system 1 of the present embodiment, automatic detection of the lesion is performed by image recognition, and therefore the captured image is evaluated from the perspective of image recognition. As an example, in the present embodiment, the image is evaluated based on the blur state and the shake state of the image. The blur state and the shake state of the image can be evaluated as the sharpness of the image, for example. Sharpness is one of the indicators that represent the clarity of the image. The photographic evaluation unit 113D2 calculates the sharpness of the image, and determines that the image whose calculated sharpness is below the threshold is an NG image (an image that produces blur or shake), and the image that exceeds the threshold is determined as an OK image (a clear image). The calculation of the sharpness can adopt a known method. In addition, the method of evaluating the image can adopt a known method of quantitatively evaluating the blur state and / or shake state of the image. In the present embodiment, the threshold value of the sharpness of the image is an example of the first criterion.

[0159] The evaluation value calculation unit 113D3 calculates the observation evaluation value (score) based on the evaluation result of the image. The observation evaluation value is an indicator that quantitatively represents the degree to which the area determined by the observation area determination unit 113D1 to have been observed is correctly observed. As an example, in the present embodiment, it is calculated as the proportion of OK images captured in the area determined by the observation area determination unit 113D1 to have been observed. Specifically, the proportion of OK images in the images captured in the area determined to have been observed is calculated. A dynamic image is captured in the area determined to have been observed. Therefore, with respect to the evaluation value, the proportion of OK images in the images of each frame constituting the dynamic image is calculated. For example, it is assumed that the total number of frames of the dynamic image captured in the area determined to have been observed is 100, of which the number of frames determined to be OK images is 82. In this case, the evaluation value becomes 82 [%].

[0160] The evaluation value may be calculated sequentially or calculated collectively after the observed area is determined. In the case of sequential calculation, the evaluation value is updated every time the evaluation result of the captured image is available.

[0161] Fig.10 This is a conceptual diagram of the determination of the photographic area and the calculation of the evaluation value performed by the observation condition determination unit.

[0162] exist Fig.10 In FIG. 1 , reference numeral I denotes an image I captured in time series by the endoscope 10 . This image corresponds to an image of each frame of a dynamic image captured by the endoscope 10 .

[0163] As described above, the observed area is determined based on the part recognized from the captured still image. That is, the area between the part recognized from the previously captured still image (first image) (first area of ​​interest) and the part recognized from the subsequently captured still image (second image) (second area of ​​interest) is determined as the observed area.

[0164] Therefore, the photography evaluation is performed based on the images captured between the previously captured still image (first image) and the subsequently captured still image (second image). The photography evaluation unit 113D2 evaluates the images captured between the previously captured still image (first image) and the subsequently captured still image (second image). The evaluation value calculation unit 113D3 calculates the ratio of OK images among the images captured between the previously captured still image (first image) and the subsequently captured still image (second image), thereby calculating the evaluation value.

[0165] Fig.11 It is a figure which shows an example of the display of the observation status display window. Fig.11 (A) to (C) show the change of the display of the observation status display window W over time.

[0166] Fig.11 (A) shows an example of the display of the observation status display window W when the part is first recognized. Fig.11 As shown in (A) of FIG. 1 , a part display frame F11 is displayed at a predetermined position (near the upper end) in the window, and information on the recognized part is displayed in the part display frame F11 . Fig.11 (A) shows an example in which the "greater curvature of the gastric body" is identified.

[0167] Fig.11 (B) shows an example of the display of the observation status display window W when a new part is identified. Fig.11 As shown in (B), a part display frame F12 is newly added below the previously displayed part display frame F11, and the information of the newly recognized part is displayed in the newly added part display frame F12. Fig.11 As shown in (B), the previously displayed part display frame Fl1 and the newly added part display frame Fl2 are connected by arrow Ar1. Fig.11 As shown in (B), an evaluation value display frame Sc1 is newly displayed between the previously displayed part display frame Fl1 and the newly added part display frame Fl2, and an evaluation value is displayed in the newly displayed evaluation value display frame Sc1. The evaluation value is an evaluation value calculated for the observation between the part displayed in the part display frame Fl1 and the part displayed in the part display frame Fl2. Fig.11(B) shows an example in which the evaluation value calculated for observation between the "greater curvature" and the "antrum" is 82 [%]. An evaluation value display frame Sc1 is displayed adjacent to the arrow Ar1.

[0168] Fig.11 (C) shows an example of the display of the observation status display window W when the part is further identified. Fig.11 As shown in (C), a part display frame F13 is newly added below the previously displayed part display frame F12, and the information of the newly recognized part is displayed in the newly added part display frame F13. Fig.11 As shown in (C), when a new part is recognized, the previously displayed part display frame F12 and the newly added part display frame F13 are connected by arrow Ar2. Fig.11 As shown in (C), an evaluation value display frame Sc2 is newly displayed between the previously displayed part display frame Fl2 and the newly added part display frame Fl3, and an evaluation value is displayed in the newly displayed evaluation value display frame Sc2. The evaluation value is an evaluation value calculated for the observation between the part displayed in the part display frame Fl2 and the part displayed in the part display frame Fl3. Fig.11 (C) shows an example in which the evaluation value calculated for observation between the "antrum" and the "gastric angle" is 99 [%].

[0169] In this way, the evaluation value display frames Sc1, Sc2, ... are displayed between the part display frames Fl1, Fl2, ... displayed in the observation status display window W, and the observation evaluation values ​​calculated between the parts are displayed in the evaluation value display frames Sc1, Sc2, .... Thus, the observation status (whether the images are correctly taken) between the parts displayed in the part display frames Fl1, Fl2, ... can be easily confirmed.

[0170] In the present embodiment, the evaluation values ​​displayed in the evaluation value display frames Sc1, Sc2, ... are another example of information indicating that the area between the first region of interest (first part) and the second region of interest (second part) is observed. Therefore, it is also possible to set a configuration in which only the evaluation values ​​are displayed without displaying the arrows Ar1, Ar2, ...

[0171] As described above, according to the endoscope system 1 and the image processing device 100 of the present embodiment, the evaluation of observation (photographing) is displayed for the observed region. This makes it possible to easily determine whether the observed region is correctly observed (photographed).

[0172] [Modifications]

[0173] [Modification of Display of Information Indicating Observation Status]

[0174] [1] Display modification example (1)

[0175] Fig.12 This is a diagram showing another example of display of information indicating the observation status.

[0176] Fig.12 An example is shown in which the observation situation is expressed using a schematic diagram of an organ to be observed.

[0177] like Fig.12 As shown, in the observation status display window W, a schematic diagram (organ drawing) Sh of the organ to be observed is displayed. Fig.12 An example is shown in which the stomach is the observation object. Therefore, in this case, a schematic diagram of the stomach is shown.

[0178] When a still image is captured and a part is identified from the captured still image, a prescribed mark M1, M2, ... is displayed on the schematic diagram Sh. The marks M1, M2, ... are displayed at positions corresponding to the identified parts. Therefore, for example, when the cardia of the stomach is identified, a mark is displayed at the position of the cardia on the schematic diagram. The part observed from the marks M1, M2, ... can be identified. Fig.12 An example in which circular marks M1, M2, ... are displayed is shown.

[0179] Furthermore, when the second and subsequent markers M2, M3, ... are displayed, arrows Ar1, Ar2, ... connecting the markers to each other are displayed on the schematic diagram Sh in the order in which they are displayed. Arrows Ar1, Ar2, ... are displayed from the marker displayed previously toward the marker displayed later. Therefore, for example, when the markers are displayed in the order of marker M1, marker M2, and marker M3, arrow Ar1 connecting marker M1 and marker M2 is displayed from marker M1 toward marker M2. Furthermore, arrow Ar2 connecting marker M2 and marker M3 is displayed from marker M2 toward marker M3.

[0180] By displaying arrows Ar1, Ar2, ..., it is shown that the areas between the markers M1, M2, ... connected by the arrows Ar1, Ar2, ... are observed. Therefore, in this example, the arrows Ar1, Ar2, ... are an example of information indicating that the area between the first region of interest (first area) and the second region of interest (second area) is observed.

[0181] in addition, Fig.12An example is shown in which the greater curvature of the gastric body, the vestibule, and the gastric angle are identified in the order of the greater curvature of the gastric body, the vestibule, and the gastric angle. A mark M1 is displayed on the schematic diagram to indicate the position of the greater curvature of the gastric body. A mark M2 is displayed on the schematic diagram to indicate the position of the vestibule. A mark M3 is displayed on the schematic diagram to indicate the position of the gastric angle.

[0182] Fig.13 1 is a diagram showing an example of transition of the display of the observation status display window W.

[0183] Fig.13 (A) shows an example of the initial display of the observation status display window W. In this case, in the observation status display window W, only the schematic diagram Sh is displayed.

[0184] Fig.13 (B) shows an example of the display of the observation status display window W when the part is first recognized. Fig.13 As shown in (B) of FIG. 1 , a mark M1 is displayed at the position of the part corresponding to the recognized part. Fig.13 (B) shows an example in which the "greater curvature of the gastric body" is recognized. In this case, a mark M1 is displayed at a position corresponding to the greater curvature of the gastric body on the schematic diagram Sh.

[0185] Fig.13 (C) shows an example of the display of the observation status display window W when a new part is identified. Fig.13 As shown in (C) , a new mark M2 is displayed at the position of the part corresponding to the newly recognized part. Fig.13 (C) shows an example in which the "vestibular part" is newly recognized. In this case, a mark M2 is displayed at a position corresponding to the vestibule part on the schematic diagram Sh.

[0186] And, if Fig.13 As shown in (C), the previously displayed mark M1 and the newly displayed mark M2 are connected by an arrow Ar1. The arrow Ar1 is displayed from the previously displayed mark M1 toward the newly displayed mark M2. The arrow Ar1 shows that the endoscope 10 moves from the part indicated by the mark M1 to the part indicated by the mark M2. That is, it shows that the area between the part indicated by the mark M1 and the part indicated by the mark M2 is observed. Fig.13 In the example shown in (C), the endoscope 10 is shown moving from the "greater curvature of the gastric body" shown by the mark M1 toward the "vestibule" shown by the mark M2, and observing the area of ​​the stomach therebetween.

[0187] Fig.13 (D) shows an example of the display of the observation status display window W when the part is further identified. Fig.13 As shown in (D) of FIG. 1 , a new mark M3 is displayed at the position of the part corresponding to the newly recognized part. Fig.13 (D) shows an example in which a "stomach angle" is newly recognized. In this case, a mark M3 is displayed at a position corresponding to the stomach angle on the schematic diagram Sh.

[0188] And, if Fig.13 As shown in (D), the previously displayed mark M2 and the newly displayed mark M3 are connected by an arrow Ar2. The arrow Ar2 is displayed from the previously displayed mark M2 toward the newly displayed mark M3. The arrow Ar2 shows that the endoscope 10 moves from the part indicated by the mark M2 to the part indicated by the mark M3. That is, it shows that the area between the part indicated by the mark M2 and the part indicated by the mark M3 is observed. Fig.13 In the example shown in (D) of FIG. 1 , the endoscope 10 moves from the "vestibule" shown by the symbol M2 toward the "gastric angle" shown by the symbol M3 , and observes the stomach area therebetween.

[0189] In this way, whenever a part is recognized, a mark M1, M2, ... is displayed on the schematic diagram Sh corresponding to the recognized part. Thus, the observed part can be easily grasped. Furthermore, whenever a mark M1, M2, ... is displayed, an arrow Ar1, Ar2, ... connecting the previously displayed mark and the next displayed mark is displayed. Thus, the observed area and the observation direction can be easily grasped.

[0190] In this example, the marks are connected to each other by arrows, but they may be connected by line segments. By connecting with arrows, the observation direction (the direction to move the endoscope 10) can be confirmed.

[0191] Furthermore, it is also possible to set a structure in which arrows (including line segments) Ar1, Ar2, ... are displayed only for a certain time. For example, it is possible to set a structure in which arrows are displayed at the same time as the mark of the newly recognized part is displayed, and only the display of the arrows is eliminated after T time has passed since the start of the display of the arrows. The time T is a preset time.

[0192] In addition, when the display of arrows Ar1, Ar2, ... is eliminated after a certain period of time, it is preferably configured to be able to be redisplayed as needed. That is, it is preferred to display the arrows Ar 1, Ar2, ... displayed in the past again. For example, it can be configured to be redisplayed according to an instruction of redisplay (historical display) from the user. And, it can be configured to be automatically redisplayed after the observation is completed. Regarding the end of the observation, in addition to the user's instruction, it can also be configured to be automatically determined. When a specific part is identified from the image, it can be determined that the observation is over. For example, when the same part as the initially identified part is identified again, it can be determined that the observation is over. Or, when a specific action is detected, it can be determined that the observation is over. For example, when the action of pulling the endoscope out of the organ of the observation object is detected, it can be determined that the observation is over. The action of pulling the endoscope out of the organ of the observation object can be detected, for example, by detecting a specific part or organ from the image.

[0193] In addition, when the function of redisplaying the arrow is provided, the processor 101 stores the display history of the arrow in the main storage device 102 or the auxiliary storage device 103. When the arrow is redisplayed, the recorded information is referred to for redisplay.

[0194] In this example, the shapes of the marks M1 , M2 , . . . are circles, but the shapes of the marks are not particularly limited.

[0195] [2] Display modification example (2)

[0196] Fig.14 This is a diagram showing another example of display of information indicating the observation status. Fig.14 Shown in Fig.13 The display method shown is an example in which the evaluation value is also displayed.

[0197] like Fig.14 As shown, the evaluation value calculated in each area is displayed adjacent to the arrows Ar1, Ar2, ... representing each area. More specifically, the evaluation value display frames Sc1, Sc2, ... are displayed adjacent to the arrows Ar1, Ar2, ... representing the observed areas, and the evaluation values ​​are displayed in the evaluation value display frames Sc1, Sc2, ...

[0198] exist Fig.14 In the example shown, an example is shown in which the observation evaluation value of the stomach area between the part indicated by mark M1 (greater curvature of the gastric body) and the part indicated by mark M2 (vestibule) is 82[%], and the observation evaluation value of the stomach area between the part indicated by mark M2 (vestibule) and the part indicated by mark M3 (gastric angle) is 99[%].

[0199] In addition, in this example, the evaluation value is displayed together with the arrows Ar1, Ar2, ..., but it can also be set to a structure that only displays the evaluation value. In this case, for example, the evaluation value display frame Sc1, Sc2, ... is displayed between the marks representing each part, and the evaluation value is displayed in the evaluation value display frame Sc1, Sc2, ...

[0200] Furthermore, it is also possible to configure the arrow and / or evaluation value to be displayed only for a certain period of time. For example, it is possible to configure the arrow and / or evaluation value to be displayed only until a time T has passed since the arrow and / or evaluation value was displayed. The time T is a preset time.

[0201] Furthermore, when the display of the arrow and / or the evaluation value is deleted after a certain period of time has passed, it is preferable to enable the display to be redisplayed as necessary.

[0202] [3] Displayed Variations (3)

[0203] Fig.15 2 is a diagram showing another example of displaying information indicating observation conditions. Fig.15 An example of the display (initial display) of the observation status display window W before part recognition is shown.

[0204] This example is a display method that is particularly useful when a portion to be observed (or a portion to be captured with a still image) is determined in advance.

[0205] like Fig.15 As shown, the part to be observed (or the part of which a still image is to be captured) is displayed in advance on the schematic diagram Sh using predetermined marks Ma, Mb, . . . Fig.15 An example in which circles are used as the marks Ma, Mb, ... is shown. The shapes and colors of the marks Ma, Mb, ... are not particularly limited. Each mark Ma, Mb, ... is displayed at a position corresponding to a part determined as a part to be observed. Fig.15 An example in which the number of sites to be observed is five is shown. Fig.15 The five parts shown are the "cardia", "fornix", "greater curvature of the gastric body", "gastric angle" and "vestibule". The symbol Ma represents the "cardia", the symbol Mb represents the "fornix", the symbol Mc represents the "greater curvature of the gastric body", the symbol Md represents the "gastric angle", and the symbol Me represents the "vestibule".

[0206] In this example, the marks Ma, Mb, ... are examples of marks indicating regions of interest. Furthermore, the schematic diagram Sh in which the marks Ma, Mb, ... are arranged in a predetermined layout is an example of the first information. Furthermore, the part predetermined as the part to be observed (or the part to be captured with a still image) is an example of a plurality of specific regions of interest selected from a plurality of regions of interest in the body.

[0207] Fig.16 1 is a diagram showing an example of the display of the observation status display window after the part is recognized.

[0208] like Fig.16 As shown, when a still image is captured and a part is identified, the display mode of the mark of the identified part is switched. For example, the color (including transparency) of the mark of the identified part is changed. Fig.16 An example is shown in which the inside of the circle representing the mark is switched from transparent to colored or achromatic. Fig.16 An example is shown in which the part indicated by the mark Mc (greater curvature of the gastric body), the part indicated by the mark Md (gastric angle), and the part indicated by the mark Me (vestibular part) are recognized. In addition, in this example, the display mode of the mark before the corresponding part is recognized (the display mode before the color is changed) is an example of the first mode. And, the display mode of the mark after the corresponding part is recognized (the display mode after the color is changed) is an example of the second mode.

[0209] And, if Fig.16 As shown, when the second and subsequent parts are recognized, arrows Ar1, Ar2, ... connecting the mark of the previously recognized part and the part recognized later are displayed. Each arrow Ar1, Ar2, ... is displayed from the mark of the previously recognized part toward the mark of the part recognized later. Fig.16 An example is shown in which the position indicated by the mark Mc (greater curvature of the gastric body), the position indicated by the mark Me (vestibular part), and the position indicated by the mark Md (gastric angle) are identified in this order. In this case, the arrow Ar1 connecting the mark Mc and the mark Me is displayed from the mark Mc toward the mark Me. In addition, the arrow Ar2 connecting the mark Me and the mark Md is displayed from the mark Me toward the mark Md.

[0210] The arrows Ar1, Ar2, ... are displayed to show that the areas between the marks Ma, Mb, ... connected by the arrows Ar1, Ar2, ... are observed. The user can check the observation status through the arrows Ar1, Ar2, ...

[0211] In this example, arrows Ar1, Ar2, ... are an example of information indicating that an area between the first region of interest (first part) and the second region of interest (second part) is observed, and are an example of second information. Furthermore, arrows Ar1, Ar2, ... are an example of information associating markers Ma, Mb, ... (first region of interest and second region of interest) in the schematic diagram Sh (first information) in which markers Ma, Mb, ... are displayed.

[0212] Fig.17 1 is a diagram showing an example of transition of the display of the observation status display window W.

[0213] Fig.17 (A) shows an example of the initial display of the observation status display window W. Fig.17 As shown in (A), a schematic diagram Sh is displayed in the observation status display window W, and marks Ma, Mb, ... indicating the part to be observed are displayed on the schematic diagram Sh. Each mark Ma, Mb, ... is displayed corresponding to the position of the part to be observed. And each mark Ma, Mb, ... is displayed in a predetermined color (first mode). In this example, the inside of the circle indicating the mark is displayed transparently.

[0214] Fig.17 (B) shows an example of the display of the observation status display window W when the part to be observed is first recognized. Fig.17 As shown in (B), the display mode of the mark Mc of the recognized part is switched (switched to the second mode). In this example, the inside of the circle representing the mark is switched from transparent to color or achromatic. Fig.17 (B) shows an example in which the "greater curvature of the gastric body" is identified.

[0215] Fig.17 (C) shows an example of the display of the observation status display window W when a new part is identified from the parts to be observed. Fig.17 As shown in (C), the color (display mode) of the mark Me of the newly recognized part is switched. Fig.17 (C) shows an example in which the "vestibular part" is newly identified.

[0216] And, if Fig.17 As shown in (C), an arrow Ar1 is displayed connecting the mark Mc of the previously recognized part and the mark Me of the newly displayed part. The arrow Ar1 is displayed from the mark Mc of the previously recognized part toward the mark Me of the newly recognized part. The arrow Ar1 shows the region of the organ observed between the part indicated by the mark Mc and the part indicated by the mark Me. Fig.17 In the example shown in (C), the area of ​​the stomach between the "greater curvature of the gastric body" indicated by the mark Mc and the "vestibule" indicated by the mark Me is observed.

[0217] Fig.17 (D) shows an example of the display of the observation status display window W when a part is further identified from the parts to be observed. Fig.17 As shown in (D) of FIG. 1 , the color (display mode) of the mark Md of the newly identified part is switched. Fig.17 (D) shows an example in which the "stomach angle" is newly identified.

[0218] And, if Fig.17As shown in (D), an arrow Ar2 connecting the previously displayed mark Me and the newly displayed mark Md is displayed. The arrow Ar2 is displayed from the previously displayed mark Me toward the newly displayed mark Md. The arrow Ar2 shows that the area of ​​the stomach between the part indicated by the mark Me and the part indicated by the mark Md is observed. Fig.17 In the example shown in (D) of FIG. 1 , the endoscope 10 is shown to move from the "vestibule" shown by the mark Me toward the "gastric angle" shown by the mark Md, and the stomach area therebetween is observed.

[0219] In this way, when the part to be observed is indicated by the marks Ma, Mb, ... on the schematic diagram Sh and is observed (recognized), the display mode of the marks Ma, Mb, ... is switched. In this way, the observed part can be easily grasped. In addition, each time a part is recognized, arrows Ar1, Ar2, ... connecting the mark of the previously recognized part and the mark of the next recognized part are displayed. In this way, the observed area and the observation direction can be easily grasped.

[0220] In this example, the marks are connected to each other by arrows, but they may be connected by line segments. By connecting with arrows, the observation direction (the direction to move the endoscope 10) can be confirmed.

[0221] Furthermore, it is also possible to set a structure in which arrows (including line segments) Ar1, Ar2, ... are displayed only for a certain period of time. For example, it is possible to set a structure in which arrows are displayed while switching the display mode of the mark of the newly identified part, and only the display of the arrows is eliminated after T time has passed since the display of the arrows began. The time T is a pre-set time. In addition, when the display of the arrows is eliminated after a certain period of time has passed, it is preferably set to be able to be redisplayed as needed.

[0222] Furthermore, in this example, the shapes of the marks Ma, Mb, . . . are set to circles, but the shapes of the marks are not particularly limited.

[0223] Furthermore, in this example, when a corresponding part is identified, the color of the mark is changed, but the switching method is not limited to this. For example, when a corresponding part is identified, the shape of the mark may be changed, or the mark may be made to flash.

[0224] In addition, in this example, the case where the part to be observed is determined in advance is described as an example, but it is also possible to indicate with a mark the part that can be recognized by the image processing device 100. In this case, all the parts that can be recognized by the image processing device 100 are displayed on the schematic diagram.

[0225] [4] Displayed Variations (4)

[0226] Fig.18This is a diagram showing another example of display of information indicating the observation status. Fig.18 Shown in Fig.15 and Fig.16 The display method shown is an example in which the evaluation value is also displayed.

[0227] like Fig.18 As shown, the evaluation values ​​calculated in each area are displayed adjacent to the arrows Ar1, Ar2, ... representing each area. More specifically, evaluation value display frames Sc1, Sc2, ... are displayed adjacent to the arrows Ar1, Ar2, ... representing the observed areas, and the evaluation values ​​are displayed in the evaluation value display frames Sc1, Sc2, ...

[0228] exist Fig.18 In the example shown, an example is shown in which the observation evaluation value of the stomach area between the part indicated by mark Mc (greater curvature of gastric body) and the part indicated by mark Me (vestibule) is 82[%], and the observation evaluation value of the stomach area between the part indicated by mark Me (vestibule) and the part indicated by mark Md (gastric angle) is 99[%].

[0229] In addition, in this example, the evaluation value is displayed together with the arrows Ar1, Ar2, ..., but it can also be set to a structure that only displays the evaluation value. In this case, for example, the evaluation value display frame Sc1, Sc2, ... is displayed between the marks representing each part, and the evaluation value is displayed in the evaluation value display frame Sc1, Sc2, ...

[0230] Furthermore, it is also possible to set a structure in which the arrow and / or evaluation value is displayed only for a certain period of time. For example, it is possible to set a structure in which only the arrow display is eliminated after T time has passed since the arrow and / or evaluation value was displayed. The time T is a pre-set time. In addition, when the arrow and / or evaluation value display is eliminated after a certain period of time, it is preferably set to be able to be redisplayed as needed.

[0231] [5] Displayed Variations (5)

[0232] Fig.19 2 is a diagram showing another example of displaying information indicating observation conditions. Fig.19 An example of the display (initial display) of the observation status display window W before part recognition is shown.

[0233] This example is also a display method that is useful when a portion to be observed (or a portion to be captured with a still image) is determined in advance.

[0234] like Fig.19 As shown in FIG. 1 , marks Ma, Mb, ... indicating the parts to be observed (or the parts to be photographed) are displayed in advance in a predetermined layout in the observation status display window W. Fig.19 An example in which the number of sites to be observed is five is shown. Fig.19 The five parts shown are the "cardia", "fornix", "greater curvature", "gastric angle" and "antrum". The mark Ma represents the "cardia", the mark Mb represents the "fornix", the mark Mc represents the "greater curvature", the mark Md represents the "gastric angle", and the mark Me represents the "antrum". Fig.19 In the example shown, each mark Ma, Mb, ... is represented by a circle, and the name of the part corresponding to the inside of the circle is indicated. Fig.19 In the example shown, marks Ma, Mb, ... representing the respective parts are arranged at the positions of the vertices of the pentagon.

[0235] In this example, the marks Ma, Mb, ... are examples of marks indicating regions of interest. Furthermore, the schematic diagram Sh in which the marks Ma, Mb, ... are arranged in a predetermined layout is an example of the first information. Furthermore, the part predetermined as the part to be observed (or the part to be captured with a still image) is an example of a plurality of specific regions of interest selected from a plurality of regions of interest in the body.

[0236] Fig. 20 1 is a diagram showing an example of the display of the observation status display window after the part is recognized.

[0237] like Fig. 20 As shown, when a still image is captured and a part is identified, the display method of the mark of the identified part is switched. For example, the color of the mark of the identified part is inverted. Fig. 20 An example is shown in which the part indicated by the mark Mc (greater curvature of the gastric body), the part indicated by the mark Md (gastric angle), and the part indicated by the mark Me (vestibular part) are recognized. In addition, in this example, the display mode of the mark before the corresponding part is recognized (the display mode before the color is changed) is an example of the first mode. And, the display mode of the mark after the corresponding part is recognized (the display mode after the color is changed) is an example of the second mode.

[0238] And, if Fig. 20 As shown, when the second and subsequent parts are recognized, arrows Ar1, Ar2, ... connecting the mark of the previously recognized part and the part recognized later are displayed. Each arrow Ar1, Ar2, ... is displayed from the mark of the previously recognized part toward the mark of the part recognized later. Fig. 20An example is shown in which the position indicated by the mark Mc (greater curvature of the gastric body), the position indicated by the mark Me (vestibular part), and the position indicated by the mark Md (gastric angle) are identified in this order. In this case, the arrow Ar1 connecting the mark Mc and the mark Me is displayed from the mark Mc toward the mark Me. In addition, the arrow Ar2 connecting the mark Me and the mark Md is displayed from the mark Me toward the mark Md.

[0239] The arrows Ar1, Ar2, ... are displayed to show that the areas between the marks Ma, Mb, ... connected by the arrows Ar1, Ar2, ... are observed. The user can check the observation status through the arrows Ar1, Ar2, ...

[0240] And, if Fig. 20 As shown, when the evaluation value is calculated between the parts, the calculated evaluation value is displayed between the marks representing the parts. Specifically, the evaluation value display frames Sc1, Sc2, ... are displayed on the arrows Ar1, Ar2, ..., and the evaluation values ​​are displayed in the evaluation value display frames Sc1, Sc2, ...

[0241] exist Fig. 20 In the example shown, an example is shown in which the observation evaluation value of the stomach area between the part indicated by mark Mc (greater curvature of gastric body) and the part indicated by mark Me (vestibule) is 82[%], and the observation evaluation value of the stomach area between the part indicated by mark Me (vestibule) and the part indicated by mark Md (gastric angle) is 99[%].

[0242] In this example, the arrow and the evaluation value are examples of information indicating that the area between the first region of interest (first part) and the second region of interest (second part) is observed, and are examples of the second information. Furthermore, the arrow and the evaluation value are examples of information associating the markers (the first region of interest and the second region of interest) in the diagram (the first information) displaying the markers.

[0243] As in the above-mentioned modified examples (3) and (4), in this example, the observed part can be easily grasped by marking. Furthermore, by displaying arrows, the observed area and the observation direction can be easily grasped. Furthermore, by displaying the evaluation value, the observation status (photography status) of the area determined to have been observed can be grasped.

[0244] In this example, the marks are connected to each other by arrows, but they may be connected by line segments. By connecting with arrows, the observation direction (the direction to move the endoscope 10) can be confirmed.

[0245] Furthermore, in this example, the shapes of the marks Ma, Mb, . . . are set to circles, but the shapes of the marks are not particularly limited.

[0246] Fig.21 This is a diagram showing another example of marking. Fig.21 An example of using icons to display marks Ma, Mb, ... representing each part is shown. The icon is composed of a schematic diagram of the organ of the observation object, and is composed of a diagram showing a point at the position of the part shown by the icon. Fig.21 , the mark Ma represents the “cardia”, the mark Mb represents the “fornix”, the mark Mc represents the “greater curvature of the gastric body”, the mark Md represents the “gastric angle”, and the mark Me represents the “vestibule”.

[0247] Furthermore, in this example, when a corresponding part is identified, the color of the mark is changed, but the switching method is not limited to this. For example, when a corresponding part is identified, the shape of the mark may be changed, or the mark may be made to flash.

[0248] Furthermore, it is also possible to set a structure in which the arrow and / or evaluation value is displayed only for a certain period of time. For example, it is possible to set a structure in which only the arrow display is eliminated after T time has passed since the arrow and / or evaluation value was displayed. The time T is a pre-set time. In addition, when the arrow and / or evaluation value display is eliminated after a certain period of time, it is preferably set to be able to be redisplayed as needed.

[0249] Furthermore, in this example, although the case where the part to be observed is predetermined is described as an example, it is also possible to use a mark to represent the part that can be recognized by the image processing device 100. In this case, all the parts that can be recognized by the image processing device 100 are displayed (the marks of all the identifiable parts are displayed in a predetermined layout.).

[0250] [6] Other variations of the display

[0251] When the evaluation value is calculated between the identified parts, if the calculated evaluation value is less than a threshold value, the evaluation value may be displayed in a manner different from the usual manner. For example, the evaluation value may be displayed with emphasis.

[0252] Fig. 22 It is a diagram showing an example of emphasized display of evaluation values.

[0253] Fig. 22 An example is shown in which the observation evaluation value in the area between the part indicated by mark Me (vestibule) and the part indicated by mark Md (gastric angle) is below the threshold, and the observation evaluation value in the area between the part indicated by mark Mc (greater curvature of gastric body) and the part indicated by mark Me (vestibule) exceeds the threshold.

[0254] like Fig. 22 As shown, evaluation values ​​below the threshold are highlighted. Fig. 22An example is shown in which the display size of the evaluation value display frame is enlarged and the color display is changed compared to normal (when the threshold value is exceeded). In addition, the method of emphasizing the display is not limited to this. In addition, for example, the evaluation value display frame can be flashed, or the shape of the evaluation value display frame can be changed to emphasize the display. When changing the color, it is preferable to select a color that is easy to attract attention. For example, when the threshold value is exceeded, it is displayed in achromatic color (for example, gray), and when it is below the threshold value, it is displayed in color (for example, red).

[0255] In this way, by highlighting the calculated evaluation value when it is equal to or less than the threshold, it is possible to urge the user to re-watch.

[0256] In addition, in the case of emphasized display, the degree of emphasis may be changed stepwise according to the evaluation value. That is, the lower the evaluation value is, the more emphasized the display is.

[0257] Furthermore, in the above example, the case of emphasizing the evaluation value is described, but the arrow can also be emphasized (including the case of displaying it as a line segment) in the same way. For example, the thickness of the arrow line of the area where the evaluation value is below the threshold can be changed, or the color of the arrow can be changed, or the arrow can be flashed to emphasize the display.

[0258] Furthermore, the evaluation value can also be configured to be displayed only when it is below a threshold. That is, when the calculated evaluation value exceeds the threshold, it is assumed that it is correctly observed, so the evaluation value is not displayed separately, and only the arrow is displayed. On the other hand, when the calculated evaluation value is below the threshold, the possibility of incorrect observation is high, so the evaluation value is displayed on the screen to prompt a warning to the user. In addition, in this case, it can also be configured to not display the evaluation value but emphasize the display of the arrow to prompt a warning. Fig.23 FIG. 1 is a diagram showing an example of a case where an arrow is highlighted and displayed. Fig.23 The example shows that the observation evaluation value in the area between the part indicated by the mark Mc (greater curvature of the gastric body) and the part indicated by the mark Me (vestibule) is below the threshold, and the observation evaluation value in the area between the part indicated by the mark Me (vestibule) and the part indicated by the mark Md (gastric angle) exceeds the threshold. Fig.23 As shown, an arrow Ar1 indicating a region where the evaluation value is less than the threshold value is highlighted. Fig.23 An example is shown in which the thickness and color of the arrow line are changed to emphasize the display.

[0259] Furthermore, the highlighted display can also be performed using a marker. Fig.23In the example shown, when the observation evaluation value in the area between the part indicated by the mark Mc (the greater curvature of the gastric body) and the part indicated by the mark Me (the vestibule) is below the threshold, the mark Mc and the mark Me are highlighted. For example, the color of the mark Mc and the mark Me can be changed, or the size or shape can be changed, or the mark Mc and the mark Me can be highlighted by flashing.

[0260] [Re-watch]

[0261] When the region for which the evaluation value is calculated is observed again (determined to be the observed region), the evaluation value is preferably recalculated. Furthermore, when the evaluation value is recalculated, the already calculated evaluation value is preferably updated with the newly calculated evaluation value.

[0262] When the evaluation value is recalculated, for example, a structure can be set to receive the designation of the area for recalculating the evaluation value from the user and recalculate the evaluation value. In addition, a structure can be set to automatically recalculate the evaluation value when a specific operation is detected as a trigger. For example, when an already recognized part is recognized again, the evaluation value of the area determined as recognized with the part as the starting point is recalculated. The following describes the processing when the evaluation value is automatically recalculated when the already recognized part is recognized again.

[0263] Fig.24 This is a flowchart showing the procedure of processing when the evaluation value is automatically recalculated when the already recognized part is recognized again.

[0264] In addition, here, the case where the part to be observed (or the part to be captured with a still image) is indicated by a mark on the schematic diagram displayed in the observation status display window W in advance will be described as an example (refer to Figures 15 to 18 ). In this case, when a part is recognized, the display mode of the mark of the recognized part is switched. Also, an arrow is displayed between the mark of the previously recognized part, and the evaluation value is displayed adjacent to the arrow.

[0265] The processor 101 of the image processing device 100 displays the observation status display window W on the screen 52 of the display device 50 (step S11 ).

[0266] When observation starts, the processor 101 determines whether a still image is captured (step S12). When a still image is captured, the processor 101 performs part recognition processing on the captured still image (step S13).

[0267] After the recognition process, the processor 101 determines whether a part is recognized (step S14). If a part is recognized, the processor 101 determines whether the recognized part is a part that has been recognized (step S15). That is, it determines whether the part recognized this time is a part that has been recognized.

[0268] When the identified part is not an already identified part, the processor 101 performs a process of determining the observation status (step S16). That is, it is determined whether there is an already identified part. If there is an already identified part, it is determined that the area between the part identified last time and the part identified this time has been observed. Afterwards, the processor 101 updates the display of the observation status display window W based on the identification result of the part and the determination result of the observation status (step S17). When the part is identified for the first time, the display mode of the mark of the identified part is switched. When an already identified part exists, the display mode of the mark of the newly identified part is switched, and an arrow is displayed between the mark of the previously identified part. Moreover, the evaluation value is displayed adjacent to the arrow.

[0269] After the display of the observation status display window W is updated, the processor 101 determines whether the observation is completed (step S18). In addition, when it is determined in step S12 that a still image is not captured, and when it is determined in step S14 that a part is not recognized, it is also determined whether the observation is completed. If it is determined that the observation is completed, the processing is terminated. If it is determined that the observation is not completed, it is determined whether a still image is captured (step S12).

[0270] In step S15 , when it is determined that the recognized part is an already recognized part, the processor 101 performs processing to cancel the “already recognized” determination for the area determined to be recognized starting from the recognized part (step S19 ).

[0271] Fig.25 This is a conceptual diagram of the process of releasing the determination of “recognized”. Fig.25 An example is shown in which, after the various parts of the stomach are identified in the order of the part marked Mc (greater curvature of the gastric body), the part marked Me (vestibule), and the part marked Md (gastric angle), the part marked Me (vestibule) is identified again. In this case, the area between the part marked Mc (greater curvature of the gastric body) and the part marked Me (vestibule) and the area between the part marked Me (vestibule) and the part marked Md (gastric angle) are set as the identified area. When the part marked Me (vestibule) is identified after the part marked Md (gastric angle), since the part marked Me (vestibule) is an identified part, the "identified" judgment is released for the area determined to be identified with the part marked Me (vestibule) as the starting point. That is, the "identified" judgment is released for the area between the part marked Me (vestibule) and the part marked Md (gastric angle).

[0272] If the "recognized" determination is cancelled for the area determined to be recognized, the recognition determination is also cancelled for the portion defining the end point of the area. Fig.25In the example shown, the determination of "recognized" is cancelled for the site (gastric angle) marked with the mark Md.

[0273] After performing the process of canceling the determination of "recognized" for the corresponding area, the processor 101 updates the display of the observation status display window W (step S17). In this case, the arrow and the evaluation value are eliminated for the corresponding area. In the above example, the arrow and the evaluation value displayed in the area between the part marked Me (vestibular part) and the part marked Md (gastric angle part) are eliminated. In addition, the display of the mark Md is switched to the unrecognized display.

[0274] After the display of the observation status display window W is updated, the processor 101 determines whether the observation is completed (step S18). If it is determined that the observation is not completed, it determines whether a still image is taken (step S12). Then, when a still image is taken and a part is newly identified from the still image, the observation status determination process is performed between the newly identified part (step S16). For example, Fig.25 In the example shown, after the site marked Me (vestibule) is recognized again, if the site marked Md (gastric angle) is further recognized, the area between the site marked Me (vestibule) and the site marked Md (gastric angle) is determined to be the recognized area, and the evaluation value is calculated for the area. Then, based on the site recognition result and the observation status determination result, the display of the observation status display window W is updated (step S17).

[0275] As described above, according to this example, the process of recalculating the evaluation value can be easily performed by simply performing an operation of re-identifying the portion serving as the starting point for the region for which the evaluation value is calculated (the region determined to have been observed).

[0276] In addition, Fig.25In the example shown, when the parts are identified in the order of (1) the part marked Mc (greater curvature of the gastric body), (2) the part marked Me (vestibule), (3) the part marked Md (gastric angle), (4) the part marked Me (vestibule), and (5) the part marked Md (gastric angle), at the stage of identifying the part marked Me (vestibule), the area between the part marked Mc (greater curvature of the gastric body) and the part marked Me (vestibule) is determined to be the observed area. Thereafter, when the part marked Md (gastric angle) is identified, the area between the part marked Me (vestibule) and the part marked Md (gastric angle) is determined to be the observed area. Then, when the part marked Me (vestibule) is identified again, the "observed" determination is released for the area between the part marked Me (vestibule) and the part marked Md (gastric angle). Thereafter, when the site (gastric angle) marked with Md is recognized again, the area between the site (vestibule) marked with Me and the site (gastric angle) marked with Md is determined to be the observed area. In this example, the site (vestibule) marked with Me is an example of the first site, and the site (gastric angle) marked with Md is an example of the second site. Furthermore, the image (still image) in which the site (vestibule) marked with Me was initially recognized is an example of the first image, and the image (still image) in which the site (gastric angle) marked with Md was initially recognized is an example of the second image. Furthermore, the image (still image) in which the site (vestibule) marked with Me is recognized for the second time is an example of the third image, and the image (still image) in which the site (gastric angle) marked with Md is recognized for the second time is an example of the fourth image.

[0277] [Variations of the Observation Evaluation Method, etc.]

[0278] [1] Evaluation method

[0279] In the above embodiment, the image structure is evaluated based on the blur state and the shaking state of the image, but the method of evaluating the image is not limited to this. In addition to the blur state and / or the shaking state of the image, or instead of them, the image structure may be evaluated from the perspective of the brightness (exposure) of the image.

[0280] Furthermore, the evaluation criteria for observation can be changed for each region. For example, when observing the stomach, the evaluation criteria for observing the region between the greater curvature of the gastric body and the vestibule and the evaluation criteria for observing the region between the vestibule and the gastric angle can be changed. Thus, an evaluation criteria suitable for observation of each region can be set, so that each region can be appropriately evaluated.

[0281] Furthermore, the evaluation of the observation can also be performed from the perspective of the time or speed of the observed area. For example, the time of observing the area can be measured, and the case where the measured time exceeds a preset reference time is judged as OK, and the case where it is less than the reference time is judged as NG. Alternatively, the speed of the observed area can be measured, and the case where the measured speed is less than a preset reference speed is judged as OK, and the case where it exceeds the reference speed is judged as NG. It is more preferable to set a reference time and a reference speed for each area. The evaluation of OK and NG based on the time or speed of observing the area is another example of an evaluation value.

[0282] In addition, since the images are captured at a predetermined frame rate, measuring the time or speed of observing the region is substantially the same as measuring the number of images captured in the region. That is, since the time can be calculated from the number of images, and the speed can be calculated from the time, the observation can be evaluated based on the number of images. For example, if the number of images captured in the region exceeds a preset reference number, it can be evaluated as "OK", and if the number is less than the reference number, it can be evaluated as "NG". It is more preferable to set the reference number for each region.

[0283] When the observation is evaluated from multiple viewpoints, each evaluation value can be displayed separately. In addition, when the observation is evaluated from multiple viewpoints, it is also possible to set a structure that further calculates and displays the evaluation value that integrates the evaluation from each viewpoint. Regarding the comprehensive evaluation, for example, in all items (viewpoints), the case where the evaluation value exceeds the benchmark or meets the benchmark can be evaluated as "OK", and the case where the evaluation value is below the benchmark or does not meet the benchmark can be evaluated as "NG".

[0284] [2] Evaluation object

[0285] In the above embodiment, all frames of images captured between the parts are evaluated, but specific images may be evaluated, for example, images extracted at a constant frame interval may be evaluated.

[0286] [3] Modification of the method for calculating the evaluation value

[0287] (1) Modification 1 of the method for calculating the evaluation value

[0288] In the above embodiment, the ratio of OK images is calculated as the evaluation value, but the ratio of NG images may also be calculated as the evaluation value. When the ratio of NG images is calculated as the evaluation value, the lower the evaluation value, the higher the evaluation value.

[0289] Furthermore, when calculating the ratio of OK images or NG images as the evaluation value, when there are images similar to each other (including the same image) in the image group (a plurality of images captured in the area) as the evaluation object, it is preferred to exclude one of them and calculate the evaluation value. For example, when an image similar to an image captured previously is captured, the image captured later is excluded from the calculation object of the evaluation value.

[0290] Fig.26 This is a block diagram of an observation status determination unit having a function of calculating an evaluation value by excluding similar images.

[0291] like Fig.26 As shown in FIG. 1 , in addition to the functions of the observation area determination unit 113D1, the photographic evaluation unit 113D2, and the evaluation value calculation unit 113D3, the observation condition determination unit 113D of this example also has the function of the similar image detection unit 113D4. The functions of the observation area determination unit 113D1, the photographic evaluation unit 113D2, and the evaluation value calculation unit 113D3 are the same as those of the observation condition determination unit 113D of the image processing device 100 of the above-mentioned embodiment. Therefore, here, the function of the similar image detection unit 113D4 is described.

[0292] The similar image detection unit 113D4 acquires images taken by the endoscope 10 in a time series and detects images that are similar to each other. Specifically, images similar to previously taken images (images that have been taken) are detected. Regarding the detection method, known techniques can be used. For example, a method of detecting similar images using the correlation of images can be used. The detection object is a group of images taken in an area determined to have been observed. Therefore, each time a part is identified, the detection object is reset.

[0293] The similar image detection unit 113D4 processes the acquired images sequentially and adds only the already captured images and dissimilar images to the photographic evaluation unit 113D2. Thus, images similar to the already captured images can be excluded from the evaluation targets (evaluation value calculation targets).

[0294] Similar images are continuously captured when the movement of the endoscope 10 is stopped, etc. As shown in this example, by excluding images similar to the already captured images from the evaluation targets, the observation of the corresponding area can be evaluated more accurately.

[0295] Furthermore, when calculating the evaluation value, an image similar to an already captured image may be excluded from the calculation target of the evaluation value.

[0296] (2) Modification 2 of the method for calculating the evaluation value

[0297] It is also possible to adopt a configuration in which the evaluation value is calculated in real time. In this case, after the part is identified, the evaluation value is calculated sequentially.

[0298] Fig. 27 This is a conceptual diagram of when the evaluation value is calculated in real time.

[0299] exist Fig. 27 In FIG. 1 , the symbol 10K indicates an image evaluated as an OK image, and the symbol ING indicates an image evaluated as an NG image.

[0300] and, Fig. 27 The state in which the 20th frame image is captured after the part is recognized is shown. The evaluation value S is calculated as the ratio of the cumulative number Nok of OK images captured after the part is recognized to the cumulative number Nall of images captured after the part is recognized (S = Nok / Nall). Therefore, at the stage of capturing the 20th frame image, when the number of NG images is 2, the evaluation value S becomes S = 18 / 20 = 90 [%]. In this way, the evaluation value can be calculated in real time.

[0301] When the evaluation value is calculated in real time, it is preferable to display the calculation result on the display device 50 in real time.

[0302] Fig.28 This is a diagram showing an example of displaying evaluation values ​​in real time.

[0303] Fig.28 FIG. 2 shows an example of displaying the evaluation value calculated in real time in the observation status display window W. Fig.28 As shown in FIG. 1 , an evaluation value display area SR is set in the observation status display window W, and the evaluation value is displayed in the evaluation value display area SR. The evaluation value display area SR is set as a blank area when the schematic diagram Sh is displayed. Fig.28 An example in which a rectangular evaluation value display area SR is set at the upper right corner in the observation status display window W is shown.

[0304] and, Fig.28 An example is shown in which the part (vestibule) indicated by the mark Me is identified after the part (greater curvature of the gastric body) indicated by the mark Mc. In this case, the area between the part (greater curvature of the gastric body) indicated by the mark Mc and the part (vestibule) indicated by the mark Me becomes the observed area. Then, the evaluation value based on the image observed after the part (vestibule) indicated by the mark Me is identified is displayed in the evaluation value display area SR. Since it is not determined which area the image observed after the part (vestibule) indicated by the mark Me is observed, the evaluation value is displayed in the evaluation value display area SR. At the stage where the observed area is determined, the evaluation value is displayed near the arrow representing the area. The evaluation value displayed at this time becomes the determined evaluation value.

[0305] Fig.29This is a diagram showing an example of transition of the display of the observation status display window. Fig.29 An example is shown in which the part to be observed is determined in advance.

[0306] Fig.29 (A) shows an example of the initial display of the observation status display window W. Fig.29 As shown in (A) of FIG. 1 , a schematic diagram Sh is displayed in the observation status display window W, and marks Ma, Mb, . . . indicating the parts to be observed are displayed on the schematic diagram Sh.

[0307] Fig.29 (B) shows an example of the display of the observation status display window W when the part to be observed is first recognized. Fig.29 As shown in (B), the display mode of the mark Mc of the identified part is switched. Fig.29 (B) shows an example when the "greater curvature of the gastric body" is identified. Fig.29 As shown in (B) of FIG. 1 , when a part is recognized, an evaluation value calculated based on an image captured after recognition is displayed in real time in the evaluation value display area SR. Fig.29 (B) shows an example in which the evaluation value calculated based on the image captured after the site (greater curvature of the gastric body) indicated by the mark Mc is recognized is 100[%] at the display time.

[0308] Fig.29 (C) shows an example of the display of the observation status display window W when a new part is identified from the parts to be observed. Fig.29 As shown in (C), the color (display mode) of the mark Me of the newly recognized part is switched. Fig.29 (C) shows an example in which the "vestibular part" is newly identified.

[0309] And, if Fig.29 As shown in (C), an arrow Ar1 connecting the mark Mc of the previously recognized part and the mark Me of the newly displayed part is displayed. And, an evaluation value display frame Sc1 is displayed adjacent to the arrow Ar1, and the determined evaluation value is displayed. Fig.29 In the example shown in (C), the area of ​​the stomach between the "greater curvature of the gastric body" indicated by the mark Mc and the "vestibule" indicated by the mark Me is observed.

[0310] Moreover, if Fig.29 As shown in (C) of FIG. 1 , when a part is recognized, an evaluation value calculated based on an image captured after recognition is displayed in real time in the evaluation value display area SR. Fig.29 (C) shows an example in which the evaluation value calculated from the image captured after the part (vestibular part) indicated by the mark Me is recognized is 98[%] at the display time.

[0311] Fig.29 (D) shows an example of the display of the observation status display window W when a part is further identified from the parts to be observed. Fig.29 As shown in (D) of FIG. 1 , the color (display mode) of the mark Md of the newly identified part is switched. Fig.29 (D) shows an example in which the "stomach angle" is newly identified.

[0312] And, if Fig.29 As shown in (D), an arrow Ar2 connecting the previously displayed mark Me and the newly displayed mark Md is displayed. And, an evaluation value display frame Sc2 is displayed adjacent to the arrow Ar2, and the determined evaluation value is displayed. Fig.29 In the example shown in (D) of FIG. 1 , the endoscope 10 is shown to move from the "vestibule" shown by the mark Me toward the "gastric angle" shown by the mark Md, and the stomach area therebetween is observed.

[0313] Moreover, if Fig.29 As shown in (D) of FIG. 1 , when a part is recognized, an evaluation value calculated based on an image captured after recognition is displayed in real time in the evaluation value display region SR. Fig.29 (D) shows an example in which the evaluation value calculated from the image captured after the site (gastric angle) indicated by the mark Md is recognized is 95[%] at the display time.

[0314] In this way, the evaluation value calculated based on the image captured after the part is recognized is displayed in real time in the evaluation value display area SR. This allows the user to understand the observation status in real time.

[0315] In addition, Fig. 27 and Fig.28 In the example shown, the case where the evaluation value calculated in real time is displayed in the observation status display window W is used as an example for explanation, but the position where the evaluation value calculated in real time is displayed is not limited to this. It can also be displayed in a position other than the observation status display window W. Moreover, even when it is displayed in the observation status display window W, its display position is not particularly limited. For example, it can also be displayed near the mark representing the part that has just been identified (a position within the range where the degree of correlation with the part can be known).

[0316] Furthermore, as shown in this example, when the evaluation value is displayed in real time, it is preferable to be able to reset the calculation of the evaluation value as needed. For example, the reset can be performed when the same part as the part just recognized is recognized. In addition, it can also be set as a structure that can indicate the reset by operating a foot switch or the like.

[0317] [Part Identification]

[0318] [1] Image to be recognized

[0319] In the above-mentioned embodiment, a structure is set to recognize the part based on the image obtained by shooting a still image, but the image to be processed for part recognition is not limited to this. For example, a structure can be set to recognize the part of an image arbitrarily selected by the user. The selection of the image can be set as follows: the image is selected by a foot switch, etc., and the image displayed in the observation image display area A1a is recognized when the foot switch is pressed. In addition, a structure can also be set to select the image using a switch provided in the operating unit of the endoscope 10, a method such as voice input, etc.

[0320] Furthermore, the identification of the part can also be set to a structure that is performed on all the images taken (images of all frames). In this case, the processor 101 processes the images acquired in chronological order respectively, and identifies the parts projected in each image. By identifying the part from all the images taken, the part can be automatically identified. In this case, in the organ of the observed object, when another part is identified after one part is identified, it is determined that the area between the two is observed. For example, in the stomach, when the vestibule is identified after the greater curvature of the gastric body is identified, it is determined that the area between the greater curvature of the gastric body and the vestibule is observed. Thereafter, when the gastric angle is further identified, it is determined that the area between the vestibule and the gastric angle is observed. In this way, every time the part identified from the image is switched, it is determined that the area between the previously identified part is observed.

[0321] In addition, when it is possible to classify multiple parts of the organ of the observed object, it is not necessarily necessary to be able to identify all parts. It is sufficient as long as a specific part determined as the object to be identified can be identified. As the object to be identified (region of interest), in addition to the parts that serve as anatomical landmarks, parts that must be observed, parts that must be photographed, etc. are selected from the perspective of inspection, etc. In addition, in addition to parts that can be classified as organs or parts in the body, parts and regions of organs or parts in the body that become features can also be selected as identification objects. In addition, areas selected by the user (for example, areas that become landmarks), lesions, inflammation areas, etc. can also be set as identification objects. These areas can be set, for example, based on images taken in the past.

[0322] Furthermore, when automatically recognizing a part, it is not necessary to use all the captured images as the recognition processing object. For example, images acquired at a predetermined frame interval may be used as the recognition processing object.

[0323] [2] Identification Processing

[0324] When recognizing a part, the shooting state may be evaluated and the part recognition process may be performed. That is, it is also possible to evaluate whether the image to be processed was properly captured and perform the part recognition process. In this case, the part recognition is determined only when it is determined that the image was properly captured. Therefore, for example, when it is determined that the image was not properly captured even when the part can be recognized from the image, it is considered that the part is not recognized.

[0325] For example, it is determined whether the image is properly captured from the perspectives of image blur, image jitter, image brightness, quality of composition, image dirt (dirt reflected in the observation window), etc. Regarding image blur and jitter, an image without image blur and jitter (including a case where blur and jitter are within the allowable range) is determined to be a properly captured image. Regarding image brightness, an image captured with appropriate brightness (appropriate exposure) is determined to be a properly captured image. Regarding the quality of composition, for example, it is determined from the perspective of whether the part to be recognized is captured with a specified composition (for example, a structure arranged in the center of the screen). Regarding image dirt, for example, a fog-free image (a clear image) is determined to be a properly captured image.

[0326] It is possible to set a structure that comprehensively judges whether the image is properly captured from multiple viewpoints. In this case, the necessary conditions can be determined for each part. For example, it is possible to set a structure as follows: from the viewpoints of image blur, image jitter, image brightness, and image dirt, it is judged whether the image is properly captured for all parts, and on the other hand, the quality of the composition is also judged for a specific part.

[0327] For example, a structure can be set to use a determiner to determine whether an image is properly captured. The determiner can be composed of a learned model, for example. In the case of determining whether an image is properly captured from multiple viewpoints, a separate determiner is prepared. For example, in the case of determining whether an image is properly captured from the viewpoints of image blur, image jitter, image brightness, quality of composition, image dirtiness, etc., a determiner for determining whether the image is blurred, a determiner for determining whether the image is jittered, a determiner for determining whether the image brightness is appropriate, a determiner for determining the quality of composition, a determiner for determining whether the image is dirty, etc. are separately prepared.

[0328] The criterion determined to determine whether an image is properly captured is an example of the second criterion. Furthermore, the determination criterion determined from the viewpoints of image blur, image shake, image brightness, quality of composition, image dirt, etc. is an example of the content of the second criterion.

[0329] [Observation object]

[0330] In the above embodiment, the observation of the stomach is described as an example, but the organ to be observed is not particularly limited. In addition, organs other than internal organs can also be observed. That is, the present invention can be applied to the observation of the body using an endoscope.

[0331] Fig.30 This is a diagram showing an example of the display of the observation status display window when observing the large intestine.

[0332] Fig.30 An example is shown in which the part to be observed is predetermined. Fig.30 As shown in FIG. 1 , a schematic diagram Sh of the large intestine as an observation object is displayed in the observation status display window W, and predetermined marks MA, MB, ... are displayed on the schematic diagram Sh (in Fig.30 In the example shown, a circle is shown, thereby indicating the part of the large intestine that should be observed (or the part where a still image should be taken). Fig.30 An example in which the number of sites to be observed is four is shown. Fig.30 The four parts shown are "ileocecal", "hepatic flexure (right flexure of colon)", "splenic flexure (left flexure of colon)" and "anus". The hepatic flexure is the transition between the ascending colon and the transverse colon. The splenic flexure is the transition between the transverse colon and the descending colon. The mark MA represents the "ileocecal", the mark MB represents the "hepatic flexure", the mark MC represents the "splenic flexure", and the mark MD represents the "anus".

[0333] Fig.30 An example is shown in which the parts of the large intestine are identified in the order of the part indicated by the mark MA (ileocecal), the part indicated by the mark MB (hepatic flexure), and the part indicated by the mark MC (splenic flexure). In this case, the area (ascending colon) between the part indicated by the mark MA (ileocecal) and the part indicated by the mark MB (hepatic flexure) and the area (transverse colon) between the part indicated by the mark MB (hepatic flexure) and the part indicated by the mark MC (splenic flexure) are determined as the identified areas.

[0334] The mark of the identified part switches its display mode (for example, changes color). In addition, arrows Ar1, Ar2, ... and evaluation values ​​are displayed in the identified area. Arrows Ar1, Ar2, ... are displayed toward the observation direction. Evaluation values ​​are displayed in evaluation value display frames Sc1, Sc2, ... displayed adjacent to arrows Ar1, Ar2, ...

[0335] Furthermore, in the above-mentioned embodiment, the case of observation using a so-called soft endoscope has been described as an example, but the present invention can also be applied to the case of observation using a so-called hard endoscope.

[0336] [Record of the results of the observation status]

[0337] As described above, the results of the determination processing regarding the observation status can be recorded in association with information on still images and / or moving images captured during observation, information on lesions detected during observation, information on identification results performed during observation, and the like.

[0338] The recorded information may include: (1) information on the order of identifying each part; (2) information on the still image of each part when the part is identified by taking a still image; (3) information on the evaluation value calculated for each area (the area between the identified parts); (4) the shooting time and / or the number of frames in each area; and (5) information on the overall shooting time and / or the number of frames. The overall shooting time is the shooting time from the first part to the last part. The overall number of frames is the total number of frames from the first part to the last part.

[0339] Furthermore, the information on the determination result of the observation condition can be recorded in an external storage device in addition to or instead of the storage device (auxiliary storage device 103) provided in the image processing device 100. For example, the information on the result can be sent to a system (endoscope information management system, etc.) that manages the inspection result of the endoscope, and recorded in the system or a data server (endoscope inspection data server, etc.) connected to the system.

[0340] Furthermore, generally, when observation (including inspection) is performed using an endoscope, a report indicating the observation result is created. Therefore, when a determination process of the observation status is performed during the observation, it is preferable to also record (record) information on the result of the determination process in the report.

[0341] Furthermore, in recent years, a report is created using a device that assists in creating a report (a report creation assisting device). The report creation assisting device acquires information required for creating a report from an image processing device or the like, but the information acquired by the report creation assisting device can include information on the result of the determination of the observation condition. Thus, the information on the result of the determination of the observation condition can be included in the information recorded (recorded) in the report, and the information can be automatically input.

[0342] [Hardware Structure]

[0343] The functions of the endoscopic image processing device are realized by various processors. The various processors include general-purpose processors such as CPUs and / or GPUs (Graphic Processing Units) that execute programs and function as various processing units, processors such as FPGAs (Field Programmable Gate Arrays) whose circuit structures can be changed after manufacturing, such as programmable logic devices (PLDs), and ASICs (Application Specific Integrated Circuits) that have circuit structures specially designed to perform specific processing, such as dedicated circuits. The meaning of program is the same as that of software.

[0344] A processing unit can be composed of one of these various processors, or can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs, or a combination of a CPU and an FPGA. In addition, multiple processing units can also be composed of one processor. As an example of a plurality of processing units composed of one processor, first, there is the following method: as represented by a computer used for a client and a server, a processor is composed of a combination of one or more CPUs and software, and the processor functions as a plurality of processing units. Second, there is the following method: as represented by a system on chip (System on Chip: SoC), a processor that implements the functions of the entire system including multiple processing units by using an IC (Integrated Circuit: Integrated Circuit) chip is used. In this way, as a hardware structure, various processing units are composed of more than one of the above-mentioned various processors.

[0345] [other]

[0346] The various modifications can be used in combination as appropriate.

[0347] Explanation of symbols

[0348] 1- endoscope system, 10- endoscope, 20- light source device, 30- processor device, 31- endoscope control unit, 32- light source control unit, 33- image processing unit, 34- input control unit, 35- output control unit, 40- input device, 50- display device, 52- screen, 100- image processing device, 101- processor, 102- main storage device, 103- auxiliary storage device, 104- input and output interface, 111- image acquisition unit, 112- command acquisition unit, 113- image processing unit, 113A- lesion detection unit, 113B- identification unit, 113C- part recognition unit, 113D- observation status determination unit, 113D1- observation area determination unit, 113D2- photography evaluation unit, 113D3- evaluation value calculation unit, 113D4- similar image detection unit, 114- recording control unit, 115- display control unit, A1- main display display area, A1a-observation image display area, A1b-information display area, A2-sub-display area, A3-identification result display area, Ar1-arrow, Ar2-arrow, Fl1-part display box, Fl2-part display box, Fl3-part display box, I-image, IOK-image (OK image), ING-image (NG image), Im-image, Ip-subject information, Is-still image, M1, M2, ...-mark, Ma, Mb, ...-mark, MA, MB, ...-mark, SR-evaluation value display area, Sc1, Sc2, ...-evaluation value display box, Sh-schematic diagram, W-observation status display window, S1~S7-the order of processing when displaying information indicating the observation status on the display device, S11~S19-the order of processing when automatically recalculating the evaluation value when the identified part is recognized again.

Claims

1. An image processing device for processing a plurality of images captured in time series by an endoscope, the image processing device comprising a processor, The processor performs the following processing: acquiring the plurality of images; Processing the image to identify the region of interest reflected in the image from a plurality of regions of interest in the body; and When a first area of ​​interest among the multiple areas of interest is identified from a first image among the multiple images and a second area of ​​interest among the multiple areas of interest is identified from a second image that is chronologically later than the first image, information indicating that an area between the first area of ​​interest and the second area of ​​interest is observed is displayed on a display device.

2. The image processing device according to claim 1, wherein: The processor performs the following processing: displaying information of a plurality of specific regions of interest selected from the plurality of regions of interest in the body as first information on the display device; and Information indicating that a region between the first region of interest and the second region of interest is observed is displayed on the display device as second information.

3. The image processing device according to claim 2, wherein: The second information is composed of information associating the first region of interest with the second region of interest in the first information.

4. The image processing device according to claim 3, wherein: The second information includes information indicating the observed moving direction.

5. The image processing device according to claim 3, wherein: The first information is composed of information in which marks indicating the target area are arranged in a specific layout. The second information is composed of information associating the flag.

6. The image processing device according to claim 5, wherein: The first information includes a schematic diagram of an organ to be observed, and is composed of information in which the marker is arranged at a position of each region of interest on the schematic diagram.

7. The image processing device according to claim 6, wherein: The second information is composed of line segments connecting the marks.

8. The image processing device according to claim 7, wherein: The line segment includes an arrow indicating the direction of movement of the observation.

9. The image processing device according to any one of claims 5 to 8, wherein: The processor switches a display mode of the marker of the recognized area of ​​interest from a first mode to a second mode.

10. The image processing device according to any one of claims 2 to 8, wherein: the processor calculating an observation evaluation value based on the image chronologically between the first image and the second image, The second information includes information of the evaluation value.

11. The image processing apparatus according to claim 10, wherein: The evaluation value is calculated based on the number of images satisfying a first criterion and / or the number of images between the first image and the second image.

12. The image processing device according to claim 11, wherein: The first reference is set based on at least one of a blur state of an image, a shake state of an image, and brightness of an image.

13. The image processing apparatus according to claim 10, wherein: The evaluation value is calculated as the ratio of images satisfying a first criterion or the ratio of images not satisfying the first criterion.

14. The image processing apparatus according to claim 13, wherein: The processor performs the following processing: extracting images that are similar to each other from among the images between the first image and the second image; and The evaluation value is calculated excluding one of the extracted images that are similar to each other.

15. The image processing apparatus according to claim 10, wherein: The processor displays the second information in a manner corresponding to the evaluation value.

16. The image processing apparatus according to claim 10, wherein: The processor updates the evaluation value and updates display of the second information when the first region of interest is recognized from a third image that is chronologically later than the second image and the second region of interest is recognized from a fourth image that is chronologically later than the third image.

17. The image processing device according to any one of claims 1 to 8, wherein: The processor performs the following processing: displaying the plurality of images on the display device in chronological order; accepting selection of the first image from among the images displayed on the display device; and A selection of the second image from among the images displayed on the display device after the first image is accepted.

18. The image processing apparatus according to claim 17, wherein: The processor records the image selected as the first image and the second image as a still image.

19. The image processing device according to any one of claims 1 to 8, wherein: The processor processes the multiple images respectively to identify the region of interest reflected in each of the images.

20. The image processing device according to any one of claims 1 to 8, wherein: When the region of interest is recognized from the image, the processor further determines whether the image in which the region of interest is recognized satisfies a second criterion, and determines recognition of the region of interest when the second criterion is satisfied.

21. The image processing apparatus according to claim 20, wherein: The content of the second reference is set for each of the regions of interest.

22. The image processing device according to any one of claims 2 to 8, wherein: The processor ends display of the second information after a time T has elapsed since display of the second information was started.

23. The image processing apparatus according to claim 22, wherein: The processor performs the following processing: recording a history of display of the second information; and When the second information is redisplayed, the second information displayed in the past is simultaneously displayed based on the history.

24. The image processing apparatus according to claim 23, wherein: When observation is completed and / or when display of the history is instructed, the processor displays the second information displayed in the past based on the history.

25. The image processing device according to any one of claims 2 to 8, wherein: The processor ends display of the second information when the first region of interest is recognized from a third image that is chronologically subsequent to the second image.

26. The image processing device according to any one of claims 1 to 8, wherein: The plurality of regions of interest are a plurality of parts of an organ to be observed.

27. An endoscope system comprising: Endoscope; display device; and The image processing device according to any one of claims 1 to 8 processes the image captured by the endoscope.

Images