Image processing device and endoscope system

Abstract

Provided is an image processing device and an endoscope system with which it is possible to ascertain an observation status by using an endoscope. This image processing device processes a plurality of images captured in time series by an endoscope, and comprises a processor. The processor acquires a plurality of images. The processor processes the images and recognizes a region-of-interest appearing in the images from among a plurality of regions-of-interest inside a body. When a first region-of-interest among the plurality of regions-of-interest is recognized from a first image among the plurality of images, and a second region-of-interest among the plurality of regions-of-interest is recognized from a second image after the first image in chronological order, the processor causes a display device to display information indicating that a region between the first region-of-interest and the second region-of-interest has been observed.

Description

Image processing device and endoscope system

[0001] The present invention relates to an image processing device and an endoscope system, and more particularly to an image processing device that processes a plurality of images captured in time series by an endoscope, and an endoscope system equipped with the image processing device.

[0002] As a technology for supporting endoscopic observation, there is known a technology for automatically recognizing and notifying a region of interest such as a lesion from an image captured by an endoscope (for example, Patent Documents 1 and 2). In particular, in recent years, advances in artificial intelligence (AI) technology have made it possible to perform image recognition with high accuracy.

[0003] International Publication No. 2020 / 110214 International Publication No. 2020 / 152758

[0004] However, no matter how excellent the image recognition is, it is not possible to detect lesions or the like that exist in areas that are not photographed.

[0005] One embodiment of the technique of the present disclosure provides an image processing device and an endoscope system that can grasp the observation situation using an endoscope.

[0006] (1) An image processing device that processes multiple images captured in chronological order by an endoscope, the image processing device comprising a processor, which acquires the multiple images, processes the images, and recognizes a region of interest that appears in the image from multiple regions of interest within the body, and when a first region of interest from the multiple regions of interest is recognized from a first image of the multiple images, and a second region of interest from the multiple regions of interest is recognized from a second image that is chronologically later than the first image, the image processing device displays information on a display device indicating that a region between the first region of interest and the second region of interest has been observed.

[0007] (2) An image processing device according to (1), in which the processor displays information on a plurality of specific regions of interest selected from a plurality of regions of interest within the body as first information on the display device, and displays information indicating that a region between the first region of interest and the second region of interest has been observed as second information on the display device.

[0008] (3) The image processing device according to (2), wherein the second information is configured by information that associates the first region of interest with the second region of interest in the first information.

[0009] (4) An image processing device according to (2) or (3), wherein the second information includes information indicating the direction of movement of the observation.

[0010] (5) The image processing device of (3), wherein the first information is composed of information in which markers representing attention areas are arranged in a specific layout, and the second information is composed of information associating the markers.

[0011] (6) The image processing device of (5), wherein the first information includes a schematic diagram of the organ to be observed and is configured with information in which markers are placed at the positions of each region of interest on the schematic diagram.

[0012] (7) An image processing device according to (5) or (6), wherein the second information is composed of line segments connecting the signs.

[0013] (8) The image processing device of (7), wherein the line segment includes an arrow indicating the direction of movement of the observation.

[0014] (9) The image processing device according to any one of (5) to (8), wherein the processor switches the display form of the marker in the recognized area of ​​interest from the first form to the second form.

[0015] (10) An image processing device according to any one of (2) to (9), wherein the processor calculates an evaluation value of the observation based on an image between the first image and the second image in chronological order, and the second information includes information on the evaluation value.

[0016] (11) The image processing device of (10), wherein the evaluation value is calculated based on the number of images that satisfy the first criterion and / or the number of images between the first image and the second image.

[0017] (12) The image processing device according to (11), wherein the first criterion is set based on at least one of the blur state of the image, the blur state of the image, and the brightness of the image.

[0018] (13) The image processing device according to any one of (10) to (12), wherein the evaluation value is calculated as a percentage of images that satisfy the first criterion or a percentage of images that do not satisfy the first criterion.

[0019] (14) An image processing device according to (13), wherein the processor extracts mutually similar images from among the images between the first image and the second image, excludes one of the extracted mutually similar images, and calculates an evaluation value.

[0020] (15) The image processing device according to any one of (10) to (14), wherein the processor displays the second information in a form according to the evaluation value.

[0021] (16) An image processing device according to any one of (10) to (15), wherein the processor updates the evaluation value and updates the display of the second information when the first area of ​​interest is recognized from a third image that is chronologically later than the second image, and the second area of ​​interest is recognized from a fourth image that is chronologically later than the third image.

[0022] (17) An image processing device according to any one of (1) to (16), in which the processor displays a plurality of images on a display device in chronological order, accepts selection of a first image from the images displayed on the display device, and accepts selection of a second image from images displayed on the display device after the first image.

[0023] (18) The image processing device of (17), wherein the processor records the images selected as the first image and the second image as still images.

[0024] (19) An image processing device according to any one of (1) to (18), wherein the processor processes each of the multiple images and recognizes an area of ​​interest in each image.

[0025] (20) An image processing device according to any one of (1) to (19), in which, when the processor recognizes an area of ​​interest from an image, it further determines whether the image in which the area of ​​interest is recognized satisfies a second criterion, and if the second criterion is satisfied, it confirms the recognition of the area of ​​interest.

[0026] (21) The image processing device according to (20), in which the content of the second criterion is set for each region of interest.

[0027] (22) The image processing device according to any one of (2) to (8), wherein the processor ends the display of the second information after a time T has elapsed since the start of the display of the second information.

[0028] (23) The image processing device of (22), wherein the processor records the history of the display of the second information, and when new second information is displayed, the processor simultaneously displays the second information that was previously displayed based on the history.

[0029] (24) The image processing device of (23), wherein the processor displays second information previously displayed based on the history when the observation is completed and / or when an instruction to display the history is given.

[0030] (25) An image processing device according to any one of (2) to (8), wherein the processor terminates the display of the second information when the first attention area is recognized from a third image that is chronologically later than the second image.

[0031] (26) An image processing device according to any one of (1) to (25), wherein the plurality of regions of interest are a plurality of parts of an organ to be observed.

[0032] (27) An endoscope system comprising an endoscope, a display device, and any one of the image processing devices (1) to (26) that processes images captured by the endoscope.

[0033] According to the present invention, the observation situation using an endoscope can be grasped.

[0034] FIG. 1 is a block diagram showing an example of the system configuration of an endoscope system; FIG. 2 is a block diagram of the main functions of a processor device; FIG. 3 is a block diagram showing an example of the hardware configuration of an image processing device; FIG. 4 is a block diagram of the main functions of an image processing device; FIG. 5 is a block diagram of the main functions of an image processing unit; FIG. 6 is a diagram showing an example of a screen display of a display device; FIG. 7 is a diagram showing an example of the transition of the display of an observation status display window; FIG. 1 shows an example of a display in the status display window. FIG. 1 shows an example of a transition in the display of the observation status display window W. FIG. 2 shows another example of a display of information indicating the observation status. FIG. 3 shows another example of a display of information indicating the observation status. FIG. 4 shows an example of a display in the observation status display window after site recognition. FIG. 5 shows another example of a mark. FIG. 6 shows an example of highlighting an evaluation value. FIG. 7 shows an example of highlighting an arrow.

[0035] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0036] First Embodiment Here, an example will be described in which the present invention is applied to an endoscope system for observing (including observation for the purpose of examination) the upper digestive organs in the body, particularly the stomach. The stomach is an example of an organ to be observed, particularly a hollow organ.

[0037] [Configuration of Endoscope System] FIG. 1 is a block diagram showing an example of the system configuration of an endoscope system.

[0038] 1 , an endoscopic system 1 of the present embodiment includes an endoscope 10, a light source device 20, a processor device 30, an input device 40, a display device 50, and an image processing device 100. The endoscope 10 is connected to the light source device 20 and the processor device 30. The light source device 20, the input device 40, and the image processing device 100 are connected to the processor device 30. The display device 50 is connected to the image processing device 100.

[0039] The endoscope system 1 of this embodiment is configured as a system capable of observation using special light (special light observation) in addition to observation using normal white light (white light observation). Special light observation includes narrow band light observation. Narrow band light observation includes blue laser imaging (BLI observation), narrow band imaging (NBI observation), linked color imaging (LCI observation), etc. Note that special light observation itself is a known technique, and therefore detailed description thereof will be omitted.

[0040] [Endoscope] The endoscope 10 of this embodiment is an electronic endoscope (flexible endoscope), particularly an electronic endoscope for the upper digestive tract. The electronic endoscope includes an operation unit, an insertion unit, a connection unit, etc., and captures an image of a subject using an image sensor built into the tip of the insertion unit. The image sensor (e.g., a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, etc.) with a predetermined color filter array (e.g., a Bayer array) is used. The operation unit includes an angle knob, air / water supply buttons, a suction button, a mode switching button, a release button, a forceps port, etc. The mode switching button switches between observation modes, for example, white light observation mode, LCI observation mode, and BLI observation mode. The release button commands capturing a still image. Note that endoscopes themselves are well known, and detailed description thereof will be omitted. The endoscope 10 is connected to a light source device 20 and a processor device 30 via a connection unit.

[0041] [Light Source Device] The light source device 20 generates illumination light to be supplied to the endoscope 10. As described above, the endoscope system 1 of this embodiment is configured as a system capable of special light observation in addition to normal white light observation. For this reason, the light source device 20 has the function of generating light (e.g., narrowband light) compatible with special light observation in addition to normal white light. As described above, special light observation itself is a well-known technique, and therefore, a description of the generation of the illumination light will be omitted. The type of light source is switched, for example, by a mode switching button provided on the operation unit of the endoscope 10.

[0042] [Processor Device] The processor device 30 controls the overall operation of the endoscope system. The processor device 30 includes, as its hardware configuration, a processor, a main storage device, an auxiliary storage device, an input / output interface, and the like.

[0043] FIG. 2 is a block diagram showing the main functions of the processor device.

[0044] 2, the processor device 30 has functions such as an endoscope control unit 31, a light source control unit 32, an image processing unit 33, an input control unit 34, and an output control unit 35. Each function is realized by the processor executing a predetermined program. The auxiliary storage device stores various programs executed by the processor and various data required for control, etc.

[0045] The endoscope control unit 31 controls the endoscope 10. The control of the endoscope 10 includes drive control of the image sensor, control of air and water supply, and control of suction.

[0046] The light source control unit 32 controls the light source device 20. The control of the light source device 20 includes light emission control of the light source, control of switching between light source types, and the like.

[0047] The image processing unit 33 performs various signal processing on the signal (image signal) output from the image sensor of the endoscope 10 to generate an image.

[0048] The input control unit 34 performs processing to receive input of operations from the input device 40 and the operation unit of the endoscope 10, and input of various information.

[0049] The output control unit 35 controls the output of information from the processor device 30 to the image processing device 100. The information output from the processor device 30 to the image processing device 100 includes images captured by the endoscope (endoscopic images), information input via the input device 40, and various types of operation information. The various types of operation information include operation information of the input device 40 as well as operation information of the operation unit of the endoscope 10. The operation information includes an instruction to capture a still image. As described above, an instruction to capture a still image is given using the release button provided on the operation unit of the endoscope 10. Alternatively, an instruction to capture a still image can be given via a foot switch, a voice input device, a touch panel, or the like. Images captured in chronological order by the endoscope are output sequentially to the image processing device 100.

[0050] [Input Device] The input device 40, together with the display device 50, constitutes a user interface in the endoscope system 1. The input device 40 is composed of, for example, a keyboard, a mouse, a foot switch, etc. In addition, the input device 40 may be configured to include a touch panel, a voice input device, a gaze input device, etc.

[0051] [Display Device] The display device 50 is used to display endoscopic images as well as various types of information. The display device 50 is configured, for example, as a liquid crystal display (LCD), an organic electroluminescence display (OELD), or the like. Alternatively, the display device 50 can be configured as a projector, a head-mounted display, or the like.

[0052] [Image Processing Device] The image processing device 100 processes images output from the processor device 30. Specifically, the image processing device 100 performs a process of sequentially displaying images sequentially output from the processor device 30 on the display device 50. The image processing device 100 also sequentially processes images sequentially output from the processor device 30 and performs a process of detecting lesions from the images. The image processing device 100 also performs a process of differentiating the detected lesions. The image processing device 100 also performs a process of displaying the lesion detection results and differentiation results on the display device 50 in response to a user instruction. The image processing device 100 also performs a process of capturing and recording still images and / or videos in response to a user instruction. When a still image is captured, the image processing device 100 also performs a process of recognizing the organ part shown in the captured still image. The image processing device 100 also performs a process of displaying information indicating the observation status of the organ by the endoscope 10 on the display device 50 based on the recognition result of the organ part. In this embodiment, the multiple parts of the organ to be recognized are an example of multiple regions of interest within the body.

[0053] FIG. 3 is a block diagram showing an example of the hardware configuration of the image processing apparatus.

[0054] The image processing device 100 is configured as a so-called computer, and its hardware configuration includes a processor 101, a main memory 102, an auxiliary memory (storage) 103, an input / output interface 104, etc. The image processing device 100 is connected to the processor device 30 and the display device 50 via the input / output interface 104. The auxiliary memory 103 is configured, for example, with a flash memory including a hard disk drive (HDD) or a solid state drive (SSD). The auxiliary memory 103 stores programs executed by the processor 101 and various data required for control, etc. In addition, images (still images and videos) captured by the endoscope are recorded in the auxiliary memory 103. Lesion detection results, differentiation results, site recognition results, and observation status determination results are also recorded in the auxiliary memory 103.

[0055] FIG. 4 is a block diagram showing the main functions of the image processing apparatus.

[0056] 4, the image processing device 100 has functions such as an image acquisition unit 111, a command acquisition unit 112, an image processing unit 113, a recording control unit 114, and a display control unit 115. The functions of each unit are realized by the processor 101 executing a predetermined program (image processing program).

[0057] The image acquisition unit 111 performs processing to sequentially acquire images sequentially output from the processor device 30. As described above, the processor device 30 sequentially outputs images captured in chronological order by the endoscope 10. Therefore, the image acquisition unit 111 acquires the images captured in chronological order by the endoscope 10 in chronological order.

[0058] The command acquisition unit 112 acquires command information, which includes information on an instruction to capture a still image.

[0059] The image processing unit 113 processes the images acquired by the image acquisition unit 111. Fig. 5 shows blocks of the main functions of the image processing unit. As shown in Fig. 5, the image processing unit 113 of the image processing device 100 of this embodiment has functions such as a lesion detection unit 113A, a discrimination unit 113B, a site recognition unit 113C, and an observation situation determination unit 113D.

[0060] The lesion detection unit 113A processes the input image and performs processing to detect lesions (such as polyps) appearing in the image. Lesions include areas that are definitely lesions, as well as areas that may be lesions (such as benign tumors or dysplasias), and areas with features that may be directly or indirectly related to lesions (such as redness). The lesion detection unit 113A is configured with a trained model that has been trained to detect lesions from images. Since the detection of lesions using a trained model is a well-known technique, detailed description thereof will be omitted. As an example, the lesion detection unit 113A is configured with a model that uses a convolutional neural network (CNN). Note that the detection of lesions may include determining the type of lesion detected.

[0061] The classification unit 113B performs classification processing on the lesions detected by the lesion detection unit 113A. As an example, in this embodiment, classification processing is performed on lesions such as polyps detected by the lesion detection unit 113A as neoplastic (NEOPLASTIC) or non-neoplastic (HYPERPLASTIC). The classification unit 113B is configured with a trained model that has been trained to classify lesions from images.

[0062] The part recognition unit 113C processes the input image and performs processing to recognize the parts of the organs shown in the image. In this embodiment, since the stomach is the object of observation, the part recognition unit 113C performs processing to recognize the parts of the stomach. The parts to be recognized are, for example, parts that are considered to be anatomical landmarks. As an example, in this embodiment, parts that serve as landmarks of the stomach, such as the cardia, fornix, greater curvature of the gastric body, gastric angle, antrum, and pylorus, are recognized. If a part cannot be recognized from the input image, the part recognition unit 113C outputs "unrecognized" as the recognition result. Furthermore, if the recognized part is other than the parts set as the recognition target, the part recognition unit 113C outputs "other."

[0063] The part recognition unit 113C is configured with a trained model that has been trained to recognize organ parts from images. In this embodiment, the part recognition unit 113C is configured with a trained model that has been trained to recognize stomach parts from images. As an example, in this embodiment, the part recognition unit 113C is configured with a CNN.

[0064] As described above, in this embodiment, a captured still image is subjected to body part recognition processing, and therefore the captured still image is input to body part recognition unit 113C.

[0065] The observation status determination unit 113D determines the observation status of the organ by the endoscope 10 based on the recognition result of the organ part by the part recognition unit 113C. Specifically, the observation status is determined in the following procedure. When a still image is captured and the imaged part is recognized by the part recognition unit 113C, information on the recognized part (information on the part recognition result) is added to the observation status determination unit 113D. The observation status determination unit 113D acquires information on the recognized part and compares it with information on the part recognized previously. If the part recognized this time differs from the part recognized previously, the observation status determination unit 113D determines that the region of the organ between the part recognized previously and the part recognized this time has been observed. For example, if the part recognized previously (the part recognized from the image obtained by the previous still image capture) was the "greater curvature of the gastric body" and the part recognized this time (the part recognized from the image obtained by the current still image capture) is the "vestimus," the observation status determination unit 113D determines that the region between the greater curvature of the gastric body and the vestibule has been observed.

[0066] In this way, the observation situation determination unit 113D determines the region of the observed organ by comparing it with the previously recognized region. Since the determination is made by comparing it with the previously recognized region, the observation situation determination unit 113D holds at least information on the previously recognized region.

[0067] In this embodiment, the previously recognized region is an example of a first region of interest (first region), and the currently recognized region is an example of a second region of interest (second region). Also, the image in which the first region of interest is recognized (the image obtained by the previous still image capture) is an example of a first image, and the image in which the second region of interest is recognized (the image obtained by the current still image capture) is an example of a second image.

[0068] When a new still image is captured and the part is recognized, the currently recognized part (second area of ​​interest) becomes the previously recognized part (first area of ​​interest) in relation to the newly recognized part. In this way, successively observed areas are determined by comparison with the previous recognition result.

[0069] In response to an instruction to capture a still image, the recording control unit 114 performs processing to capture a still image and record it in the auxiliary storage device 103. The still image recorded is the image displayed on the display device 50 at the time the instruction to capture a still image is accepted. This allows the user to record a desired image as a still image during observation. In response to an instruction to capture a still image, the recording control unit 114 acquires an image of the frame displayed on the display device 50 and records it in the auxiliary storage device 103.

[0070] Furthermore, when the part recognition process is performed by the part recognition unit 113C, the recording control unit 114 performs a process of associating information on the recognition result (information on the recognized part) with a still image and recording the information in the auxiliary storage device 103. The association method is not particularly limited. It is sufficient that the information is recorded in a format that allows the correspondence between the still image and the part recognition result information based on the still image to be understood. Therefore, for example, the correspondence between the two may be managed using a separately generated management file. Furthermore, for example, the part recognition result information may be recorded as auxiliary information (so-called meta information) of the still image.

[0071] Furthermore, when the observation situation determination unit 113D determines the observation situation, the recording control unit 114 performs processing to record information on the determination result in the auxiliary storage device 103. The information on the determination result is recorded in chronological order. This allows the history of the observed areas to be recorded in chronological order.

[0072] The display control unit 115 controls the display of the display device 50. The display control unit 115 performs processing to cause the display device 50 to sequentially display images sequentially output from the processor device 30. As described above, the processor device 30 sequentially outputs images captured in time series by the endoscope 10 in chronological order. Therefore, the display device 50 displays the images captured in time series by the endoscope 10 in chronological order. In addition, the display control unit 115 performs processing to cause the display device 50 to display the observation status of an organ by the endoscope 10.

[0073] 6 is a diagram showing an example of a screen display of the display device, in which the display device 50 is a so-called wide monitor (a monitor with a horizontally long screen).

[0074] As shown in Fig. 6, a main display area A1 and a sub-display area A2 are set on the screen 52. The main display area A1 and the sub-display area A2 are set by dividing the screen 52 in half in the horizontal direction. The large area on the left side of Fig. 6 is the main display area A1, and the small area on the right side is the sub-display area A2.

[0075] The main display area A1 is an area that mainly displays images captured by the endoscope 10. An observation image display area A1a is set in the main display area A1, and an image Im captured by the endoscope 10 is displayed in real time in the observation image display area A1a. As an example, the observation image display area A1a is configured as an area shaped like a circle with the top and bottom cut out.

[0076] When the lesion detection support function is turned on, the lesion detection results are displayed superimposed on the image Im. The lesion detection results are displayed by surrounding the detected lesion with a frame (so-called bounding box) B. When the lesion detection unit 113A determines the type of lesion, information on the determined type of lesion is displayed instead of or in addition to information indicating the position of the lesion. The information on the type of lesion is displayed, for example, near the detected lesion. The information on the type of lesion can also be displayed in the secondary display area A2.

[0077] In the main display area A1, a strip-shaped information display area A1b is set below the observation image display area A1a. The information display area A1b is used to display various information. For example, when the differentiation support function is turned on, the differentiation result is displayed in the information display area A1b. Figure 6 shows an example where the differentiation result is "neoplastic (NEOPLASTIC)."

[0078] The sub-display area A2 is an area for displaying various information related to the observation. For example, information Ip about the subject (patient) and still images Is taken during the observation are displayed. The still images Is are displayed, for example, from top to bottom of the screen 52 in the order in which they were taken. If the number of still images taken exceeds the number that can be displayed, the oldest images Im are deleted and the display is switched. Alternatively, the display size of each image is reduced so that all images are displayed.

[0079] Furthermore, when the observation situation determination function is ON, information indicating the observation situation is displayed in the sub-display area A2. As shown in FIG. 6, information indicating the observation situation is displayed in a predetermined observation situation display window W. The observation situation display window W is configured as a rectangular display area, and is displayed at a predetermined position in the sub-display area A2. FIG. 6 shows an example in which the observation situation display window W is displayed near the bottom end of the sub-display area A2. The observation situation display window W displays information about the parts recognized by the part recognition unit 113C as information indicating the observation situation. The information about the recognized parts is displayed in the specified part display frames Fl1, Fl2, .... Part display frames Fl1, Fl2, ... are added each time a new part is recognized.

[0080] FIG. 7 is a diagram showing an example of the transition of the display of the observation status display window.

[0081] 7A shows an example of the initial display of the observation status display window W. That is, it shows an example of the display of the observation status display window W when no part has been recognized since the start of observation (when no still image has been taken). In this case, the observation status display window W is blank; that is, nothing is displayed.

[0082] Figure 7(B) shows an example of the display of the observation status display window W when a region is recognized for the first time. That is, it shows an example of the display of the observation status display window W when a still image is captured for the first time and a region is recognized from the captured image. As shown in the figure, a region display frame Fl1 is displayed at a predetermined position (near the top edge) within the window, and information about the recognized region is displayed in the region display frame Fl1. Figure 7(B) shows an example when the "greater curvature of the gastric body" is recognized.

[0083] 7C shows an example of the display of the observation status display window W when a new region is recognized. That is, it shows an example of the display of the observation status display window W when another still image is captured and a region is recognized from the captured image. As shown in the figure, a new region display frame Fl2 is added below the previously displayed region display frame Fl1, and information about the newly recognized region is displayed in the newly added region display frame Fl2. FIG. 7C shows an example when a new "vestibule (antrum)" is recognized.

[0084] Furthermore, as shown in FIG. 7C , when a new region is recognized, an arrow Ar1 connects the previously displayed region display frame Fl1 and the newly added region display frame Fl2. The arrow Ar1 is displayed in a direction from the previously displayed region display frame Fl1 to the newly added region display frame Fl2. This arrow Ar1 indicates that the endoscope 10 has moved from the region displayed in the region display frame Fl1 to the region displayed in the region display frame Fl2. That is, it indicates that the region between the region displayed in the region display frame Fl1 and the region displayed in the region display frame Fl2 has been observed. It also indicates that the region displayed in the region display frame Fl1 has been observed toward the region displayed in the region display frame Fl2. That is, it indicates the direction of movement of the observation. The example shown in FIG. 7C indicates that the endoscope 10 has moved from the "greater curvature of the gastric body" to the "antrum," and that the region of the stomach between them has been observed.

[0085] Figure 7(D) shows an example of the display of the observation status display window W when a further region is recognized. That is, it shows an example of the display of the observation status display window W when a further still image is captured and a region is recognized from the captured image. As shown in the figure, a new region display frame Fl3 is added below the previously displayed region display frame Fl2, and information about the newly recognized region is displayed in the newly added region display frame Fl3. Figure 7(D) shows an example when a new "gastric angle" is recognized.

[0086] Furthermore, as shown in FIG. 7(D), when a new site is recognized, an arrow Ar2 connects the previously displayed site display frame Fl2 and the newly added site display frame Fl3. The arrow Ar2 is displayed in a direction from the previously displayed site display frame Fl2 to the newly added site display frame Fl3. This arrow Ar2 indicates that the endoscope 10 has moved from the site displayed in the site display frame Fl2 to the site displayed in the site display frame Fl3. That is, it indicates that the region between the site displayed in the site display frame Fl2 and the site displayed in the site display frame Fl3 has been observed. It also indicates that the region from the site displayed in the site display frame Fl2 to the site displayed in the site display frame Fl3 has been observed. That is, it indicates the direction of movement of the observation. The example shown in FIG. 7(D) indicates that the endoscope 10 has moved from the "antrum" to the "gastric angle," and that the region of the stomach between them has been observed.

[0087] In this way, each time a new site is recognized, a site display frame Fl1, Fl2, ... is added and displayed in the observation status display window W. Information about the newly recognized site is then displayed in the newly added site display frame. Furthermore, an arrow connects the previously displayed site display frame with the newly added site display frame. This allows the user to grasp the observed sites, observed areas, and observation direction from the display in the observation status display window W.

[0088] In the examples shown in Figures 6 and 7, up to four region display frames can be displayed in the observation status display window W. If more than four regions are recognized, the display size of the observation status display window W is enlarged. More specifically, it is enlarged and extended upward. This allows information on newly recognized regions to be displayed sequentially. Alternatively, the observation status display window W can be configured to display only information on the most recently recognized region without changing its size. In other words, it can be configured to display only information on the most recently recognized n regions without changing the number n of region display frames displayed. Furthermore, the observation status display window W can be configured to be scrolled to change the display range without changing its size.

[0089] 6, the observation status display window W is displayed in the sub-display area A2 with priority over other displays. That is, when the display overlaps with other information, the observation status display window W is displayed at the top.

[0090] In this embodiment, the arrows Ar1, Ar2, ... connecting the area display frames Fl1, Fl2, ... are an example of information indicating that the area between the first area of ​​interest (first area) and the second area of ​​interest (second area) has been observed.

[0091] [Operation of Endoscope System] As a basic operation of the endoscope system 1, the image processing device 100 causes the display device 50 to display images captured by the endoscope 10 in real time.

[0092] In addition to this basic operation, the image processing device 100 displays various types of support information on the display device 50. In this embodiment, the support information displayed includes information on the lesion detection results, information on the lesion differentiation results, and information indicating the observation status. Each piece of support information is displayed when the corresponding function is turned on. For example, when the lesion detection support function is turned on, the detected lesion is displayed surrounded by a frame as the lesion detection result. Furthermore, when the differentiation support function is turned on, the differentiation result is displayed in the differentiation result display area A3. Furthermore, when the observation status display function is turned on, an observation status display window W is displayed on the screen 52 as information indicating the observation status.

[0093] Furthermore, the image processing device 100 records the image of the frame being displayed on the display device 50 as a still image in response to a user's instruction to capture a still image. When the lesion detection support function is ON, information on the lesion detected from the captured still image is recorded in association with the captured still image. Furthermore, when the differentiation support function is ON, information on the differentiation result of the lesion is recorded in association with the captured still image. Furthermore, when the observation status display function is ON, information on the recognized area and information on the observation status determination result are recorded in association with the captured still image.

[0094] The various support functions can be turned on and off, for example, on a setting screen. Alternatively, the functions can be turned on and off on the operation unit of the processor device 30 or the operation unit of the endoscope 10.

[0095] [Display of Information Indicating Observation Status] FIG. 8 is a flowchart showing a processing procedure when information indicating the observation status is displayed on the display device.

[0096] As described above, the information indicating the observation status is displayed on the display device 50 when the observation status display function is turned on.

[0097] When the observation status display function is turned on, the processor 101 of the image processing device 100 displays an observation status display window W on the screen 52 of the display device 50 (step S1). The observation status display window W is displayed at a predetermined position (sub-display area A2) on the screen 52 (see FIG. 6). The initial display of the observation status display window W is blank (see FIG. 7(A)).

[0098] When observation is started, the processor 101 determines whether or not a still image is to be captured (step S2). When a still image is captured, the captured still image is subjected to a part recognition process (step S3). That is, a process is performed to recognize the part of the organ (the stomach part in this embodiment) captured in the captured still image.

[0099] After the recognition process, the processor 101 determines whether or not a region has been recognized (step S4). If a region has been recognized, the processor 101 performs a process for determining the observation status (step S5). That is, the processor 101 determines whether or not there is a recognized region, and if there is a recognized region, it determines that the area between the previously recognized region and the currently recognized region has been observed.

[0100] Thereafter, the processor 101 updates the display in the observation status display window W based on the site recognition results and the observation status determination results (step S6). When a site is recognized for the first time, a site display frame is displayed in the observation status display window W, and information about the recognized site is displayed in that site display frame (see FIG. 7(B)). When a site has already been recognized, a site display frame is added to the observation status display window W and displayed. Information about the recognized site is then displayed in the newly added site display frame (see FIGS. 7(C) and (D)). In addition, an arrow connects the site display frame of the previously recognized site to the site display frame of the currently recognized site, and information indicating that the region of the organ between the previously recognized site and the currently recognized site is displayed (see FIGS. 7(C) and (D)).

[0101] After updating the display of the observation status display window W, the processor 101 determines whether the observation has ended (step S7). Note that if it is determined in step S2 that a still image has not been taken, or if it is determined in step S4 that the part has not been recognized, the processor 101 also determines whether the observation has ended. If it is determined that the observation has ended, the processing ends. If it is determined that the observation has not ended, the processor 101 determines whether a still image has been taken (step S2).

[0102] The process also ends when the observation status display function is turned off. In this case, the observation status display window W is removed from the screen 52.

[0103] As described above, according to the endoscope system 1 and the image processing device 100 of this embodiment, each time a still image is captured, the part of the organ shown in the captured still image is recognized. Information about the recognized part is then displayed on the display device 50. This makes it easy to understand which parts have already been observed.

[0104] Additionally, information on the previously recognized region and information on the newly recognized region are displayed connected by an arrow, making it easy to grasp which regions of the organ have already been observed.

[0105] [Second Embodiment] According to the endoscope system 1 of the first embodiment, the user can ascertain the observed areas from the display in the observation status display window W. However, it is not possible to determine whether an area determined to have been observed was correctly observed (photographed). If the area has not been observed correctly, there is a risk that the area will not be accurately detected when automatically detecting a lesion from an image. Therefore, it is more preferable not only to be able to ascertain the observed areas, but also to be able to confirm whether an area determined to have been observed was correctly observed.

[0106] In this embodiment, an endoscope system having a function for evaluating the observation state will be described. The basic configuration of the system is substantially the same as that of the endoscope system 1 of the first embodiment. Therefore, the following will only describe the main differences from the endoscope system 1 of the first embodiment, particularly the differences in the image processing device 100.

[0107] [Image Processing Device] The image processing device 100 of this embodiment further has a function of evaluating the observation of an organ by the endoscope 10. The evaluation of the observation is performed by the observation status determination unit 113D.

[0108] FIG. 9 is a block diagram of the main functions of the observation situation determination unit of the image processing device.

[0109] 9, the observation situation determination unit 113D has the functions of an observation area determination unit 113D1, an image capture evaluation unit 113D2, and an evaluation value calculation unit 113D3. These functions are realized by the processor 101 executing a predetermined program.

[0110] The observation area determination unit 113D1 determines the observed area (observation area) based on the area recognition result by the area recognition unit 113C. Specifically, the observation area is determined in the following procedure. When a still image is captured and the captured area is recognized by the area recognition unit 113C, information about the recognized area (information about the area recognition result) is added to the observation area determination unit 113D1. The observation area determination unit 113D1 obtains information about the recognized area and compares it with information about the area recognized last time. If the area recognized this time is different from the area recognized last time, the observation area determination unit 113D1 determines that the area of ​​the organ between the area recognized last time and the area recognized this time is the observed area.

[0111] The imaging evaluation unit 113D2 evaluates images captured by the endoscope 10. In particular, the images are evaluated from the perspective of image recognition. That is, as described above, the endoscopic system 1 of this embodiment performs automatic detection of lesions through image recognition, and therefore evaluates the captured images from the perspective of image recognition. As an example, in this embodiment, the images are evaluated based on the blur and blur of the image. The blur and blur of the image can be evaluated, for example, as image sharpness. Sharpness is one index that represents the clarity of the image. The imaging evaluation unit 113D2 calculates the sharpness of the image and determines an image with a calculated sharpness below a threshold as an NG image (an image with blur or blur), and an image with a calculated sharpness above the threshold as an OK image (a clear image). A known method can be used to calculate the sharpness. Alternatively, a known method for quantitatively evaluating the blur and / or blur of an image can be used to evaluate the image. In this embodiment, the threshold for image sharpness is an example of a first criterion.

[0112] The evaluation value calculation unit 113D3 calculates an evaluation value (score) of the observation based on the evaluation result of the image. The evaluation value of the observation is an index that quantitatively indicates how accurately the area determined to have been observed by the observation area determination unit 113D1 was observed. As an example, in this embodiment, the evaluation value is calculated as the percentage of OK images captured in the area determined to have been observed by the observation area determination unit 113D1. Specifically, the percentage of OK images among the images captured in the area determined to have been observed is calculated. A video is captured in the area determined to have been observed. Therefore, the evaluation value is calculated as the percentage of OK images among the images of each frame that constitutes the video. For example, suppose that the total number of frames of a video captured in the area determined to have been observed is 100, and 82 of those frames are determined to be OK images. In this case, the evaluation value is 82%.

[0113] The evaluation value may be calculated sequentially, or may be calculated all at once after the observation area is determined. In the case of sequential calculation, the evaluation value is updated each time an evaluation result of a captured image is obtained.

[0114] FIG. 10 is a conceptual diagram of the determination of the photographing area and the calculation of the evaluation value performed by the observation situation determination unit.

[0115] 10, reference symbol I denotes images I captured in time series by the endoscope 10. These images correspond to the images of each frame of a moving image captured by the endoscope 10.

[0116] As described above, the observed area is determined based on the part recognized from the captured still image. That is, the area between the part recognized from the first captured still image (first image) (first area of ​​interest) and the part recognized from the second captured still image (second image) (second area of ​​interest) is determined to be the observed area.

[0117] Therefore, the evaluation of the photography is performed based on the images taken between the first still image (first image) and the second still image (second image). The photography evaluation unit 113D2 evaluates each of the images taken between the first still image (first image) and the second still image (second image). The evaluation value calculation unit 113D3 calculates the proportion of OK images among the images taken between the first still image (first image) and the second still image (second image), thereby calculating the evaluation value.

[0118] 11A to 11C are diagrams showing an example of the display of the observation status display window W. (A) to (C) of Fig. 11 show changes over time in the display of the observation status display window W.

[0119] Fig. 11(A) shows an example of the display of the observation status display window W when a region is recognized for the first time. As shown in Fig. 11(A), a region display frame Fl1 is displayed at a predetermined position (near the top edge) within the window, and information about the recognized region is displayed in that region display frame Fl1. Fig. 11(A) shows an example when the "greater curvature of the gastric body" is recognized.

[0120] FIG. 11B shows an example of the display of the observation status display window W when a new site is recognized. As shown in FIG. 11B, a new site display frame Fl2 is added below the previously displayed site display frame Fl1, and information about the newly recognized site is displayed in the newly added site display frame Fl2. Also, as shown in FIG. 11B, an arrow Ar1 connects the previously displayed site display frame Fl1 and the newly added site display frame Fl2. Furthermore, as shown in FIG. 11B, a new evaluation value display frame Sc1 is displayed between the previously displayed site display frame Fl1 and the newly added site display frame Fl2, and an evaluation value is displayed in the newly displayed evaluation value display frame Sc1. This evaluation value is an evaluation value calculated for the observation performed between the site displayed in the site display frame Fl1 and the site displayed in the site display frame Fl2. FIG. 11B shows an example in which the evaluation value calculated for the observation performed between the "greater curvature of the gastric body" and the "antrum" is 82%. The evaluation value display frame Sc1 is displayed adjacent to the arrow Ar1.

[0121] FIG. 11C shows an example of the display of the observation status display window W when an additional site is recognized. As shown in FIG. 11C, a new site display frame Fl3 is added below the previously displayed site display frame Fl2, and information about the newly recognized site is displayed in the newly added site display frame Fl3. Also, as shown in FIG. 11C, when a new site is recognized, an arrow Ar2 connects the previously displayed site display frame Fl2 and the newly added site display frame Fl3. Furthermore, as shown in FIG. 11C, a new evaluation value display frame Sc2 is displayed between the previously displayed site display frame Fl2 and the newly added site display frame Fl3, and an evaluation value is displayed in the newly displayed evaluation value display frame Sc2. This evaluation value is an evaluation value calculated for the observation performed between the site displayed in the site display frame Fl2 and the site displayed in the site display frame Fl3. FIG. 11C shows an example in which the evaluation value calculated for the observation performed between the "antrum" and the "gastric angle" is 99%.

[0122] In this way, evaluation value display frames Sc1, Sc2, ... are displayed between each of the region display frames Fl1, Fl2, ... displayed in the observation status display window W, and the evaluation values ​​of the observation calculated between each of the regions are displayed in these evaluation value display frames Sc1, Sc2, .... This makes it easy to check the state of observation (whether or not the images were captured correctly) between the regions displayed in each of the region display frames Fl1, Fl2, ...

[0123] In this embodiment, the evaluation values ​​displayed in the evaluation value display frames Sc1, Sc2, etc. are also another example of information indicating that the area between the first region of interest (first part) and the second region of interest (second part) has been observed. Therefore, it is also possible to configure the system so that only the evaluation values ​​are displayed, without displaying the arrows Ar1, Ar2, etc.

[0124] As described above, the endoscope system 1 and image processing device 100 of this embodiment display an evaluation of the observation (photography) of an area that has been determined to have been observed, making it easy to determine whether the area that has been determined to have been observed has been correctly observed (photographed).

[0125] [Modifications] [Modifications of Display of Information Indicating Observation Status] [1] Modification of Display (1) FIG. 12 is a diagram showing another example of display of information indicating observation status.

[0126] FIG. 12 shows an example of a case where an observation situation is shown using a schematic diagram of an organ to be observed.

[0127] As shown in Fig. 12, a schematic diagram (picture of the organ) Sh of the organ to be observed is displayed in the observation status display window W. Fig. 12 shows an example in which the stomach is the observation target. Therefore, in this case, a schematic diagram of the stomach is displayed.

[0128] When a still image is captured and a region is recognized from the captured still image, predetermined marks M1, M2, ... are displayed on the schematic diagram Sh. The marks M1, M2, ... are displayed at positions corresponding to the recognized region. Therefore, for example, when the cardia of the stomach is recognized, a mark is displayed at the position of the cardia on the schematic diagram. The observed region can be recognized from these marks M1, M2, .... Figure 12 shows an example in which circular marks M1, M2, ... are displayed.

[0129] Furthermore, when the second and subsequent marks M2, M3, ... are displayed, arrows Ar1, Ar2, ... connecting the marks in the order in which they were displayed are displayed on the schema Sh. The arrows Ar1, Ar2, ... are displayed from the first displayed mark to the last displayed mark. Therefore, for example, when the marks are displayed in the order of mark M1, mark M2, and mark M3, an arrow Ar1 connecting mark M1 and mark M2 is displayed from mark M1 to mark M2. Furthermore, an arrow Ar2 connecting mark M2 and mark M3 is displayed from mark M2 to mark M3.

[0130] Displaying the arrows Ar1, Ar2, ... indicates that the areas between the marks M1, M2, ... connected by the arrows Ar1, Ar2, ... have been observed. Therefore, in this example, the arrows Ar1, Ar2, ... are an example of information indicating that the area between the first region of interest (first region) and the second region of interest (second region) has been observed.

[0131] 12 shows an example in which the greater curvature of the gastric body, the antrum, and the gastric angle are recognized in this order: greater curvature of the gastric body, antrum, gastric angle. Mark M1 is displayed at a position on the schematic diagram that indicates the greater curvature of the gastric body. Mark M2 is displayed at a position on the schematic diagram that indicates the antrum. Mark M3 is displayed at a position on the schematic diagram that indicates the gastric angle.

[0132] FIG. 13 is a diagram showing an example of the transition of the display of the observation status display window W.

[0133] 13A shows an example of the initial display of the observation status display window W. In this case, only the schematic diagram Sh is displayed in the observation status display window W.

[0134] Fig. 13(B) shows an example of the display of the observation status display window W when a region is recognized for the first time. As shown in Fig. 13(B), a mark M1 is displayed at the position of the region corresponding to the recognized region. Fig. 13(B) shows an example when the "greater curvature of the gastric body" is recognized. In this case, a mark M1 is displayed at the position on the schematic image Sh corresponding to the greater curvature of the gastric body.

[0135] Fig. 13(C) shows an example of the display of the observation status display window W when a new region is recognized. As shown in Fig. 13(C), a new mark M2 is displayed at the position of the region corresponding to the newly recognized region. Fig. 13(C) shows an example when a "vestibule" is newly recognized. In this case, mark M2 is displayed at the position corresponding to the vestibule on the schematic diagram Sh.

[0136] 13(C), an arrow Ar1 connects the previously displayed mark M1 and the newly displayed mark M2. The arrow Ar1 is displayed pointing from the previously displayed mark M1 to the newly displayed mark M2. This arrow Ar1 indicates that the endoscope 10 has moved from the region indicated by mark M1 to the region indicated by mark M2. In other words, it indicates that the region between the region indicated by mark M1 and the region indicated by mark M2 has been observed. The example shown in FIG. 13(C) indicates that the endoscope 10 has moved from the "greater curvature of the gastric body" indicated by mark M1 to the "vestimum" indicated by mark M2, and that the region of the stomach between them has been observed.

[0137] Fig. 13(D) shows an example of the display of the observation status display window W when a further region is recognized. As shown in Fig. 13(D), a new mark M3 is displayed at the position of the region corresponding to the newly recognized region. Fig. 13(D) shows an example when a "gastric angle" is newly recognized. In this case, mark M3 is displayed at the position corresponding to the gastric angle on the schematic image Sh.

[0138] 13(D), an arrow Ar2 connects the previously displayed mark M2 and the newly displayed mark M3. The arrow Ar2 is displayed pointing from the previously displayed mark M2 to the newly displayed mark M3. This arrow Ar2 indicates that the endoscope 10 has moved from the region indicated by mark M2 to the region indicated by mark M3. In other words, it indicates that the region between the region indicated by mark M2 and the region indicated by mark M3 has been observed. The example shown in FIG. 13(D) indicates that the endoscope 10 has moved from the "vestibule" indicated by mark M2 to the "gastric angle" indicated by mark M3, and that the region of the stomach between them has been observed.

[0139] In this way, each time a region is recognized, a mark M1, M2, ... is displayed on the schematic diagram Sh corresponding to the recognized region. This makes it easy to understand the observed region. Also, each time a mark M1, M2, ... is displayed, an arrow Ar1, Ar2, ... is displayed connecting the previously displayed mark with the subsequently displayed mark. This makes it easy to understand the observed region and the observation direction.

[0140] In this example, the marks are connected by arrows, but they may be connected by lines. By connecting the marks by arrows, the observation direction (the direction in which the endoscope 10 is moved) can be confirmed.

[0141] The arrows (including line segments) Ar1, Ar2, ... may be configured to be displayed for only a certain period of time. For example, the arrows may be displayed simultaneously with the mark of a newly recognized body part, and the arrows may be hidden after a period T has elapsed since the arrows were first displayed. The period T is a preset period of time.

[0142] Note that, when the display of the arrows Ar1, Ar2, ... is erased after a certain period of time, it is preferable to re-display them as necessary. That is, it is preferable to re-display the arrows Ar1, Ar2, ... that were previously displayed. For example, a configuration can be adopted in which the arrows are re-displayed in response to a user instruction to re-display (display history). Furthermore, a configuration can be adopted in which the arrows are re-displayed automatically after the observation is completed. The end of the observation can be determined automatically, in addition to a user instruction. The end of the observation can be determined when a specific region is recognized from the image. For example, the end of the observation can be determined when the same region as the first recognized region is recognized again. Alternatively, the end of the observation can be determined when a specific action is detected. For example, the end of the observation can be determined when the action of withdrawing the endoscope from the organ to be observed is detected. The action of withdrawing the endoscope from the organ to be observed can be detected, for example, by detecting a specific region or organ from the image.

[0143] If the device has a function for re-displaying the arrow, the processor 101 records the history of arrow display in the main storage device 102 or the auxiliary storage device 103. When re-displaying the arrow, the processor 101 refers to the recorded information.

[0144] In this example, the marks M1, M2, . . . are circular in shape, but the shape of the marks is not particularly limited.

[0145] [2] Display Modification (2) Fig. 14 is a diagram showing another example of the display of information indicating the observation status. Fig. 14 shows an example in which an evaluation value is further displayed in the display format shown in Fig. 13.

[0146] 14, the evaluation value calculated for each region is displayed adjacent to the arrows Ar1, Ar2, ... indicating each region. More specifically, evaluation value display frames Sc1, Sc2, ... are displayed adjacent to the arrows Ar1, Ar2, ... indicating the observed region, and the evaluation values ​​are displayed within the evaluation value display frames Sc1, Sc2, ....

[0147] In the example shown in Figure 14, the evaluation value of the observation of the stomach region between the area indicated by mark M1 (greater curvature of the gastric body) and the area indicated by mark M2 (antrum) is 82%; and the evaluation value of the observation of the stomach region between the area indicated by mark M2 (antrum) and the area indicated by mark M3 (gastric angle) is 99%.

[0148] In this example, the evaluation values ​​are displayed together with the arrows Ar1, Ar2, ..., but it is also possible to display only the evaluation values. In this case, for example, evaluation value display frames Sc1, Sc2, ... are displayed between the marks indicating each part, and the evaluation values ​​are displayed within the evaluation value display frames Sc1, Sc2, ....

[0149] The arrow and / or the rating value may be displayed for only a certain period of time. For example, the arrow may be hidden after a period T has elapsed since the arrow and / or the rating value was first displayed. The period T is a preset period of time.

[0150] When the display of the arrow and / or the evaluation value is erased after a certain period of time has elapsed, it is preferable to make it possible to redisplay the arrow and / or the evaluation value as necessary.

[0151] [3] Display Modification (3) Fig. 15 is a diagram showing another example of the display of information indicating the observation status. Note that Fig. 15 shows an example of the display (initial display) of the observation status display window W before part recognition.

[0152] This example is a display format that is particularly advantageous when the area to be observed (or the area to be photographed as a still image) is predetermined.

[0153] As shown in FIG. 15 , regions to be observed (or regions for which still images should be captured) are displayed on the schematic diagram Sh in advance using predetermined marks Ma, Mb, etc. FIG. 15 shows an example in which circles are used as the marks Ma, Mb, etc. The shapes and colors of the marks Ma, Mb, etc. are not particularly limited. Each mark Ma, Mb, etc. is displayed at a position corresponding to the region determined as the region to be observed. FIG. 15 shows an example in which five regions are to be observed. The five regions shown in FIG. 15 are the "cardia," "fundus," "greater curvature of the gastric body," "gastric angle," and "antrum." Mark Ma indicates the "cardia," mark Mb indicates the "fundus," mark Mc indicates the "greater curvature of the gastric body," mark Md indicates the "gastric angle," and mark Me indicates the "antrum."

[0154] In this example, the marks Ma, Mb, ... are examples of markers representing regions of interest. Also, a schematic diagram Sh in which the marks Ma, Mb, ... are arranged in a predetermined layout is an example of first information. Furthermore, the regions determined in advance as regions to be observed (or regions from which still images should be captured) are examples of multiple specific regions of interest selected from multiple regions of interest within the body.

[0155] FIG. 16 is a diagram showing an example of the display of the observation status display window after the region is recognized.

[0156] As shown in FIG. 16 , when a still image is captured and a region is recognized, the display mode of the mark for the recognized region changes. For example, the color (including transparency) of the mark for the recognized region changes. FIG. 16 shows an example in which the circle representing the mark changes from transparent to a chromatic or achromatic color. FIG. 16 also shows an example in which the region indicated by mark Mc (greater curvature of the gastric body), the region indicated by mark Md (gastric angle), and the region indicated by mark Me (vestibule) are recognized. Note that in this example, the display mode of the mark before the corresponding region is recognized (the display mode before the color is changed) is an example of the first mode. Also, the display mode of the mark after the corresponding region is recognized (the display mode after the color is changed) is an example of the second mode.

[0157] Furthermore, as shown in FIG. 16 , when the second or subsequent region is recognized, arrows Ar1, Ar2, ... are displayed connecting the mark of the previously recognized region to the mark of the subsequently recognized region. Each arrow Ar1, Ar2, ... is displayed from the mark of the previously recognized region to the mark of the subsequently recognized region. FIG. 16 shows an example in which the region indicated by mark Mc (greater curvature of the gastric body), the region indicated by mark Me (vestibule), and the region indicated by mark Md (gastric angle) are recognized in this order. In this case, arrow Ar1 connecting mark Mc and mark Me is displayed from mark Mc to mark Me. Furthermore, arrow Ar2 connecting mark Me and mark Md is displayed from mark Me to mark Md.

[0158] By displaying the arrows Ar1, Ar2, ..., it is shown that the areas between the marks Ma, Mb, ... connected by the arrows Ar1, Ar2, ... have been observed. The arrows Ar1, Ar2, ... allow the user to confirm the observation status.

[0159] In this example, the arrows Ar1, Ar2, ... are an example of information indicating that a region between the first region of interest (first part) and the second region of interest (second part) has been observed, and are also an example of second information. Also, the arrows Ar1, Ar2, ... are an example of information associating the marks Ma, Mb, ... (first region of interest and second region of interest) in the schematic diagram Sh (first information) in which the marks Ma, Mb, ... are displayed.

[0160] FIG. 17 is a diagram showing an example of the transition of the display of the observation status display window W.

[0161] Fig. 17(A) shows an example of the initial display of the observation status display window W. As shown in Fig. 17(A), a schematic diagram Sh is displayed in the observation status display window W, and marks Ma, Mb, ... indicating the area to be observed are displayed on the schematic diagram Sh. Each mark Ma, Mb, ... is displayed corresponding to the position of the area to be observed. Furthermore, each mark Ma, Mb, ... is displayed in a predetermined color (first form). In this example, the inside of the circle indicating the mark is displayed transparent.

[0162] Fig. 17(B) shows an example of the display of the observation status display window W when a region to be observed is recognized for the first time. As shown in Fig. 17(B), the display form of the mark Mc of the recognized region changes (switches to the second form). In this example, the inside of the circle indicating the mark changes from transparent to a chromatic or achromatic color. Fig. 17(B) shows an example when the "greater curvature of the stomach body" is recognized.

[0163] 17(C) shows an example of the display of the observation status display window W when a new site is recognized from the sites to be observed. As shown in FIG. 17(C), the color (display form) of the mark Me for the newly recognized site changes. FIG. 17(C) shows an example when the "vestibular region" is newly recognized.

[0164] 17(C), an arrow Ar1 is displayed connecting the mark Mc of the previously recognized region and the mark Me of the newly displayed region. The arrow Ar1 is displayed from the mark Mc of the previously recognized region to the mark Me of the newly recognized region. This arrow Ar1 indicates that the region of the organ between the region indicated by mark Mc and the region indicated by mark Me has been observed. In the example shown in FIG. 17(C), it indicates that the region of the stomach between the "greater curvature of the gastric body" indicated by mark Mc and the "vestimula" indicated by mark Me has been observed.

[0165] Fig. 17(D) shows an example of the display of the observation status display window W when a further site is recognized from among the sites to be observed. As shown in Fig. 17(D), the color (display form) of the mark Md of the newly recognized site changes. Fig. 17(D) shows an example when a "gastric angle" is newly recognized.

[0166] 17(D), an arrow Ar2 is displayed connecting the previously displayed mark Me and the newly displayed mark Md. The arrow Ar2 is displayed pointing from the previously displayed mark Me toward the newly displayed mark Md. This arrow Ar2 indicates that the region of the stomach between the location indicated by mark Me and the location indicated by mark Md has been observed. In the example shown in FIG. 17(D), the endoscope 10 has moved from the "vestibule" indicated by mark Me toward the "gastric angle" indicated by mark Md, indicating that the region of the stomach between them has been observed.

[0167] In this way, the area to be observed is indicated on the schematic diagram Sh by marks Ma, Mb, ..., and when observed (recognized), the display form of the marks Ma, Mb, ... is switched. This makes it easy to understand the observed area. Furthermore, each time an area is recognized, arrows Ar1, Ar2, ... are displayed connecting the mark of the previously recognized area with the mark of the subsequently recognized area. This makes it easy to understand the observed area and the observation direction.

[0168] In this example, the marks are connected by arrows, but they may be connected by lines. By connecting the marks by arrows, the observation direction (the direction in which the endoscope 10 is moved) can be confirmed.

[0169] The arrows (including line segments) Ar1, Ar2, ... may also be configured to be displayed for only a certain period of time. For example, the arrows may be displayed at the same time as the display mode of the mark of a newly recognized part is switched, and the arrows may be removed after a period T has elapsed since the arrows began to be displayed. The period T is a period that is set in advance. When the arrows are removed after a certain period of time has elapsed, it is preferable that they be able to be redisplayed as necessary.

[0170] In addition, in this example, the marks Ma, Mb, . . . are circular in shape, but the shape of the marks is not particularly limited.

[0171] In addition, in this example, when the corresponding part is recognized, the color of the mark is changed, but the manner of switching is not limited to this. For example, when the corresponding part is recognized, the shape of the mark may be changed or the mark may be made to blink.

[0172] Furthermore, in this example, a case has been described in which the region to be observed is determined in advance, but the region that can be recognized by the image processing device 100 may be indicated by a mark. In this case, all the regions that can be recognized by the image processing device 100 are displayed on the schematic diagram.

[0173] [4] Display Modification Example (4) Fig. 18 is a diagram showing another example of the display of information indicating the observation status. Fig. 18 shows an example in which an evaluation value is further displayed in the display format shown in Figs. 15 and 16.

[0174] 18, the evaluation value calculated for each region is displayed adjacent to the arrows Ar1, Ar2, ... indicating each region. More specifically, evaluation value display frames Sc1, Sc2, ... are displayed adjacent to the arrows Ar1, Ar2, ... indicating the observed region, and the evaluation values ​​are displayed in the evaluation value display frames Sc1, Sc2, ....

[0175] In the example shown in Figure 18, the evaluation value of the observation of the stomach region between the area indicated by mark Mc (greater curvature of the gastric body) and the area indicated by mark Me (vestibule) is 82%; and the evaluation value of the observation of the stomach region between the area indicated by mark Me (vestibule) and the area indicated by mark Md (gastric angle) is 99%.

[0176] In this example, the evaluation values ​​are displayed together with the arrows Ar1, Ar2, ..., but it is also possible to display only the evaluation values. In this case, for example, evaluation value display frames Sc1, Sc2, ... are displayed between the marks indicating each part, and the evaluation values ​​are displayed within the evaluation value display frames Sc1, Sc2, ....

[0177] The arrow and / or the rating value may also be configured to be displayed for only a certain period of time. For example, the display of only the arrow may be removed after a time T has elapsed since the start of the display of the arrow and / or the rating value. The time T is a preset time. When the display of the arrow and / or the rating value is removed after a certain period of time has elapsed, it is preferable to be able to redisplay them as necessary.

[0178] [5] Display Modification (5) Fig. 19 is a diagram showing another example of the display of information indicating the observation status. Note that Fig. 19 shows an example of the display (initial display) of the observation status display window W before part recognition.

[0179] This example is also an advantageous display format when the area to be observed (or the area to be photographed as a still image) is predetermined.

[0180] As shown in FIG. 19 , marks Ma, Mb, etc. indicating regions to be observed (or regions for which still images should be captured) are displayed in a predetermined layout in the observation status display window W. FIG. 19 shows an example in which five regions to be observed are present. The five regions shown in FIG. 19 are the "cardia," "fornix," "greater curvature," "gastric angle," and "antrum." Mark Ma indicates the "cardia," mark Mb indicates the "fornix," mark Mc indicates the "greater curvature," mark Md indicates the "gastric angle," and mark Me indicates the "antrum." In the example shown in FIG. 19 , each mark Ma, Mb, etc. is represented by a circle, and the name of the corresponding region is written inside the circle. In addition, in the example shown in FIG. 19 , the marks Ma, Mb, etc. indicating each region are arranged at the vertices of a pentagon.

[0181] In this example, the marks Ma, Mb, ... are examples of markers representing regions of interest. Also, a schematic diagram Sh in which the marks Ma, Mb, ... are arranged in a predetermined layout is an example of first information. Furthermore, the regions determined in advance as regions to be observed (or regions from which still images should be captured) are examples of multiple specific regions of interest selected from multiple regions of interest within the body.

[0182] FIG. 20 is a diagram showing an example of the display of the observation status display window after the region is recognized.

[0183] As shown in FIG. 20 , when a still image is captured and a region is recognized, the display mode of the mark for the recognized region changes. For example, the color of the mark for the recognized region is inverted. FIG. 20 shows an example in which the region indicated by mark Mc (greater curvature of the gastric body), the region indicated by mark Md (gastric angle), and the region indicated by mark Me (vestibule) are recognized. Note that in this example, the display mode of the mark before the corresponding region is recognized (the display mode before the color is changed) is an example of the first mode. Furthermore, the display mode of the mark after the corresponding region is recognized (the display mode after the color is changed) is an example of the second mode.

[0184] Furthermore, as shown in FIG. 20 , when the second or subsequent region is recognized, arrows Ar1, Ar2, ... are displayed connecting the mark of the previously recognized region to the mark of the subsequently recognized region. Each arrow Ar1, Ar2, ... is displayed from the mark of the previously recognized region to the mark of the subsequently recognized region. FIG. 20 shows an example in which the region indicated by mark Mc (greater curvature of the gastric body), the region indicated by mark Me (vestibule), and the region indicated by mark Md (gastric angle) are recognized in this order. In this case, arrow Ar1 connecting mark Mc and mark Me is displayed from mark Mc to mark Me. Furthermore, arrow Ar2 connecting mark Me and mark Md is displayed from mark Me to mark Md.

[0185] By displaying the arrows Ar1, Ar2, ..., it is shown that the areas between the marks Ma, Mb, ... connected by the arrows Ar1, Ar2, ... have been observed. The arrows Ar1, Ar2, ... allow the user to confirm the observation status.

[0186] 20, when evaluation values ​​are calculated between each part, the calculated evaluation values ​​are displayed between the marks representing each part. Specifically, evaluation value display frames Sc1, Sc2, ... are displayed above the arrows Ar1, Ar2, ..., and the evaluation values ​​are displayed in the evaluation value display frames Sc1, Sc2, ....

[0187] In the example shown in Figure 20, the evaluation value of the observation of the stomach region between the area indicated by mark Mc (greater curvature of the gastric body) and the area indicated by mark Me (vestibule) is 82%; and the evaluation value of the observation of the stomach region between the area indicated by mark Me (vestibule) and the area indicated by mark Md (gastric angle) is 99%.

[0188] In this example, the arrow and the evaluation value are an example of information indicating that the area between the first region of interest (first part) and the second region of interest (second part) has been observed, and are an example of second information. Also, the arrow and the evaluation value are an example of information associating the marks (first region of interest and second region of interest) in the diagram (first information) in which the marks are displayed.

[0189] As in the above-described modified examples (3) and (4), in this example, the marks allow the user to easily grasp the observed portions. Furthermore, the arrows allow the user to easily grasp the observed regions and the observation direction. Furthermore, the evaluation values ​​allow the user to grasp the observation status (photography status) of the regions determined to have been observed.

[0190] In this example, the marks are connected by arrows, but they may be connected by lines. By connecting the marks by arrows, the observation direction (the direction in which the endoscope 10 is moved) can be confirmed.

[0191] In addition, in this example, the marks Ma, Mb, . . . are circular in shape, but the shape of the marks is not particularly limited.

[0192] Fig. 21 is a diagram showing another example of marks. Fig. 21 shows an example of a case where icons are used to display marks Ma, Mb, ... indicating each part. The icons are configured using a schematic diagram of the organ to be observed, and are configured as a diagram in which a dot is displayed at the position of the part indicated by the icon. In Fig. 21, mark Ma indicates the "cardia," mark Mb indicates the "fundus," mark Mc indicates the "greater curvature of the gastric body," mark Md indicates the "gastric angle," and mark Me indicates the "antrum."

[0193] In addition, in this example, when the corresponding part is recognized, the color of the mark is changed, but the manner of switching is not limited to this. For example, when the corresponding part is recognized, the shape of the mark may be changed or the mark may be made to blink.

[0194] The arrow and / or the rating value may also be configured to be displayed for only a certain period of time. For example, the display of only the arrow may be removed after a time T has elapsed since the start of the display of the arrow and / or the rating value. The time T is a preset time. When the display of the arrow and / or the rating value is removed after a certain period of time has elapsed, it is preferable to be able to redisplay them as necessary.

[0195] Furthermore, in this example, the case where the parts to be observed are determined in advance has been described as an example, but parts that can be recognized by the image processing device 100 may be indicated by marks. In this case, all parts that can be recognized by the image processing device 100 are displayed (the marks of all parts that can be recognized are displayed in a predetermined layout).

[0196] [6] Other Display Modifications When calculating an evaluation value between recognized parts, if the calculated evaluation value is equal to or less than a threshold, the evaluation value may be displayed in a different manner from the usual manner. For example, the evaluation value may be highlighted.

[0197] FIG. 22 is a diagram showing an example of highlighting the evaluation value.

[0198] Figure 22 shows an example in which the evaluation value of the observation in the region between the area indicated by mark Me (vestibule) and the area indicated by mark Md (gastric angle) is below the threshold, and the evaluation value of the observation in the region between the area indicated by mark Mc (greater curvature of the gastric body) and the area indicated by mark Me (vestibule) exceeds the threshold.

[0199] As shown in FIG. 22 , evaluation values ​​that are equal to or less than the threshold are highlighted. FIG. 22 shows an example in which the display size of the evaluation value display frame is enlarged compared to normal (when the threshold is exceeded) and the frame is displayed in a different color. Note that the highlighting format is not limited to this. Other examples include highlighting the evaluation value display frame by flashing it or by changing its shape. When changing the color, it is preferable to select a color that easily draws attention. For example, when the threshold is exceeded, the evaluation value is displayed in achromatic color (e.g., gray), and when the evaluation value is equal to or less than the threshold, the evaluation value is displayed in chromatic color (e.g., red).

[0200] In this way, by highlighting when the calculated evaluation value is equal to or less than the threshold, it is possible to prompt the user to reconsider their observation.

[0201] When highlighting, the degree of highlighting may be changed in stages according to the evaluation value. That is, the lower the evaluation value, the stronger the degree of highlighting.

[0202] In the above example, the evaluation value is highlighted, but the arrow (including the case where it is displayed as a line segment) may also be highlighted in the same way. For example, the arrow in the area where the evaluation value is equal to or less than the threshold value may be highlighted by changing the line thickness of the arrow, changing the color of the arrow, or blinking the arrow.

[0203] Alternatively, the evaluation value may be displayed only when it is equal to or less than a threshold value. That is, if the calculated evaluation value exceeds the threshold value, it is assumed that the observation was correct, and therefore only an arrow is displayed without displaying the evaluation value. On the other hand, if the calculated evaluation value is equal to or less than the threshold value, it is highly likely that the observation was incorrect, and therefore the evaluation value is displayed on the screen to alert the user. In this case, the evaluation value may not be displayed, and the arrow may be highlighted to alert the user. FIG. 23 is a diagram illustrating an example of a case in which the arrow is highlighted. FIG. 23 illustrates an example in which the evaluation value of the observation in the region between the region indicated by mark Mc (the greater curvature of the gastric body) and the region indicated by mark Me (the antrum) is equal to or less than the threshold value, and the evaluation value of the observation in the region between the region indicated by mark Me (the antrum) and the region indicated by mark Md (the angle of the stomach) is greater than the threshold value. As shown in FIG. 23, the arrow Ar1 in the region where the evaluation value is equal to or less than the threshold value is highlighted. FIG. 23 illustrates an example in which the arrow is highlighted by changing the line thickness and color.

[0204] The highlighting can also be configured to be performed using marks. For example, in the example shown in Fig. 23, if the evaluation value of the observation in the region between the part indicated by mark Mc (greater curvature of the gastric body) and the part indicated by mark Me (vestibule) is equal to or less than a threshold, the marks Mc and Me are highlighted. For example, the marks Mc and Me can be highlighted by changing their color, size, shape, or blinking.

[0205] [Re-observation] When an area for which an evaluation value has been calculated (an area determined to have been observed) is observed again, it is preferable to recalculate the evaluation value. Furthermore, when the evaluation value is recalculated, it is preferable to update the calculated evaluation value with the newly calculated evaluation value.

[0206] When recalculating the evaluation value, for example, the system may be configured to accept a user's designation of the area for which the evaluation value is to be recalculated, and then recalculate the evaluation value. Alternatively, the system may be configured to automatically recalculate the evaluation value when a specific operation is detected as a trigger. For example, when a recognized body part is recognized again, the evaluation value of the area determined to have been recognized is recalculated starting from that body part. The following describes the process of automatically recalculating the evaluation value when a recognized body part is recognized again.

[0207] FIG. 24 is a flowchart showing the procedure of processing when an evaluation value is automatically recalculated when a recognized part is recognized again.

[0208] Here, an example will be described in which a region to be observed in advance (or a region to be photographed for a still image) is indicated by a mark on a schematic diagram displayed in the observation status display window W (see FIGS. 15 to 18). In this case, when a region is recognized, the display form of the mark for the recognized region is switched. Also, an arrow is displayed between the mark for the previously recognized region and the mark, and an evaluation value is displayed adjacent to the arrow.

[0209] The processor 101 of the image processing device 100 displays an observation status display window W on the screen 52 of the display device 50 (step S11).

[0210] When observation is started, the processor 101 determines whether or not a still image is to be captured (step S12). When a still image is captured, the processor 101 performs a part recognition process on the captured still image (step S13).

[0211] After the recognition process, the processor 101 determines whether or not the part has been recognized (step S14). If the part has been recognized, the processor 101 determines whether or not the recognized part is a previously recognized part (step S15). That is, the processor 101 determines whether or not the currently recognized part is a previously recognized part.

[0212] If the recognized part is not a previously recognized part, the processor 101 performs a process to determine the observation status (step S16). That is, it determines whether or not there is a previously recognized part, and if there is a previously recognized part, it determines that the area between the previously recognized part and the currently recognized part has been observed. The processor 101 then updates the display of the observation status display window W based on the part recognition result and the observation status determination result (step S17). If a part is recognized for the first time, the display form of the mark for the recognized part is switched. If there is a previously recognized part, the display form of the mark for the newly recognized part is switched, and an arrow is displayed between the mark for the previously recognized part and the mark for the previously recognized part. Furthermore, an evaluation value is displayed adjacent to the arrow.

[0213] After updating the display of the observation status display window W, the processor 101 determines whether the observation has ended (step S18). Note that if it is determined in step S12 that a still image has not been taken, or if it is determined in step S14 that the part has not been recognized, the processor 101 also determines whether the observation has ended. If it is determined that the observation has ended, the processing ends. If it is determined that the observation has not ended, the processor 101 determines whether a still image has been taken (step S12).

[0214] In step S15, if it is determined that the recognized part is a part that has already been recognized, the processor 101 performs a process to cancel the determination that the area that has been determined to be recognized, starting from the recognized part, is "already recognized" (step S19).

[0215] 25 is a conceptual diagram of a process for canceling a "recognized" determination. FIG. 25 illustrates an example in which the regions of the stomach are recognized in the order of the mark Mc region (greater curvature of the gastric body), the mark Me region (vestibule), and the mark Md region (gastric angle), and then the mark Me region (vestibule) is recognized again. In this case, the region between the mark Mc region (greater curvature of the gastric body) and the mark Me region (vestibule) and the region between the mark Me region (vestibule) and the mark Md region (gastric angle) are considered to be recognized regions. When the mark Me region (vestibule) is recognized after the mark Md region (gastric angle), the mark Me region (vestibule) is recognized, and therefore the "recognized" determination is canceled for the region starting from the mark Me region (vestibule) that was determined to be recognized. That is, the determination that the area between the part of mark Me (vestibule) and the part of mark Md (angle of the stomach) has been "recognized" is cancelled.

[0216] When the "recognized" determination for a region that has been determined to have been recognized is cancelled, the "recognized" determination for the portion defining the end point of that region is also cancelled. For example, in the example shown in Figure 25, the "recognized" determination for the portion of mark Md (the angle of the stomach) is cancelled.

[0217] After performing the process of canceling the determination that the corresponding region is "recognized," the processor 101 updates the display of the observation status display window W (step S17). In this case, the arrow and evaluation value for the corresponding region are erased. In the above example, the arrow and evaluation value displayed in the region between the location of mark Me (vestibular region) and the location of mark Md (gastric angle) are erased. In addition, the display of mark Md is switched to an unrecognized display.

[0218] After updating the display of the observation status display window W, the processor 101 determines whether the observation has ended (step S18). If it determines that the observation has not ended, it determines whether a still image has been taken (step S12). Then, when a still image is taken and a new region is recognized from the captured still image, a process of determining the observation status between the newly recognized region and the newly recognized region is performed (step S16). For example, in the example shown in FIG. 25 , if the region of mark Me (the antrum) is recognized again and then the region of mark Md (the angle of the stomach) is recognized, the region between the region of mark Me (the antrum) and the region of mark Md (the angle of the stomach) is determined to be a recognized region, and an evaluation value is calculated for that region. Then, the display of the observation status display window W is updated based on the region recognition results and the observation status determination results (step S17).

[0219] In this way, according to this example, for an area for which an evaluation value has been calculated (an area determined to have been observed), the evaluation value can be easily recalculated simply by performing an operation to re-recognize the starting point.

[0220] In the example shown in FIG. 25 , when the regions are recognized in the order of (1) the region of mark Mc (greater curvature of the gastric body), (2) the region of mark Me (vestrum), (3) the region of mark Md (gastric angle), (4) the region of mark Me (vestrum), and (5) the region of mark Md (gastric angle), once the region of mark Me (vestrum) is recognized, the region between the region of mark Mc (greater curvature of the gastric body) and the region of mark Me (vestrum) is determined to be the observed region. When the region of mark Md (gastric angle) is subsequently recognized, the region between the region of mark Me (vestrum) and the region of mark Md (gastric angle) is determined to be the observed region. Then, when the region of mark Me (vestrum) is recognized again, the region between the region of mark Me (vestrum) and the region of mark Md (gastric angle) is no longer determined to be “observed.” Thereafter, when the location of mark Md (gastric angle) is recognized again, the region between the location of mark Me (vestibule) and the location of mark Md (gastric angle) is determined to be the observed region. In this example, the location of mark Me (vestibule) is an example of the first location, and the location of mark Md (gastric angle) is an example of the second location. The image (still image) in which the location of mark Me (vestibule) is first recognized is an example of the first image, and the image (still image) in which the location of mark Md (gastric angle) is first recognized is an example of the second image. The image (still image) in which the location of mark Me (vestibule) is second recognized is an example of the third image, and the image (still image) in which the location of mark Md (gastric angle) is second recognized is an example of the fourth image.

[0221] [Variations of the Observation Evaluation Method, etc.] [1] Evaluation Method In the above embodiment, the image is evaluated based on the blur and blur of the image, but the method for evaluating the image is not limited to this. In addition to or instead of the blur and / or blur of the image, the image may be evaluated from the perspective of image brightness (exposure).

[0222] Furthermore, the criteria for evaluating the observation may be different for each region. For example, when observing the stomach, the evaluation criteria for observing the region between the greater curvature of the gastric body and the antrum may be different from the evaluation criteria for observing the region between the antrum and the gastric angle. This allows evaluation criteria appropriate for the observation of each region to be set, allowing each region to be appropriately evaluated.

[0223] The observation can also be evaluated from the perspective of the time or speed at which the area is observed. For example, the time at which the area is observed can be measured, and if the measured time exceeds a predetermined reference time, it can be judged as OK, and if it is equal to or less than the reference time, it can be judged as NG. Alternatively, the speed at which the area is observed can be measured, and if the measured speed is equal to or less than a predetermined reference speed, it can be judged as OK, and if it exceeds the reference speed, it can be judged as NG. It is more preferable to set the reference time and the reference speed for each area. The evaluation of OK or NG based on the time or speed at which the area is observed is another example of an evaluation value.

[0224] Since images are captured at a predetermined frame rate, measuring the time or speed of observing an area is essentially the same as measuring the number of images captured in the area. That is, the time can be calculated from the number of images, and the speed can be calculated from the time, so the observation can be evaluated based on the number of images. For example, if the number of images captured in an area exceeds a predetermined reference number, it can be evaluated as OK, and if it is below the reference number, it can be evaluated as NG. It is more preferable to set the reference number for each area.

[0225] When an observation is evaluated from multiple viewpoints, each evaluation value can be displayed individually. Also, when an observation is evaluated from multiple viewpoints, an evaluation value that is a composite of the evaluations from each viewpoint can be calculated and displayed. The composite evaluation can be, for example, evaluated as "OK" when the evaluation value exceeds or meets the standard for all items (viewpoints), or as "NG" when even one evaluation value is below the standard or does not meet the standard.

[0226] [2] Evaluation Target In the above embodiment, all frame images captured between body parts are evaluated, but specific images can also be evaluated. For example, images extracted at regular frame intervals can also be evaluated.

[0227] [3] Modifications of the Evaluation Value Calculation Method (1) Modification 1 of the Evaluation Value Calculation Method In the above embodiment, the evaluation value is calculated as the percentage of OK images, but the evaluation value can also be calculated as the percentage of NG images. When the evaluation value is calculated as the percentage of NG images, the lower the evaluation value, the higher the evaluation.

[0228] Furthermore, when calculating the percentage of OK images or the percentage of NG images as the evaluation value, if there are mutually similar images (including the same image) among the group of images to be evaluated (plurality of images taken in an area), it is preferable to calculate the evaluation value by excluding one of the images. For example, if an image similar to an image taken earlier is taken, the image taken later is excluded from the calculation of the evaluation value.

[0229] FIG. 26 is a block diagram of an observation situation determination unit having a function of calculating an evaluation value by excluding similar images.

[0230] 26 , the observation situation determination unit 113D of this example has the functions of a similar image detection unit 113D4 in addition to the observation area determination unit 113D1, the photography evaluation unit 113D2, and the evaluation value calculation unit 113D3. The functions of the observation area determination unit 113D1, the photography evaluation unit 113D2, and the evaluation value calculation unit 113D3 are the same as the functions of the observation situation determination unit 113D of the image processing device 100 of the above embodiment. Therefore, the function of the similar image detection unit 113D4 will be described here.

[0231] The similar image detection unit 113D4 acquires images captured in chronological order by the endoscope 10 and detects images that are similar to one another. Specifically, it detects images that are similar to previously captured images (images that have already been captured). Known techniques can be used for the detection method. For example, a method can be used to detect similar images using image correlation. The detection target is a group of images captured in an area that has been determined to have been observed. Therefore, the detection target is reset each time a site is recognized.

[0232] The similar image detection unit 113D4 sequentially processes the acquired images and adds only images that are dissimilar to the captured image to the image evaluation unit 113D2, thereby making it possible to exclude images that are similar to the captured image from the evaluation targets (targets for calculating the evaluation value).

[0233] Similar images are captured continuously when the movement of the endoscope 10 is stopped, etc. As in this example, by excluding images similar to previously captured images from the evaluation targets, the observation of the relevant region can be evaluated more accurately.

[0234] When calculating the evaluation value, images similar to the captured image may be excluded from the calculation of the evaluation value.

[0235] (2) Modification 2 of the Evaluation Value Calculation Method The evaluation value may be calculated in real time. In this case, the evaluation value is calculated sequentially after the parts are recognized.

[0236] FIG. 27 is a conceptual diagram showing how the evaluation value is calculated in real time.

[0237] 27, the symbol IOK indicates an image that has been evaluated as an OK image, and the symbol ING indicates an image that has been evaluated as an NG image.

[0238] 27 shows the state in which the 20th frame image is captured after the body part is recognized. The evaluation value S is calculated as the ratio (S = Nok / Nall) of the accumulated number of OK images Nok captured after body part recognition to the accumulated number of images Nall captured after body part recognition. Therefore, if the number of NG images is two when the 20th frame image is captured, the evaluation value S is S = 18 / 20 = 90 [%]. In this way, the evaluation value can be calculated in real time.

[0239] When the evaluation value is calculated in real time, it is preferable to display the calculation result on the display device 50 in real time.

[0240] FIG. 28 is a diagram showing an example of displaying the evaluation values ​​in real time.

[0241] Fig. 28 shows an example of a case where evaluation values ​​calculated in real time are displayed in the observation status display window W. As shown in Fig. 28, an evaluation value display area SR is set in the observation status display window W, and evaluation values ​​are displayed in the evaluation value display area SR. The evaluation value display area SR is set in a blank space when a schematic diagram Sh is displayed. Fig. 28 shows an example of a case where a rectangular evaluation value display area SR is set in the upper right corner of the observation status display window W.

[0242] 28 shows an example in which the region indicated by mark Mc (greater curvature of the gastric body) is followed by the region indicated by mark Me (vestibule). In this case, the region between the region indicated by mark Mc (greater curvature of the gastric body) and the region indicated by mark Me (vestibule) is the observed region. An evaluation value based on the image observed after recognizing the region indicated by mark Me (vestibule) is displayed in the evaluation value display area SR. Since it is not yet determined which region of the image was observed after recognizing the region indicated by mark Me (vestibule), an evaluation value is displayed in the evaluation value display area SR. Once the observed region is determined, an evaluation value is displayed near the arrow indicating the region. The evaluation value displayed at this time is the determined evaluation value.

[0243] 29 is a diagram showing an example of the transition of the display in the observation status display window, which shows an example in which the region to be observed is predetermined.

[0244] Fig. 29(A) shows an example of the initial display of the observation status display window W. As shown in Fig. 29(A), a schematic diagram Sh is displayed in the observation status display window W, and marks Ma, Mb, ... indicating the areas to be observed are displayed on the schematic diagram Sh.

[0245] FIG. 29(B) shows an example of the display of the observation status display window W when a region to be observed is recognized for the first time. As shown in FIG. 29(B), the display form of the mark Mc for the recognized region changes. FIG. 29(B) shows an example when the "greater curvature of the gastric body" is recognized. As shown in FIG. 29(B), when the region is recognized, an evaluation value calculated based on an image captured after recognition is displayed in real time in the evaluation value display area SR. FIG. 29(B) shows an example when the evaluation value calculated from an image captured after recognizing the region indicated by the mark Mc (greater curvature of the gastric body) is 100% at the time of display.

[0246] 29(C) shows an example of the display of the observation status display window W when a new site is recognized from the sites to be observed. As shown in FIG. 29(C), the color (display form) of the mark Me of the newly recognized site changes. FIG. 29(C) shows an example when a "vestibular region" is newly recognized.

[0247] 29(C), an arrow Ar1 is displayed connecting the mark Mc of the previously recognized region and the mark Me of the newly displayed region. An evaluation value display frame Sc1 is displayed adjacent to the arrow Ar1, and the final evaluation value is displayed. The example shown in FIG. 29(C) indicates that the region of the stomach between the "greater curvature of the gastric body" indicated by the mark Mc and the "vestimula" indicated by the mark Me has been observed.

[0248] Furthermore, as shown in Fig. 29(C), when a region is recognized, an evaluation value calculated based on an image captured after the recognition is displayed in real time in the evaluation value display area SR. Fig. 29(C) shows an example in which the evaluation value calculated from an image captured after the region (vestibular region) indicated by the mark Me is recognized is 98[%] at the time of display.

[0249] Figure 29(D) shows an example of the display of the observation status display window W when a further site is recognized from among the sites to be observed. As shown in Figure 29(D), the color (display form) of the mark Md of the newly recognized site changes. Figure 29(D) shows an example when a new "gastric angle" is recognized.

[0250] 29(D), an arrow Ar2 is displayed connecting the previously displayed mark Me and the newly displayed mark Md. An evaluation value display frame Sc2 is displayed adjacent to the arrow Ar2, and the final evaluation value is displayed. The example shown in FIG. 29(D) indicates that the endoscope 10 has moved from the "vestibule" indicated by mark Me toward the "gastric angle" indicated by mark Md, and that the region of the stomach between them has been observed.

[0251] Furthermore, as shown in Fig. 29(D), when a region is recognized, an evaluation value calculated based on an image captured after the recognition is displayed in real time in the evaluation value display area SR. Fig. 29(D) shows an example in which the evaluation value calculated from an image captured after the region indicated by mark Md (gastric angle) is recognized is 95[%] at the time of display.

[0252] In this way, the evaluation value calculated based on the image captured after the part recognition is displayed in real time in the evaluation value display area SR, allowing the user to grasp the observation situation in real time.

[0253] 27 and 28, an example has been described in which evaluation values ​​calculated in real time are displayed in the observation status display window W, but the location where evaluation values ​​calculated in real time are displayed is not limited to this. They may be displayed in a location other than the observation status display window W. Furthermore, even when displayed in the observation status display window W, there are no particular limitations on the display position. For example, they may be displayed near the mark indicating the most recently recognized part (in a position within a range where the relationship with the part can be seen).

[0254] Furthermore, when the evaluation value is displayed in real time as in this example, it is preferable to be able to reset the calculation of the evaluation value as needed. For example, a reset can be performed when the same part as the part recognized immediately before is recognized. Alternatively, a reset command can be given by operating a foot switch or the like.

[0255] [Area Recognition] [1] Image to be Processed for Recognition In the above embodiment, an area is recognized from an image captured as a still image, but the image to be processed for area recognition is not limited to this. For example, an arrangement may be made in which an area is recognized from an image arbitrarily selected by the user. Image selection may be performed, for example, using a foot switch or the like, and area recognition may be performed from the image displayed in the observation image display area A1a at the time the foot switch is pressed. In addition, an arrangement may be made in which an image is selected using a switch provided on the operation unit of the endoscope 10, voice input, or other means.

[0256] Alternatively, the recognition of regions can be performed on all captured images (images of all frames). In this case, the processor 101 processes each image acquired in chronological order to recognize the regions depicted in each image. Recognizing regions from all captured images enables automatic recognition of regions. In this case, when one region is recognized in an organ to be observed and then another region is recognized, the area between the two regions is determined to have been observed. For example, in the stomach, when the greater curvature of the gastric body is recognized and then the antrum is recognized, the area between the greater curvature of the gastric body and the antrum is determined to have been observed. After that, when the gastric angle is recognized, the area between the antrum and the gastric angle is determined to have been observed. In this way, each time the region recognized from the image changes, the area between the previously recognized region is determined to have been observed.

[0257] Note that if the parts of an organ to be observed can be classified into multiple types, it is not necessary to be able to recognize all parts. It is sufficient to recognize only the specific parts that have been determined as targets to be recognized. Targets to be recognized (areas of interest) include parts that are considered anatomical landmarks, parts that must be observed from the perspective of examinations, and parts for which still images must be taken. In addition to parts that can be classified as organs or internal body parts, characteristic parts and areas of organs or internal bodies can also be selected as targets to be recognized. Furthermore, areas selected by the user (for example, landmark areas), lesions, inflammation areas, etc. can also be set as targets to be recognized. These areas can be set, for example, from images taken in the past.

[0258] Furthermore, when automatically recognizing a body part, it is not necessary to process all captured images for recognition. For example, images captured at a predetermined frame interval can be processed for recognition.

[0259] [2] Recognition Processing When recognizing a body part, the photographing condition may also be evaluated to perform body part recognition processing. That is, the body part recognition processing may also be performed by evaluating whether the image to be processed has been photographed appropriately. In this case, body part recognition is confirmed only if it is determined that the image has been photographed appropriately. Therefore, for example, even if a body part can be recognized from an image, if it is determined that the image has not been photographed appropriately, the body part is considered not to have been recognized.

[0260] Whether an image has been captured appropriately is determined from the perspective of, for example, image blur, image shake, image brightness, composition, and image dirt (reflection of dirt on the observation window). Regarding image blur and shake, an image without blur or shake (including cases where blur and shake are within an acceptable range) is determined to be an appropriately captured image. Regarding image brightness, an image captured with appropriate brightness (appropriate exposure) is determined to be an appropriately captured image. Regarding composition, for example, a determination is made from the perspective of whether the part to be recognized has been captured in a predetermined composition (for example, a structure arranged in the center of the screen). Regarding image dirt, for example, an image without cloudiness (a clear image) is determined to be an appropriately captured image.

[0261] Whether an image has been properly captured can be determined based on a composite of multiple perspectives. In this case, requirements can be defined for each part. For example, while determining whether an image has been properly captured for all parts based on image blur, image shake, image brightness, and image dirt, determining the quality of the composition for a specific part can also be configured.

[0262] Whether an image has been appropriately captured can be determined using, for example, a determiner. The determiner can be configured, for example, with a trained model. When determining whether an image has been appropriately captured from multiple perspectives, separate determiners are prepared. For example, when determining whether an image has been appropriately captured from the perspectives of image blur, image shake, image brightness, quality of composition, image dirt, etc., separate determiners are prepared, such as a determiner for determining whether an image is blurred, a determiner for determining whether an image is shaken, a determiner for determining whether the image brightness is appropriate, a determiner for determining whether the composition is good, and a determiner for determining whether an image is dirty.

[0263] The criteria established for determining whether an image has been appropriately captured are an example of the second criteria. Furthermore, the criteria established in terms of image blur, image shake, image brightness, quality of composition, image contamination, etc. are examples of the content of the second criteria.

[0264] [Observation Object] In the above embodiment, the case of observing the stomach has been described as an example, but the organ to be observed is not particularly limited. Also, organs other than internal organs can be observed. That is, the present invention can be applied to the case of observing the inside of a body using an endoscope.

[0265] FIG. 30 shows an example of the display of the observation status display window when observing the large intestine.

[0266] FIG. 30 shows an example in which the region to be observed has been determined in advance. As shown in FIG. 30 , a schematic diagram Sh of the large intestine to be observed is displayed in the observation status display window W, and predetermined marks MA, MB, ... (circles in the example shown in FIG. 30 ) are displayed on the schematic diagram Sh to indicate the region of the large intestine to be observed (or the region to be photographed for a still image). FIG. 30 shows an example in which four regions to be observed. The four regions shown in FIG. 30 are the "ileocecal," "hepatic flexure (right colic flexure)," "splenic flexure (left colic flexure)," and "anus." The hepatic flexure is the transition region between the ascending colon and the transverse colon. The splenohepatic flexure is the transition region between the transverse colon and the descending colon. Mark MA indicates the "ileocecal," mark MB indicates the "hepatic flexure," mark MC indicates the "splenic flexure," and mark MD indicates the "anus."

[0267] 30 shows an example in which the regions of the large intestine are recognized in the order of the region indicated by mark MA (ileocecal), the region indicated by mark MB (hepatic flexure), and the region indicated by mark MC (splenic flexure). In this case, the region (ascending colon) between the region indicated by mark MA (ileocecal) and the region indicated by mark MB (hepatic flexure) and the region (transverse colon) between the region indicated by mark MB (hepatic flexure) and the region indicated by mark MC (splenic flexure) are determined to be recognized regions.

[0268] The display form of the mark of the recognized part is switched (for example, the color is changed). Also, the recognized area is displayed with arrows Ar1, Ar2, ... and an evaluation value. The arrows Ar1, Ar2, ... are displayed pointing in the observation direction. The evaluation value is displayed in evaluation value display frames Sc1, Sc2, ... displayed adjacent to the arrows Ar1, Ar2, ....

[0269] Although the above embodiment has been described taking as an example the case where observation is made with a so-called flexible endoscope, the present invention can also be applied to the case where observation is made with a so-called rigid endoscope.

[0270] [Recording of Observation Status Judgment Results] As described above, the results of the observation status judgment process can be recorded in association with information on still images and / or videos taken during observation, information on lesions detected during observation, information on the results of differentiation performed during observation, etc.

[0271] The information to be recorded may include (1) information on the order in which each part was recognized, (2) if parts were recognized by capturing a still image, information on the still image in which each part was recognized, (3) information on the evaluation value calculated for each area (the area between recognized parts), (4) the shooting time and / or number of frames for each area, and (5) the overall shooting time and / or number of frames. The overall shooting time is the shooting time from the recognition of the first part to the recognition of the last part. The overall number of frames is the total number of frames from the recognition of the first part to the recognition of the last part.

[0272] Furthermore, the information on the determination results of the observation situation can be configured to be recorded in an external storage device in addition to or instead of the storage device (auxiliary storage device 103) provided in the image processing device 100. For example, the result information can be transmitted to a system that manages endoscopic examination results, etc. (such as an endoscope information management system), and recorded in that system or a data server (such as an endoscopic examination data server) connected to that system.

[0273] Furthermore, when an observation (including an examination) is performed using an endoscope, a report showing the observation results is usually created. Therefore, if a process for determining the observation status is performed during the observation, it is preferable to also include (record) information about the results of that determination process in the report.

[0274] Furthermore, in recent years, reports have been created using devices that assist in the creation of reports (report creation support devices). The report creation support devices acquire information necessary for creating a report from an image processing device or the like, and the information acquired by the report creation support device can include information on the determination results of the observation situation. This allows the information to be written (recorded) in the report to include information on the determination results of the observation situation, and the information can be automatically entered.

[0275] [Hardware Configuration] The functions of the endoscopic image processing device are realized by various processors. The various processors include a CPU and / or a GPU (Graphic Processing Unit), which are general-purpose processors that execute programs and function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processing. The term "program" is synonymous with "software."

[0276] A single processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Furthermore, multiple processing units may be composed of a single processor. Examples of multiple processing units composed of a single processor include: a first configuration, as typified by computers used as clients or servers, in which a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are composed of one or more of the above-mentioned various processors as a hardware structure.

[0277] [Others] The various modifications can be used in appropriate combinations.

[0278] 1 Endoscope system 10 Endoscope 20 Light source device 30 Processor device 31 Endoscope control unit 32 Light source control unit 33 Image processing unit 34 Input control unit 35 Output control unit 40 Input device 50 Display device 52 Screen 100 Image processing device 101 Processor 102 Main memory device 103 Auxiliary memory device 104 Input / output interface 111 Image acquisition unit 112 Command acquisition unit 113 Image processing unit 113A Lesion detection unit 113B Differentiation unit 113C Site recognition unit 113D Observation situation determination unit 113D1 Observation area determination unit 113D2 Photography evaluation unit 113D3 Evaluation value calculation unit 113D4 Similar image detection unit 114 Recording control unit 115 Display control unit A1 Main display area A1a Observation image display area A1b Information display area A2 Sub-display area A3 Identification result display area Ar1 Arrow Ar2 Arrow Fl1 Region display frame Fl2 Region display frame Fl3 Region display frame I Image IOK Image (OK image) ING Image (NG image) Im Image Ip Subject information Is Still image M1, M2, ... Mark Ma, Mb, ... Mark MA, MB, ... Mark SR Evaluation value display area Sc1, Sc2, ... Evaluation value display frame Sh Schematic diagram W Observation status display window S1 to S7 Processing procedure when information showing the observation status is displayed on the display device S11 to S19 Processing procedure when automatically recalculating the evaluation value when a recognized region is recognized again

Claims

1. An image processing device that processes multiple images taken in time series by an endoscope, comprising a processor that acquires the multiple images, processes the images, and recognizes a region of interest that appears in the image from multiple regions of interest within the body, and when a first region of interest among the multiple regions of interest is recognized from a first image among the multiple images, and a second region of interest among the multiple regions of interest is recognized from a second image that is chronologically later than the first image, displays information on a display device indicating that a region between the first region of interest and the second region of interest has been observed.

2. The image processing device described in claim 1, wherein the processor causes the display device to display information on a plurality of specific regions of interest selected from the plurality of regions of interest within the body as first information, and causes the display device to display information indicating that a region between the first region of interest and the second region of interest has been observed as second information.

3. The image processing device according to claim 2, wherein the second information is configured from information that associates the first region of interest with the second region of interest in the first information.

4. The image processing device according to claim 3, wherein the second information includes information indicating a direction of movement of the observation.

5. The image processing device according to claim 3, wherein the first information is composed of information in which signs representing the region of interest are arranged in a specific layout, and the second information is composed of information associating the signs.

6. The image processing device according to claim 5, wherein the first information includes a schematic diagram of the organ to be observed, and is configured with information in which the markers are arranged at the positions of each region of interest on the schematic diagram.

7. The image processing device according to claim 6, wherein the second information is composed of line segments connecting the signs.

8. The image processing device according to claim 7, wherein the line segment includes an arrow indicating the direction of movement of the observation.

9. The image processing device according to any one of claims 5 to 8, wherein the processor switches the display form of the sign in the recognized region of interest from a first form to a second form.

10. An image processing device according to any one of claims 2 to 8, wherein the processor calculates an evaluation value of the observation based on the images between the first image and the second image in chronological order, and the second information includes information on the evaluation value.

11. The image processing device according to claim 10, wherein the evaluation value is calculated based on the number of images that satisfy a first criterion and / or the number of images between the first image and the second image.

12. The image processing device according to claim 11, wherein the first criterion is set based on at least one of the blur state of the image, the blur state of the image, and the brightness of the image.

13. The image processing device according to claim 10, wherein the evaluation value is calculated as a percentage of images that satisfy a first criterion or a percentage of images that do not satisfy the first criterion.

14. The image processing device according to claim 13, wherein the processor extracts mutually similar images from among the images between the first image and the second image, and calculates the evaluation value by excluding one of the extracted mutually similar images.

15. The image processing device according to claim 10, wherein the processor displays the second information in a form according to the evaluation value.

16. The image processing device described in claim 10, wherein the processor updates the evaluation value and updates the display of the second information when the first area of ​​interest is recognized from a third image that is chronologically later than the second image, and the second area of ​​interest is recognized from a fourth image that is chronologically later than the third image.

17. An image processing device according to any one of claims 1 to 8, wherein the processor displays the plurality of images on the display device in chronological order, accepts selection of the first image from the images displayed on the display device, and accepts selection of the second image from the images displayed on the display device after the first image.

18. The image processing device according to claim 17, wherein the processor records the images selected as the first image and the second image as still images.

19. The image processing device according to any one of claims 1 to 8, wherein the processor processes each of the plurality of images to recognize the region of interest appearing in each of the images.

20. An image processing device according to any one of claims 1 to 8, wherein, when the processor recognizes the region of interest from the image, it further determines whether the image in which the region of interest is recognized satisfies a second criterion, and if the second criterion is satisfied, it confirms the recognition of the region of interest.

21. The image processing device according to claim 20, wherein the content of the second criterion is set for each of the regions of interest.

22. The image processing device according to any one of claims 2 to 8, wherein the processor ends the display of the second information after a time T has elapsed since the start of displaying the second information.

23. The image processing device according to claim 22, wherein the processor records a history of the display of the second information, and when newly displaying the second information, simultaneously displays the second information that was previously displayed based on the history.

24. The image processing device according to claim 23, wherein the processor displays the second information previously displayed based on the history when the observation is completed and / or when an instruction to display the history is given.

25. An image processing device according to any one of claims 2 to 8, wherein the processor terminates the display of the second information when the first attention area is recognized from a third image that is chronologically later than the second image.

26. An image processing device according to any one of claims 1 to 8, wherein the plurality of regions of interest are a plurality of parts of an organ to be observed.

27. An endoscope system comprising: an endoscope; a display device; and an image processing device according to any one of claims 1 to 8 that processes images captured by the endoscope.

Images