Object detection device, object detection method, object detection program, and endoscope system

Abstract

In order to provide an object detection device (10) that detects a specific object image from an image acquired by an endoscope 20 wherein, even if a detected specific object is lost from view, the same specific object can be easily detected, the present invention is provided with: a specific object image detection unit 14 that detects an image area including a specific object from an endoscope image based on first image data acquired by the endoscope 20; a storage control unit 13 that temporarily stores second image data including the detected specific object in a temporary storage unit 51; and a display control unit 12 that displays a reference image based on the second image data on a display device 30 when the disappearance of the specific object from the endoscope image is detected.

Description

Object detection device, object detection method, object detection program, and endoscope system

[0001] The present invention relates to an object detection device, an object detection method, an object detection program, and an endoscope system that detect an image area of ​​a specific object from an image acquired by an endoscope.

[0002] Conventionally, endoscopic systems that include an endoscope that captures images of the inside of a subject and acquires image data, a processor that performs various types of image processing on the image data acquired by the endoscope, a display device that displays the image data processed by the processor as a visible image, and a storage device that records or stores the image data have been widely used in, for example, the medical field and the industrial field.

[0003] Furthermore, in the medical field, various examinations are widely performed using this type of endoscope system to observe the inside of living organs, etc. In an examination using a medical endoscope system (hereinafter referred to as an endoscopic examination), for example, an image is captured by an imaging device provided at the tip of the insertion section while the insertion section of the endoscope inserted into the inside of the organ is being withdrawn. At this time, the endoscopic images acquired by the imaging device are sequentially displayed as moving images on a display device in real time. At the same time, the moving image data can also be stored in a storage device.

[0004] During such an endoscopic examination, the doctor performs so-called screening, searching for lesions such as polyps or tumors occurring on the inner walls of organs as specific objects (hereinafter referred to as specific objects) while observing the real-time endoscopic image displayed on the display device (or, after the examination, a replayed endoscopic image based on recorded or stored endoscopic image data).

[0005] Normally, specific objects such as lesions are very small and have similar colors to the inner walls of surrounding organs, etc., so skill is required to accurately distinguish and detect the image area of ​​the specific object from an endoscopic image. In addition, in endoscopic examinations, the operation of the endoscope itself, such as the operation of pointing the tip of the insertion portion of the endoscope (the observation window of the imaging device) at the desired object and capturing an image, requires skill.

[0006] Therefore, for example, Japanese Patent Publication No. 2004-207842 discloses an image processing device in which a target image for identifying an object to be saved is registered in advance, and a series of image data being captured is sequentially analyzed based on the registered target image data, and when an image containing the registered target is detected from the series of captured images, the captured image is extracted as an object to be saved and saved (recorded or stored).

[0007] Meanwhile, in recent years, image processing technology for so-called computer-aided diagnosis (CAD) has been put into practical use. This is an image diagnostic system that uses a computer or the like to process medical image data acquired by an endoscope or the like, and assists doctors in making diagnoses based on the obtained analysis results and processed images.

[0008] In the image processing technology presented in this type of conventional computer-aided diagnosis (CAD), for example, when a specific object is detected on the display screen of an endoscopic image, various object detection devices are being put into practical use, which mark the image area containing the specific object by displaying a frame or other marking to indicate that the specific object has been detected.

[0009] Generally, during an endoscopic examination in which the inside of a living organ is observed, the condition inside the organ, including the lesion (specific object), is constantly changing due to various disturbances, such as the patient's heartbeat, breathing, or changes in body position, or the peristaltic movement of the organ, or bleeding or food residue inside the organ.

[0010] Therefore, for example, when screening the inside of an organ during an endoscopic examination, even if a doctor detects a specific object, he or she often loses sight of the specific object due to changes in the state of the inside of the organ caused by peristaltic movement, etc.

[0011] However, in an endoscopic examination, a doctor's main objective is not only to detect a specific object but also to observe the specific object thoroughly, and therefore, even if a specific object is once detected and then lost, there is always a desire to quickly detect the same specific object again.

[0012] However, even if a specific object is detected from an endoscopic image using the technology disclosed in the above-mentioned Japanese Patent Publication No. 2004-207842, etc., only a captured image containing a registered target is stored when the captured image is detected. Therefore, the technology disclosed in this publication, etc., does not take into consideration, for example, re-detecting a specific object when the specific object is lost after detection.

[0013] Furthermore, in conventional object detection devices using computer-aided diagnosis (CAD) technology, if a specific object disappears from the displayed image, the marking display is generally canceled. Typically, a situation in which a detected specific object is lost is considered to be a change in the state inside an organ, such as when the specific object becomes separated from the tip (observation window) of the endoscope, or when foreign matter gets between the tip (observation window) of the endoscope and the specific object. In such a case, it is extremely difficult to detect the same specific object again.

[0014] Therefore, in conventional object detection devices, if a specific object that has been detected in an endoscopic image is lost, it takes time to detect the same specific object again, resulting in problems such as an increase in the time required for the endoscopic examination, or the object being forgotten or overlooked. For these reasons, conventional endoscopic examinations have had the problem of imposing a heavy burden on doctors, medical professionals, examiners, etc., and subjects (patients, etc.).

[0015] An object of the present invention is to provide an object detection device, an object detection method, an object detection program, and an endoscopic system that can easily detect the same specific object again even if the detected specific object is lost, in an object detection device that detects the image area of ​​a specific object from an image acquired by medical equipment such as an endoscope or inspection equipment (such as an image inspection device).

[0016] In order to achieve the above-mentioned object, an object detection device of one embodiment of the present invention comprises a specific object image detection unit that detects an image area containing a specific object from an endoscopic image based on first image data acquired by an endoscope, a memory control unit that temporarily stores second image data containing the detected specific object in a temporary memory unit, and a display control unit that displays a reference image based on the second image data on a display device when disappearance of the specific object is detected from the endoscopic image.

[0017] An object detection method according to one aspect of the present invention includes a screening step of displaying endoscopic images continuously acquired by an endoscope in chronological order, a detection step of detecting a specific object from the endoscopic image, a temporary storage step of temporarily storing image data corresponding to an image area including the detected specific object, a reference image display step of displaying the image area including the specific object as a reference image when the disappearance of the specific object from the endoscopic image is detected, and a focus control step of terminating the display of the reference image and performing automatic focus adjustment control on the specific object when the specific object corresponding to the reference image and identical to the specific object that disappeared from the endoscopic image is detected again within the endoscopic image.

[0018] An object detection program according to one embodiment of the present invention causes a computer to execute the following processes: continuously displaying endoscopic images; detecting specific objects from the endoscopic images; temporarily storing image data corresponding to image areas containing the detected specific objects; displaying the image areas containing the specific objects as reference images when the disappearance of the specific objects from the endoscopic images is detected; and terminating the display of the reference images and performing automatic focus adjustment on the specific objects when the specific objects corresponding to the reference images are detected again in the endoscopic images and are identical to the specific objects that disappeared from the endoscopic images.

[0019] An endoscopic system according to one aspect of the present invention is an endoscopic system comprising: an endoscope including an imaging unit having an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system; and a processor, wherein the processor comprises a specific object image detection unit that detects an image area including a specific object from an endoscopic image based on first image data acquired by the imaging unit; a memory control unit that temporarily stores second image data including the detected specific object in a temporary memory unit; a display control unit that, when disappearance of the specific object is detected from the endoscopic image, displays a reference image based on the second image data in a second display area within the display screen of a display device that is different from the first display area in which the endoscopic image is displayed; and a focus control unit that drives and controls the imaging optical system, wherein when the object image detection unit detects that the same specific object that disappeared from the endoscopic image has been detected again in the endoscopic image and that the same specific object continues to be displayed in the endoscopic image, the focus control unit drives and controls the imaging optical system to perform automatic focus adjustment control for the specific object.

[0020] An object detection method according to one aspect of the present invention detects an image area containing a specific object based on image data acquired by an image inspection device, temporarily stores image data of the specific object image capture result containing the detected specific object in a temporary storage unit, and, if the disappearance of the specific object is detected from an image based on the image data, displays a reference image based on the image data of the specific object image capture result on a display unit that displays an image based on the acquired image data.

[0021] According to the present invention, in an object detection device that detects an image area of ​​a specific object from an image acquired by a medical device such as an endoscope or an inspection device (such as an image inspection device), it is possible to provide an object detection device, an object detection method, an object detection program, and an endoscopic system that can easily detect the same specific object again even if the detected specific object is lost.

[0022] a block diagram showing an outline of the internal configuration of an endoscopic system including an object detection device according to an embodiment of the present invention; a flowchart explaining the operation of an endoscopic system including an object detection device according to an embodiment of the present invention; an example of a display screen of a display device in the endoscopic system of Figure 1; an example of a display screen of a display device at the time when a specific object is detected during an endoscopic examination using the endoscopic system of Figure 1; an example of a display screen of a display device at the time when a detected specific object disappears and a second image (reference image) is displayed during an endoscopic examination using the endoscopic system of Figure 1; an example of a display screen of a display device at the time when the same specific object is detected again during an endoscopic examination using the endoscopic system of Figure 1; an example of a display screen of a display device when the imaging unit is brought closer to the specific object in the state of Figure 7; an example of a display screen of a display device when the imaging unit is brought even closer to the specific object from the state of Figure 8; an explanatory diagram of an inference model obtained as a result of learning in a learning device; and a flowchart showing a modified example of the operation of an endoscopic system including an object detection device according to an embodiment of the present invention.

[0023] The present invention will be described below with reference to illustrated embodiments, taking an endoscope system as an example. The drawings used in the following description are schematic. Therefore, in these drawings, each component is shown at a size that allows it to be recognized. Therefore, the dimensional relationships and scales of the components in the drawings may be different for each component. The present invention is not limited to the illustrated embodiments with regard to the quantities, shapes, size ratios, relative positional relationships, and the like of the components shown in the drawings.

[0024] First, before describing the detailed configuration of an object detection device according to one embodiment of the present invention, the schematic configuration of an entire endoscope system including the object detection device of this embodiment will be described below with reference to Fig. 1. Fig. 1 is a configuration diagram that schematically shows the overall configuration of an endoscope system including an object detection device according to one embodiment of the present invention.

[0025] 1, the endoscope system 1 is mainly configured to include a processor 10, an endoscope 20, a display device 30, a light source device 40, and a storage device 50. This endoscope system 1 illustrates a general configuration of an endoscope system used for endoscopic examinations in which the inside of organs such as the upper digestive tract (esophagus, stomach, duodenum, etc.) or the lower digestive tract (large intestine, etc.) of a subject (patient, etc.) such as a living body is observed.

[0026] The endoscope 20 is an image inspection device that includes an insertion section 21, an operation section 22, a universal cable 23, and the like.

[0027] The insertion section 21 is a component that is inserted into a subject such as a living organism. The insertion section 21 is formed by continuously connecting a distal section 21a, a bending section 21b, and a flexible tubular section 21c, in that order from the distal end. The insertion section 21 is formed in a generally elongated tubular shape. The insertion section 21 has a treatment tool insertion channel 21d therein, which is a conduit for inserting an endoscopic treatment tool (not shown). This treatment tool insertion channel 21d is provided so as to pass through the insertion section 21 from the distal end to the proximal end. The operation section 22 is connected to the proximal end side of the insertion section 21.

[0028] The tip portion 21a is a structural unit provided at the most distal end of the insertion portion 21. Various structural members (not shown in FIG. 1; see FIG. 2) such as an illumination unit 25 and an imaging unit 26 are provided inside the tip portion 21a.

[0029] Here, the illumination unit 25 is a structural unit that includes an optical element (illumination lens; not shown) that emits a light beam guided from the light source device 40 (described later) forward from the tip surface of the tip portion 21 a to illuminate an observation area including a lesion or the like within the subject.

[0030] The imaging unit 26 is an electronic device unit including an optical lens (imaging optical system 26a; see FIG. 2) that forms an optical image of a specific object (a lesion such as a tumor) to be observed inside the subject, and a photoelectric conversion element (imaging element 26b; see FIG. 2) that generates image information (still image data, moving image data, etc.) based on the optical image.

[0031] The bending portion 21b is a tubular portion that can be actively bent by a bending operation mechanism (not shown) that acts in conjunction with the operation of a bending operation member 22b (described later). The flexible tube portion 21c is a tubular member that extends from the distal end of the operation portion 22 and is connected to the proximal end of the bending portion 21b.

[0032] The operation section 22 is connected to the proximal end of the insertion section 21. The operation section 22 is configured to include an operation section main body 22a, a bending operation member 22b, a plurality of operation members 22c, a treatment tool insertion port 22d, and the like.

[0033] The operation portion main body 22a has a generally box-like shape as a whole and constitutes a grip portion that is held with the fingers of a user (such as a doctor) of the endoscope 20. As described above, the insertion portion 21 extends from the operation portion main body 22a.

[0034] The bending operation member 22b and the plurality of operation members 22c are operation members for performing various operations of the endoscope 20. The bending operation member 22b and the plurality of operation members 22c are respectively provided at predetermined positions on the outer surface of the operation portion main body 22a.

[0035] The treatment instrument insertion port 22d is provided at a predetermined position near the distal end of the operation portion main body 22a. The treatment instrument insertion port 22d is a proximal end opening of the treatment instrument insertion channel 21d of the insertion portion 21. The treatment instrument insertion channel 21d is connected at its distal end to a distal channel opening (not shown) of the distal end portion 21a.

[0036] With this configuration, an endoscopic treatment tool (not shown) inserted through the treatment tool insertion port 22d is configured to pass through the treatment tool insertion channel 21d and then protrude outward from the tip channel opening of the tip portion 21a.

[0037] The universal cable 23 is a connection cord for connecting the endoscope 20 to the light source device 40 and the processor 10. For this purpose, the universal cable 23 is made of a tubular member extending from the side of the operation unit body 22a of the operation unit 22. A scope connector 23a is provided at the tip of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

[0038] An electric cable 23b extends from the scope connector 23a. A connector 23c is provided at the tip of the electric cable 23b. The connector 23c is connected to the front panel of the processor 10. Various signal transmission cables, optical fiber cables, etc. (not shown) are inserted into the universal cable 23.

[0039] The light source device 40 is a device that supplies illumination light to the illumination unit 25 (see FIG. 2) provided inside the tip portion 21a of the insertion section 21 of the endoscope 20. The illumination light emitted from the light source device 40 is transmitted to the illumination unit 25 in the tip portion 21a through an optical fiber cable (not shown) or the like that is arranged passing through the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21. The illumination light then passes through an illumination lens (not shown) included in the illumination unit 25 in the tip portion 21a and is irradiated toward the observation target area in front of the tip portion 21a.

[0040] The processor 10 is a control device, a signal processing device, or a circuit unit that includes a control circuit and a signal processing circuit that control the entire endoscope system 1. The processor 10 is also configured to include the function of the object detection device of this embodiment (described in detail below; see FIG. 2).

[0041] A control circuit included in the processor 10 receives, for example, an operation instruction signal from the operation member 22c of the operation section 22 of the endoscope 20, and outputs various control signals for driving and controlling the imaging unit 26, the light source device 40, the illumination unit 25, etc. Furthermore, a signal processing circuit included in the processor 10 receives, for example, an imaging signal from the imaging unit 26, and performs predetermined image signal processing, etc.

[0042] For this purpose, the processor 10 and the imaging unit 26 are electrically connected by a signal transmission cable (not shown). The signal transmission cable is inserted and arranged from the connector 23c through the electrical cable 23b, the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21 to the imaging unit at the tip end 21a.

[0043] With this configuration, control signals output from the processor 10 and imaging signals output from the imaging unit 26 are transmitted between the imaging unit 26 and the processor 10 through the signal transmission cable. Note that one form of the signal transmission cable is, for example, a composite cable in which a plurality of cables are bundled together and covered with an outer sheath shield, an outer sheath tube, or the like.

[0044] The display device 30 receives image signals and the like output from the processor 10 and displays endoscopic images and various types of information in a predetermined format. To this end, the display device 30 and the processor 10 are electrically connected using a video cable 24. The display device 30 may be configured as a display device using, for example, a general liquid crystal panel.

[0045] The storage device 50 is a storage device that saves (records or stores) image data for storage that has been generated for storage by various processes performed by the processor 10 based on image information generated by the imaging unit 26 .

[0046] As shown in Fig. 1, the storage device 50 is configured as an integral part of the processor 10. As shown in Fig. 2, a temporary storage unit 51 is provided inside the storage device 50 in addition to a main storage unit (not shown) in the storage device 50. The temporary storage unit 51 is configured, for example, by a semiconductor memory or the like, and is a component that functions as a temporary memory area for temporarily storing predetermined image data output from the image processing unit 11, etc.

[0047] 1 illustrates a configuration in which the storage device 50 is integrally disposed inside the housing of the processor 10, but the present invention is not limited to this. For example, the storage device 50 may be configured as an external storage device configured using a housing separate from the processor 10.

[0048] 2 shows an example of a configuration in which the temporary storage unit 51 is provided inside the storage device 50, but this is not limiting. The temporary storage unit 51 can also be provided integrally inside the storage control unit 13 (see FIG. 2), which will be described later, for example.

[0049] Furthermore, the processor 10 and the light source device 40 are not limited to being configured as separate entities, as in the example configuration shown in Fig. 1. For example, the processor 10 and the light source device 40 may be configured as an integrated unit using a single housing.

[0050] The configuration of the illumination unit 25 is not limited to the above-described configuration (a configuration in which illumination light from the light source device 40 is transmitted to the distal end portion 21a through an optical fiber cable or the like). As a configuration of the illumination unit 25 other than this, for example, a light-emitting element such as an LED (Light Emitting Diode) as an illumination light source can be provided inside the distal end portion 21a, and power supply to the illumination light source (LED) and light emission control thereof can be controlled by a predetermined control circuit included in the processor 10.

[0051] The endoscope system 1 configured as described above basically has a configuration substantially similar to that of a conventional endoscope system of the same type, and therefore, other detailed configurations will not be shown or described in detail.

[0052] In the endoscope system 1 configured as described above, the object detection device of this embodiment is configured to be included in the processor 10. Next, the internal configuration of the processor in the endoscope system including the object detection device of one embodiment of the present invention will be described in detail below with reference to FIG.

[0053] The processor 10 is configured to include an image processing unit 11, a display control unit 12, a memory control unit 13, a specific object image detection unit 14, an operation determination unit 15, a focus control unit 16, an illumination control unit 17, and a memory device 50 including a temporary memory unit 51.

[0054] The image processing unit 11 is a structural unit or circuit unit that receives image information output from the imaging unit 26 and performs various types of information processing based on the image information. The image processing performed here includes, for example, normal image information processing, image data processing for display, image data processing for storage, and various other types of image processing.

[0055] The normal image processing includes an AD conversion process that converts the analog image signal output from the image sensor 26b into a digital image signal, and is a basic image information processing that is normally performed on image information. The normal image processing is assumed to be a conventional process, and a detailed description thereof will be omitted.

[0056] The display image data processing is processing for generating image data for normal display, such as processing for generating first image data representing a first image (main endoscopic image) to be displayed in a first display area on the display screen of the display device 30, or processing for generating second image data (reference image data) representing a second image (reference image) to be displayed in a second display area different from the first display area on the same display screen.

[0057] FIG. 4 shows an example of a display screen of the display device in the endoscope system of this embodiment. In FIG. 4, reference numeral 100 denotes a display screen frame. Reference numeral 101 in FIG. 4 denotes a first display area. Reference numeral 102 in FIG. 4 denotes a second display area. Reference numeral 103 in FIG. 4 denotes an area for displaying information other than images. Reference numeral 104 in FIG. 4 denotes a predetermined alarm display area (described in detail later). The display example in FIG. 4 illustrates a case in which an image (reference numeral 101a; first image) is displayed only in the first display area 101.

[0058] The storage image data processing is processing to generate image data for recording (first image data, second image data, etc.).

[0059] The various types of image processing are image processing that are appropriately added to normal image data for display. Specific examples of the various types of image processing include normal image adjustment processing such as brightness adjustment processing and white balance adjustment processing, and image enhancement processing such as edge enhancement processing and texture and color enhancement imaging (TXI).

[0060] The display control unit 12 is a structural unit or circuit unit that controls the display device 30 so that an appropriate display format is displayed according to the display image data output from the image processing unit 11. The display control unit 12 controls the display of the display image data (various settings such as display position, display area, display size, etc.) as well as the display of various information other than image information (text information, icon information, etc.). Here, the various information other than image information includes examination date and time information, subject (patient) information, medical record information, etc., as well as various alarm information (notification information) that is displayed as appropriate during an endoscopic examination, for example.

[0061] Specifically, the display control unit 12 performs display control to continuously display the endoscopic image in the first display area, and when disappearance of a specific object is detected from the endoscopic image, the display control unit 12 performs display control to display a reference image based on the second image data in a predetermined area (second display area) of the display device 30.

[0062] The disappearance here refers to cases where the specific object is displayed on the screen but is overlooked by the operator without noticing, or where the operator loses sight of the object by unconsciously moving or operating the device, or where the object itself moves and becomes lost. In these cases, it is best to search for the object as soon as it disappears. This is because the area to be inspected is large, and if it is overlooked once, it will be time-consuming to inspect the same position again. This disappearance can be detected and an alarm can be issued. This can be done by applying various methods to alert the operator, such as sound or vibration.

[0063] Furthermore, when a specific object contained in the reference image that is identical to a specific object that disappeared from the endoscopic image is detected again in the endoscopic image by the specific object image detection unit 14, the display control unit 12 performs display control to display a notification in a predetermined area of ​​the display screen of the display device 30 (for example, the alarm display area 104 (see Figure 4)) that the same specific object contained in the reference image has been detected again.

[0064] The memory control unit 13 is a structural unit or circuit unit that controls the memory device 50 (including the temporary memory unit 51) so that the image data for storage output from the image processing unit 11 is saved (recorded or stored) in an appropriate storage format.

[0065] The storage control unit 13 performs storage control to permanently store first image data representing an endoscopic image acquired by the endoscope 20 in a storage area of ​​the storage device 50. The storage control unit 13 also performs storage control to temporarily store second image data (reference image data) including a specific object detected by a specific object image detection unit 14 (described later) in a temporary storage unit 51.

[0066] Furthermore, the memory control unit 13 also performs memory control to associate and store various information related to the second image data (e.g., information indicating the position of a specific object in an endoscopic image, unique characteristic information of the specific object (position, shape, size, color, etc.) etc.) with the corresponding second image data.

[0067] In this way, by storing the characteristic information of a specific object in association with the second image data, even if there are multiple specific objects within one frame of an endoscopic image, for example, each of the multiple specific objects can be identified and stored.

[0068] The specific object image detection unit 14 is a structural unit or circuit unit that detects an image area including a specific object from an endoscopic image based on the first image data acquired by the endoscope 20 .

[0069] In detail, the specific object image detection unit 14 detects a specific object (such as a lesion such as a tumor) within each frame of the endoscopic image that is displayed continuously based on the first image data output from the image processing unit 11.

[0070] The specific object image detection unit 14 detects specific objects, for example, by performing similar image recognition based on multiple specific object images (case images, etc.) prepared in advance, or by performing data analysis using deep learning or machine learning to detect the desired specific object.

[0071] For example, in recent years, object detection technology using AI (Artificial Intelligence) has been developed that obtains desired inference results (e.g., detection of specific objects such as tumors) through machine learning of endoscopic images acquired by an endoscope.

[0072] Machine learning is a method of obtaining inference results about unknown matters by learning the features, time-series information, spatial information, etc. of known input information and making inferences based on the learning results. That is, in machine learning, a trained model (also called an inference model) is first obtained that enables inference of a determinable output result from specific input information.

[0073] In this case, when generating a trained model, a large amount of information with known input-output relationships is used as training data to obtain highly reliable inference results. For example, in deep learning, a network is designed using a large amount of training data so that expected outputs can be obtained for known inputs. The trained model obtained through this process can be used independently of the network that was used for training.

[0074] Therefore, by providing the processor 10 with this type of inference model, it becomes possible to accurately and quickly detect specific objects in the specific object image detection unit 14, or to accurately and quickly determine endoscope operation in the operation determination unit 15, for example.

[0075] Here, the inference model is constructed roughly as follows. For example, a large amount of image data corresponding to input and output is provided as training data to a predetermined network (not shown) for constructing the inference model. Here, image data of an image (such as a case image) in which a specific object is captured in an endoscopic image is input as training data. In this case, an annotation may be set to surround the image area of ​​the specific object with a frame or the like.

[0076] By performing learning using a large amount of training data, a network design is determined for a specific network (not shown) so that an output corresponding to an input can be obtained. For example, Fig. 10 is an explanatory diagram of an inference model obtained as a result of learning in a specific learning device.

[0077] 10, when a group of endoscopic images (first training data group 201) including a specific object (such as a tumor) is input, information on the relative positional relationship between the object captured in the image (such as the lumen or inner wall of an organ) and the specific object, and additional information such as adding a frame display surrounding the image area of ​​the specific object, are obtained along with reliability information. This allows the construction of a first inference model 200 for detecting the specific object from the endoscopic image.

[0078] When endoscopic image data (first image data; moving image data) 202 taken during an endoscopic examination is input to the first inference model 200 constructed in this manner, the first inference model 200 detects an image area containing the specific object in the frame in which the specific object is captured from the endoscopic image data 202, and outputs image data in which a frame display is added to the image area of ​​the specific object (inference result image 203). This image 203 is temporarily stored as second image data and can be displayed in the second display area of ​​the display device 30 as needed, providing reference for the endoscope user during the endoscopic examination.

[0079] 10, when a group of search operation images (second training data group 204; short moving images) during an endoscopic examination is input, information regarding the search operation for a specific object during the endoscopic examination is obtained along with reliability information. This allows the construction of a second inference model 210 that determines the search operation for a specific object during the endoscopic examination.

[0080] When endoscopic image data (second image data; moving image data) 202 during an endoscopic examination is input to the second inference model 210 constructed in this manner, the second inference model 210 searches for and outputs a moving image (inference result moving image 205) including a search operation from the endoscopic image data 202. This moving image 205 is temporarily stored as second image data and can be displayed in the second display area of ​​the display device 30 as needed, providing reference for the endoscope user during the endoscopic examination.

[0081] "Deep learning" is a multilayered version of the "machine learning" process using neural networks. A typical example is a "forward propagation neural network," which sends information from front to back and makes a judgment. In its simplest form, it requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons determined by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights, and a bias value is added between the hidden and output layers, making it easy to form logic gates. While three layers are sufficient for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, models with 9 to 152 layers have become practical due to their training time, judgment accuracy, and energy consumption.

[0082] Various well-known networks may be used for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) using CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses image features, operates with minimal processing, and is strong in pattern recognition. Furthermore, "recurrent neural networks" (fully connected recurrent neural networks) that can handle more complex information and allow information analysis whose meaning changes depending on the order or sequence of information may be used.

[0083] To realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, devices specialized for matrix calculations such as GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) may also be used. In recent years, such dedicated artificial intelligence (AI) hardware "neural network processing units (NPUs)" have been designed to be integrated and embeddable with CPUs and other circuits, and may even become part of the processing circuit.

[0084] Furthermore, inference models may be obtained by employing various well-known machine learning techniques, not limited to deep learning. For example, techniques such as support vector machines and support vector regression are available. Here, learning involves calculating the weights, filter coefficients, and offsets of a classifier; other techniques include using logistic regression processing. When a machine is to make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method for deriving an image judgment using machine learning is employed. However, a rule-based method for applying rules acquired by humans through experience or heuristics to make a specific judgment may also be used.

[0085] Furthermore, when multiple specific objects are detected in one frame of an endoscopic image, for example, the specific object image detection unit 14 detects feature information for each of the specific objects. Each piece of feature information detected in this way is stored in the storage control unit 13 in association with the second image data of the corresponding specific object.

[0086] The operation determination unit 15 is a configuration unit or circuit unit that determines the operation status of the operation unit 22 in the endoscope 20. The operation determination unit 15 determines the operation status of the operation unit 22, for example, based on the first image data output from the image processing unit 11. In this case, as described above, the operation determination may be performed using the second inference model 210. Furthermore, the operation determination unit 15 may receive operation instruction signals from various operation members 22c provided in the operation unit 22, and determine the operation status of the operation unit 22 based on the received instruction signals.

[0087] The focus control unit 16 is a structural unit or circuit unit that controls the driving of the imaging optical system 26a of the imaging unit 26. The focus control unit 16 controls the driving of the imaging optical system 26a based on the detection result of the specific object image detection unit 14 and the determination result of the operation determination unit 15, and performs, for example, a predetermined automatic focus adjustment operation. When a specific object is found, focus control may be performed for that object. Focus control may also be performed according to the contents of a memory that records the characteristics of the object.

[0088] The illumination control unit 17 is a structural unit or circuit unit that controls the driving of the light source device 40 and the illumination unit 25. The illumination control unit 17 controls the light source device 40 or the illumination unit 25 in response to operation instruction signals from various operation members 22c provided in the operation unit 22, or instruction signals based on the detection results of the specific object image detection unit 14 or the determination results of the operation determination unit 15, and controls switching of the light source type (white light, special light, etc.).

[0089] Here, the types of light source include, for example, white light that is irradiated onto the subject when performing normal observation, as well as predetermined special light that is irradiated onto the subject when performing image-enhanced observation such as narrow band imaging (NBI) or red dichromatic imaging (RDI).

[0090] It should be noted that all or part of the processor 10 includes hardware. Here, the processor 10 is configured by a well-known configuration including, for example, a central processing unit (CPU), random access memory (RAM), read only memory (ROM), non-volatile memory, non-volatile storage, and a non-transitory computer readable medium, as well as peripheral devices thereof.

[0091] Software programs to be executed by the CPU and fixed data such as data tables are stored in advance in ROM, nonvolatile memory, nonvolatile storage devices, etc. The CPU reads out the software programs stored in ROM, etc., expands them into RAM, and executes them, and the software programs refer to various data, etc. as appropriate, thereby realizing the functions of the processor 10.

[0092] The processor 10 may also be configured with a semiconductor chip such as an FPGA (Field Programmable Gate Array), etc. Furthermore, the processor 10 may also be configured with an electronic circuit.

[0093] Furthermore, the software program may be recorded or stored, in whole or in part, as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.

[0094] The operation of the endoscope system 1 including the object detection device of this embodiment configured as described above will be described below. Fig. 3 is a flowchart explaining the operation of the endoscope system including the object detection device of one embodiment of the present invention. The flowchart shown in Fig. 3 is a diagram explaining the operation when an endoscopic examination is performed using the endoscope system 1.

[0095] First, it is assumed that the endoscope system 1 has been activated and is ready to perform an endoscopic examination. At this time, in step S1 of Fig. 3, the processor 10 checks whether the endoscope 20 in the endoscope system 1 is inserted into an organ or the like to be examined and is ready to start screening for the endoscopic examination, or whether screening is already being performed (either state will be referred to as a screening state).

[0096] Here, whether or not the endoscopic system 1 is in the screening state can be determined by, for example, confirming that the endoscopic system 1 has been started up in a predetermined procedure and that predetermined settings for performing an endoscopic examination have been made, or by checking the operation history after starting up the endoscopic system 1. The state of the endoscopic system 1 can also be determined by checking the image data of the endoscopic image acquired by the endoscope 20 itself or changes in the image data.

[0097] 4 shows an example of a display on the display screen of the display device 30 when screening is being performed during an endoscopic examination. The display example shown in FIG. 4 shows that a first image 101a (endoscopic image) is displayed in the first display area 101.

[0098] 3, if it is confirmed that the endoscope system 1 is in a screening state, the process proceeds to the next step S2. If it is confirmed that the endoscope system 1 is not in a screening state, the process proceeds to step S11. Note that, although screening is described here as a representative operation for searching for a specific object, it is not necessary to perform control from S2 onwards only in the screening state, and similar processing may be performed in other states.

[0099] In step S11, the processor 10 confirms an instruction to end the endoscopic examination. Here, the instruction to end the endoscopic examination is, for example, a predetermined examination end instruction signal generated by operating a predetermined operating member among the processor 10 of the endoscopic system 1 or the operating members 22c provided on the operating unit 22 of the endoscope 20. In addition, the instruction to end the endoscopic examination can also be determined by confirming image data of the endoscopic image acquired by the endoscope 20, for example.

[0100] If the instruction to end the examination is confirmed in the process of step S11, the series of processes is ended (END). If the instruction to end the examination is not confirmed, the process returns to the process of step S1.

[0101] When it is confirmed that the screening state is in progress and the process proceeds to step S2, the processor 10 checks in step S2 whether a specific object has been detected as a result of the ongoing screening. More specifically, it checks whether an image area including a specific object has been detected from the endoscopic image based on the first image data acquired by the endoscope 20.

[0102] If a specific object is detected, the process proceeds to step S3. If a specific object is not detected, the process proceeds to step S11. The process from step S11 onwards is as described above.

[0103] 5 shows an example of the display screen of the display device 30 at the time when a specific object is detected during an endoscopic examination. At this time, a first image 101a (endoscopic image) is displayed in the first display area 101. A specific object 101b is also displayed within the first image 101a displayed in the first display area 101. This specific object designated by the symbol 101b is the specific object detected in the processing of step S2 described above.

[0104] 5 denotes a frame display corresponding to the image region including the detected specific object 101b. By displaying the frame 106 in the first display region 101 in this way, it is clearly indicated to a user (such as a doctor) of the endoscope system 1 that the specific object 101b has been detected in the first image 101a. In this way, at the time of detection of the specific object 101b, for example, a display such as that shown in FIG. 5 is displayed.

[0105] In step S3 of FIG. 3, the processor 10 controls the storage control unit 13 to temporarily store, in the temporary storage unit 51, the second image data including the specific object image detected in the processing of step S2 described above.

[0106] Here, for example, if multiple image regions containing specific objects are detected, each of the detected multiple specific object images is stored as a separate file of second image data. At the same time, feature information for each of the multiple specific objects is acquired. Then, each acquired piece of feature information is stored in association with the corresponding second image data.

[0107] Furthermore, for example, if the same specific object is detected consecutively in the endoscopic image, the second image data is temporarily stored for each frame. In this case, it is not necessary to accumulate and store all of the second image data. For example, it is sufficient to store only the most recent second image data, or a predetermined period of second image data, such as the past few seconds, including the most recent image data, in the temporary storage area.

[0108] Next, in step S4, the processor 10 checks whether the specific object detected in the processing of step S2 has disappeared (lost sight) from the endoscopic image being displayed. If it is confirmed that the detected specific object has disappeared from the endoscopic image, the process proceeds to the next step S5. If it is not confirmed that the detected specific object has disappeared from the endoscopic image, the process proceeds to step S12.

[0109] In step S12, the processor 10 refers to an internal clock (not shown) or the like and waits for a predetermined time to elapse, and after the predetermined time has elapsed, proceeds to the processing of step S10.

[0110] At this point, the detected specific object continues to be displayed in the endoscopic image (the confirmation result of the processing in step S4 described above). Therefore, at this time, the user (doctor, etc.) of the endoscopic system 1 can fully observe the specific object in the endoscopic image being displayed. In this case, once it is confirmed in the processing in step S12 that the specific object has been displayed continuously for a predetermined period of time, the process proceeds to the processing in step S10, and the subsequent examination continues.

[0111] On the other hand, when the disappearance of a detected specific object is confirmed and processing proceeds to step S5, in this step S5, the processor 10 displays a second image based on the latest second image data temporarily stored in the processing of step S3 described above as a reference image in the second display area 102 on the display screen of the display device 30.

[0112] Figure 6 shows an example of the display screen of the display device 30 at the point in time (step S5) when the second image (reference image) is displayed in the second display area after the detected specific object has disappeared (step S4) during an endoscopic examination.

[0113] 6, the specific object 101b displayed in Fig. 5 has disappeared from the first image 101a (endoscopic image) displayed in the first display area 101. At this time, the display of the frame 106 also disappears at the same time as the specific object 101b disappears (step S4).

[0114] Simultaneously with the disappearance of the specific object 101b, a second image 102a is displayed in the second display area 102 (step S5). At this time, the second image 102a displayed in the second display area 102 is a still image in a reduced form of the first image 101a (see FIG. 5) immediately before the disappearance of the specific object 101b. This second image 102a is a second image based on the latest second image data temporarily stored (temporarily recorded) in the processing of step S3 described above. At this point (after the disappearance of the specific object 101b (S3) and at the point of time when the reference image is displayed (S4)), for example, the display will be as shown in FIG. 6.

[0115] 3, the processor 10 checks whether the same specific object has been detected again within a predetermined time (e.g., about 5 seconds). Here, the same specific object is assumed to be the specific object that was once detected in the endoscopic image in the processing of step S2 described above and that disappeared (lost sight or overlooked) from the endoscopic image in the processing of step S4 described above.

[0116] It may be better to prioritize the observation of this missed (disappeared) object when it is found again. This is because if the focus is not correct, there is a possibility that it will be missed again, and because rediscovery means that it is what was being searched for and is considered to be an important object. In other words, this need can be met by temporarily recording (temporarily storing) the image area containing the specific object contained in the reference image based on the specific object imaging result image data as a candidate for focusing.

[0117] If it is confirmed in step S6 that the same specific object has been detected again within the predetermined time, the process proceeds to step S7. If the same specific object has not been detected again, the process proceeds to step S10, where the subsequent inspections are continued.

[0118] In step S7, the processor 10 checks whether the same specific object detected again in the processing of step S6 described above has been continuously displayed for a predetermined time. If it is confirmed that the same specific object has been continuously displayed for the predetermined time, the processor 10 proceeds to the processing of the next step S8. If the same specific object has not been continuously displayed for the predetermined time, the processor 10 returns to the processing of step S4 described above.

[0119] In this case, when the same specific object has not been displayed continuously for a predetermined period of time, it is assumed that the specific object cannot be stably observed on the display screen, for example, when it has disappeared again, or when it has disappeared and been detected again momentarily, etc. In such a case, since the specific object cannot be observed sufficiently, it is assumed that it has disappeared again, and the process returns to step S4, and the same process is repeated.

[0120] 7 shows an example of the display screen of the display device 30 at the time when the same specific object is detected again during an endoscopic examination. In FIG. 7, the same specific object 101b is simultaneously displayed within the first image 101a displayed in the first display area 101. At this time, the frame 106 is also displayed simultaneously with the redetection of the specific object 101b. At this time, the display of the second image 102a in the second display area 102 is maintained.

[0121] At this time, a misalignment occurs between the display of the first image 101a and the display of the second image 102a. This means that after the specific object disappeared in the processing of step S4 described above, the user (doctor, etc.) of the endoscopic system 1 operated the endoscope 20 from the state shown in FIG. 5 to perform an operation for redetection, resulting in a change in the angle of the first image, etc. Therefore, in this case, it can be seen that the position of the redetected identical specific object 101b in the first display area 101 is different from the position of the specific object 101b in the second image 102a. In this way, even if the display position of the redetected specific object 101b on the screen is different, it is possible to determine that the redetected specific object 101b is the same specific object based on the feature information stored in association with the second image data.

[0122] In this way, when it is determined that the redetected specific object 101b is the same specific object, a predetermined notification display is made in a predetermined area on the display screen of the display device 30. For example, in the example shown in Fig. 7, a predetermined notification display, "! Redetected!", appears in the alarm display area 104. This notification may be assisted by audio.

[0123] This notification display notifies the user that the specific object 101b contained in the area surrounded by a frame 106 in the first display area 101 is the specific object that has been redetected this time, and that this specific object is the same specific object as the specific object that was detected before it disappeared.

[0124] It is desirable that the notification display be devised to use, for example, a blinking display or a colored display so as to clearly indicate the notification items to the user (doctor, etc.) of the endoscope system 1. At this point (the point at which the specific object 101b is redetected), for example, the display will be as shown in Fig. 7.

[0125] 3, the processor 10 controls the display control unit 12 to stop displaying the reference image currently being displayed in the second display area. At the same time, the processor 10 controls the focus control unit 16 to execute automatic focus adjustment control (autofocus control; AF control) with a specific object as a specific target. The automatic focus adjustment control in this case may include, for example, an object tracking function. Then, the process proceeds to step S9.

[0126] By executing automatic focus adjustment control in this manner, it becomes easier to reliably and constantly capture the same re-detected specific object within the field of view of the endoscopic image in a focused state.

[0127] In such a situation, the user (doctor, etc.) of the endoscope system 1 may perform an operation to observe the specific object in more detail, for example, by bringing the front surface of the distal end portion 21 a of the endoscope 20 (the objective lens of the imaging unit 26) closer to the specific object to be observed.

[0128] The closer the imaging unit 26 is to the specific object 101b, the larger the specific object 101b is displayed in the endoscopic image in the first display area 101, as shown in Fig. 8. In this case, too, an enlarged frame 106 is displayed in the area including the specific object 101b so as to surround the specific object 101b in accordance with the size of the specific object 101b.

[0129] 8, the display of the second image in the second display area 102 has stopped, as shown in the processing of step S8 described above. Here, the timing for stopping the display of the second image is, for example, the time when it is confirmed that the same specific object has been redetected and continued to be displayed for a predetermined period of time, as described in the processing of step S7 above, but is not limited to this. For example, the time when a predetermined operation of the operation unit 22 is confirmed based on the output of the operation determination unit 15 after the same specific object has been redetected may be the timing for stopping the display of the second image.

[0130] In this case, the predetermined operation of the operation section 22 may be, for example, an operation of moving the imaging unit 26 closer to the specific object 101b to be observed. This operation can be determined, for example, by receiving an instruction signal from an operation member or the like provided on the operation section 22 or an instruction signal from sensors provided on the endoscope 20. Alternatively, this can be determined based on changes over time in the endoscopic image acquired by the endoscope 20. That is, if the size of the specific object 101b to be observed is displayed gradually increasing over time in the endoscopic image, it can be determined that the imaging unit 26 is approaching the specific object 101b.

[0131] In general, in endoscopes, during normal observation such as screening, where the imaging unit and the object to be observed are somewhat distant (about a few centimeters), sufficient observation can be performed within the depth of field of the imaging optical system (so-called pan focus). However, as the imaging unit approaches the object to be observed, the depth of field becomes shallower, making it desirable to perform detailed focus control.

[0132] Therefore, in the endoscopic system 1 of this embodiment, as described above, after a specific object is redetected, automatic focus adjustment control is performed with the specific object as the specific target, so that even if an operation is performed to bring the imaging unit 26 closer to the specific object 101b, a clear image in an accurate focus state can always be obtained.

[0133] Therefore, even when the imaging unit 26 is moved closer to the specific object 101b from the state shown in Figure 8 and the display changes to the state shown in Figure 9, focus control continues, so the specific object 101b is displayed in an enlarged state and a clear image in an accurately focused state can always be obtained.

[0134] 9 shows an example of a display in which the surface of the enlarged specific object 101b, for example, capillaries, etc., can be observed (displayed). Fig. 9 shows an example of an endoscopic image displayed in such a state.

[0135] 3, in step S9, the processor 10 checks whether the individual detailed observation (inspection) of the specific object being observed has been completed. This check may be performed, for example, based on the output of the operation determination unit 15, and the time when a predetermined operation of the operation unit 22 is confirmed may be set as the observation (inspection) end timing.

[0136] In this case, the predetermined operation of the operation unit 22 may be, for example, an operation of moving the imaging unit 26 away from the specific object 101b to be observed. The determination of this operation is substantially the same as the determination of the above-mentioned operation of moving closer.

[0137] If it is confirmed in the process of step S9 that the detailed observation (inspection) of the specific object to be observed has been completed, the process proceeds to the next step S10. Furthermore, if the completion of the detailed observation (inspection) of the specific object has not been confirmed (if the detailed observation is continuing), the confirmation is repeated until the completion of the detailed observation (inspection) is confirmed.

[0138] When the end of detailed observation (inspection) of the individual specific object is confirmed in this way, the processor 10 then controls the focus control unit 16 in step S10 to switch the focus control setting of the current automatic focus adjustment control to the normal control setting, after which the process returns to step S1, and the process for searching for the next new specific object and observing (inspecting) it is repeated.

[0139] As described above, according to the present application, in an image inspection device such as an endoscope having a specific object image detection unit that detects an image area including a specific object based on acquired image data, the detected specific object imaging result image data is temporarily stored in a temporary storage unit, and when the disappearance of the specific object is detected, a display control unit that displays a reference image based on the specific object imaging result image data on a display unit that displays the acquired image data together, so that even if the specific object is overlooked, it can be quickly rechecked. Here, the form in which the reference image is displayed together may be displayed in different areas of the same display, or both displays may be partially overlapped, or may be displayed separately on different displays.

[0140] As described above, according to the above embodiment, in an endoscopic examination in which an image area of ​​a specific object such as a tumor is detected from an endoscopic image and the specific object is observed in detail, second image data including the detected specific object is temporarily stored. If the specific object disappears from the endoscopic image, a reference image based on the second image data is displayed as a reference image in the second display area of ​​the display device 30. Then, if the same specific object is detected again in the endoscopic image, automatic focus adjustment control is performed on the same specific object.

[0141] With this configuration, according to the endoscopic system 1 including the object detection device of one embodiment of the present invention, even if a specific object to be observed is lost during an endoscopic examination, it is possible to easily detect it again based on a reference image of the specific object before it disappeared.

[0142] Furthermore, if the same specific object is detected again, automatic focus adjustment control is performed on the specific object, so that the specific object can always be observed with a clear image in an accurately focused state.

[0143] Furthermore, when the same specific object is detected again after the specific object has disappeared, a notice is displayed to the effect that the specific object has been detected again, so that the user can easily notice that the same specific object has been detected again.

[0144] The specific object image detection unit 14 acquires feature information about the specific object, and the storage control unit 13 temporarily stores the feature information about the specific object in association with the second image data. With this configuration, even if multiple image areas containing a specific object are detected in a single frame, each of the detected multiple specific object images can be stored separately and referenced separately. This allows for more precise inspection.

[0145] In the above-described embodiment, in the reference image display process (step S5 in FIG. 3), an example is given in which a still image in a reduced form of the first image (the image temporarily stored as the second image in step S3) immediately before the disappearance of the specific object is displayed in the second display area of ​​the display device 30.

[0146] The form of the reference image is not limited to this example. For example, the second image may be in the form of a search support moving image display (navigation display) for re-searching for a lost specific object.

[0147] In this case, the second image displayed in the second display area may be a short video image of the period from the time the specific object is detected to the time it disappears, or for a few seconds after it disappears, or for a predetermined period of time (for example, a few seconds) prior to the time it disappears. Such short video image data is temporarily stored and is repeatedly displayed in the second display area.

[0148] By doing so, it is possible to replay the events leading up to the disappearance of the specific object. This may potentially clarify the cause of the disappearance of the specific object, such as the specific object being hidden by folds inside an organ due to peristaltic movement, or being covered by bleeding or food residue. Therefore, if the second image (reference image) is a short video image, it becomes easier to determine the direction of the search operation after the disappearance. This makes it easier to detect the same specific object again.

[0149] Furthermore, in the case of this playback display, for example, when a short video based on the second image data is played in the second display area, playback in reverse may be performed. This makes it easier to clarify the circumstances of the disappearance. This may make it even easier to redetect the same specific object.

[0150] On the other hand, in the above-described embodiment, after the redetection of the disappeared specific object (step S6 in FIG. 3), it is confirmed that the same specific object continues to be displayed for a predetermined period of time (step S7 in FIG. 3), and automatic focus adjustment control is executed (step S8 in FIG. 3).

[0151] The operational control after the re-detection of the lost specific object is not limited to the focus control described above. In the process of step S8 in Fig. 3, the following control may be performed instead of the focus control described above.

[0152] First, for example, the illumination control unit 17 may drive and control the illumination unit 25 and the light source device 40 to irradiate a region including a specific object with a predetermined special light. As a result, an image-enhanced observation image in a predetermined format is displayed in the first display region of the display device 30. Here, the image-enhanced observation image may be, for example, a narrowband observation image (NBI) or a red light observation image (RDI).

[0153] With this configuration, the specific object can be displayed more clearly by the image-enhanced observation image, making it difficult to lose sight of the specific object. At the same time, the image-enhanced observation image allows for more detailed observation.

[0154] Secondly, for example, control may be considered in which the image processing unit 11 executes a predetermined image adjustment process or a predetermined image enhancement process. As a result, an image reflecting the results of the predetermined image processing or the like is displayed in the first display area of ​​the display device 30. Here, the image adjustment process may be, for example, a brightness adjustment process, a white balance adjustment process, or the like. Furthermore, the image enhancement process may be, for example, an edge enhancement process, a structural color enhancement process, or the like.

[0155] In addition, the control timing of the illumination control unit 17 or the processing start timing of the image processing unit 11 may be, as described above, when a specific object disappears during an endoscopic examination and then the same specific object is detected again and continues to be displayed, or may be, for example, when it is determined that a detailed observation operation is being performed on the specific object.

[0156] Here, the detailed observation operation of the specific object may be, for example, an operation of moving the tip 21 a (imaging unit 26) of the endoscope 20 closer to the re-detected specific object. Specifically, for example, when an operation of moving the tip 21 a from its current position toward the position where the tip 21 a was located at the time the second image data was acquired is detected, it is determined that an operation of moving the imaging unit 26 closer to the specific object is being performed.

[0157] As described above, when an operation such as moving the imaging unit 26 of the distal end 21 a of the endoscope 20 closer to the redetected specific object is detected, it is determined that an attempt is being made to perform a detailed observation operation on the specific object. At this time, in the above-described embodiment, focus control is executed, and automatic focus adjustment control is performed on the specific object (step S8 in FIG. 3).

[0158] In this case, if it is determined that the imaging unit 26 has approached within a predetermined distance from the specific object, the focus control may be switched to a magnification observation mode. A part of the processing sequence in this case is shown in Fig. 11. Fig. 11 is a flowchart showing the operation of a modified version of this embodiment, which shows a part of the processing sequence in Fig. 3.

[0159] First, in Fig. 11, the processes of steps S1 to S6 are omitted from the illustration because they are the same as those of Fig. 10. Furthermore, the processes of steps S7 and S8 in Fig. 11 are the same as those of steps S7 and S8 in Fig. 10.

[0160] That is, in step S7, the processor 10 confirms that the same specific object that has been redetected has been continuously displayed for a predetermined time. Next, in step S8, the processor 10 stops displaying the reference image in the second display area and executes focus control for automatic focus adjustment.

[0161] Next, in the next step S8a, the processor 10 determines whether or not an operation of bringing the imaging unit 26 closer to the specific object within a predetermined distance (for example, about 3 millimeters) has been performed, based on the output of the operation determination unit 15. If an operation of bringing the imaging unit 26 closer to the specific object within the predetermined distance has been determined, the process proceeds to step S8b. If an operation of bringing the imaging unit 26 closer to the specific object within the predetermined distance has not been determined, the process proceeds to step S9.

[0162] In step S8b, the processor 10 controls the focus control unit 16 to set the focus control appropriate for the predetermined magnification observation mode. Then, the process proceeds to step S9. The other processing steps are substantially the same as those in FIG. 10.

[0163] With this configuration, even when further magnification observation is performed, a clear image in a correct focus state can always be obtained, allowing for more detailed observation of a specific object with a better image.

[0164] The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the spirit and scope of the invention. For example, while an endoscope has been described as an example, the invention can be applied to any device that sequentially acquires and evaluates images (such as an image inspection device). Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple components. For example, if the problem to be solved by the invention can be solved and the effects of the invention can be obtained even if some components are deleted from all the components shown in one embodiment, the configuration from which these components are deleted can be extracted as the invention. Furthermore, components from different embodiments may be appropriately combined. The present invention is not limited by specific embodiments other than as limited by the appended claims.

Claims

1. An object detection device comprising: a specific object image detection unit that detects an image region including a specific object from an endoscopic image based on first image data acquired by an endoscope; a storage control unit that temporarily stores second image data including the detected specific object in a temporary storage unit; and a display control unit that, when disappearance of the specific object is detected from the endoscopic image, displays a reference image based on the second image data on a display device.

2. The object detection device according to claim 1, wherein the display control unit displays the reference image in a second display region different from a first display region in which the endoscopic image is displayed within a display screen of the display device.

3. The object detection device according to claim 2, wherein, when the specific object included in the reference image, which is the same specific object that has disappeared from the endoscopic image, is redetected in the endoscopic image by the specific object image detection unit, the display control unit causes the display device to display a notification indicating that the same specific object included in the reference image has been redetected.

4. The object detection device according to claim 2, wherein the storage control unit temporarily stores, as the second image data, moving image data of an image region including the specific object from a time point a certain time before the disappearance of the specific object from the endoscopic image in the temporary storage unit, and the display control unit reproduces and displays the moving image data based on the second image data in the second display region.

5. The object detection device according to claim 2, further comprising a focus control unit that drives and controls an imaging optical system of an imaging unit included in the endoscope, wherein, when the specific object image detection unit redetects, in the endoscopic image, a specific object that has disappeared from the endoscopic image and is the same specific object included in the reference image, and detects that the same specific object is continuously displayed in the endoscopic image, the focus control unit drives and controls the imaging optical system to perform automatic focus adjustment control for the specific object.

6. The object detection device further includes an illumination control unit that drives and controls an illumination unit and a light source device included in the endoscope. When the specific object detection unit redetects, within the endoscope image, a specific object that has disappeared from the endoscope image and is the same as the specific object included in the reference image, and detects that the same specific object is continuously displayed in the endoscope image, the illumination control unit drives and controls the illumination unit and the light source device to irradiate special light on the region including the specific object. The display control unit displays an image enhancement observation image in the first display area. The object detection device according to claim 2, characterized in that.

7. The object detection device according to claim 6, characterized in that the image enhancement observation image is a narrow-band light observation image (NBI) or a red light observation image (RDI).

8. The object detection device further includes an image processing unit that performs image information processing on the first image data representing the endoscope image, and an operation determination unit that determines the operation status of the operation unit of the endoscope. When the specific object detection unit redetects, within the endoscope image, a specific object that has disappeared from the endoscope image and is the same as the specific object included in the reference image, and detects that the same specific object is continuously displayed in the endoscope image, or when the operation determination unit determines that an observation operation on the specific object is being performed, in either case, the image processing unit executes image adjustment processing or image enhancement processing, and the display control unit displays, in the first display area of the display device, an image output as a result of the image processing by the image processing unit. The object detection device according to claim 2, characterized in that.

9. The image adjustment processing executed by the image processing unit is brightness adjustment processing and white balance adjustment processing, and the image enhancement processing executed by the image processing unit is contour enhancement processing and structural color enhancement processing. The object detection device according to claim 8, characterized in that.

10. The operation determination unit determines a detailed observation operation of the specific object based on the change over time of the image area including the specific object in the endoscope image. The object detection device according to claim 8, characterized in that.

11. The specific object image detection unit further acquires feature information regarding the specific object, and the storage control unit temporarily stores the feature information regarding the specific object in association with the second image data. The object detection apparatus according to claim 1, characterized in that.

12. The object detection apparatus according to claim 2, further comprising: a focus control unit that drives and controls an imaging optical system of an imaging unit included in the endoscope; and an operation determination unit that determines an operation state of an operation unit of the endoscope. When the operation determination unit detects an operation of moving the distal end portion of the endoscope toward a position where the distal end portion of the endoscope existed at a time point when the specific object image detection unit redetected the same specific object in the endoscope image and the second image data was acquired, the focus control unit drives and controls the imaging optical system to perform automatic focus adjustment control for the specific object.

13. The object detection apparatus according to claim 2, wherein the specific object image detection unit has a first inference model constructed using a large amount of specific object image data as teacher data and annotated by a frame display surrounding an image region including the specific object.

14. The object detection apparatus according to claim 8, wherein the operation determination unit has a second inference model constructed using a large amount of moving image data configured by arranging a plurality of endoscope images sequentially acquired during an endoscope examination performed using the endoscope in a time series as teacher data and annotated with a search operation of the specific object performed when the specific object detected during the endoscope examination disappears.

15. A screening step of continuously displaying endoscopic images obtained continuously by an endoscope in a time series, a detection step of detecting a specific object from the endoscopic images, a temporary storage step of temporarily storing image data corresponding to an image area including the detected specific object, a reference image display step of displaying, as a reference image, an image area including the specific object when disappearance of the specific object is detected from the endoscopic images, and a focus control step of ending the display of the reference image and performing autofocus control for the specific object when the same specific object as the specific object that has disappeared from the endoscopic images is detected again in the endoscopic images. A method for detecting an object, characterized by comprising the steps.

16. A computer is caused to execute a process of continuously displaying endoscopic images, a process of detecting a specific object from the endoscopic images, a process of temporarily storing image data corresponding to an image area including the detected specific object, a process of displaying, as a reference image, an image area including the specific object when disappearance of the specific object is detected from the endoscopic images, and a process of ending the display of the reference image and performing autofocus adjustment for the specific object when the same specific object as the specific object that has disappeared from the endoscopic images is detected again in the endoscopic images. An object detection program, characterized by causing the above processes to be executed.

17. An endoscope system comprising an endoscope including an imaging optical system for forming an optical image of an object and an imaging element for photoelectrically converting the optical image formed by the imaging optical system, and a processor, wherein the processor includes a specific object image detection unit for detecting an image region including a specific object from an endoscope image based on first image data acquired by the imaging unit, a storage control unit for temporarily storing second image data including the detected specific object in a temporary storage unit, a display control unit for displaying a reference image based on the second image data in a second display region different from a first display region where the endoscope image is displayed within a display screen of a display device when disappearance of the specific object is detected from the endoscope image, and a focus control unit for driving and controlling the imaging optical system, and when the specific object image detection unit detects that the same specific object that has disappeared from the endoscope image is detected again in the endoscope image and the same specific object is continuously displayed in the endoscope image, the focus control unit drives and controls the imaging optical system to perform autofocus adjustment control for the specific object. An endoscope system characterized by the above.

18. Detecting an image region including a specific object based on image data acquired by an image inspection device, temporarily storing specific object imaging result image data including the detected specific object in a temporary storage unit, and when disappearance of the specific object is detected from an image based on the image data, also displaying a reference image based on the specific object imaging result image data on a display unit that displays the image based on the acquired image data. A target object detection method characterized by the above.

19. Further, the moving image data of the image region including the specific object from a time point a certain time before the disappearance of the specific object from the endoscope image is temporarily stored as disappearance process moving image data. The object detection method according to claim 18, characterized by the above.

20. Further, an image region including a specific object included in a reference image based on the specific object imaging result image data is temporarily recorded as a focusing candidate. The object detection method according to claim 18, characterized by the above.

Images