Focus control device, focus control method, focus control program, and endoscope system

Abstract

In order to provide a focus control device capable of performing automatic control switching to enable proper focus control of an imaging unit according to the usage state of equipment such as an endoscope, the present invention comprises: a time-series change determination unit (15a) that detects a time-series image change pattern of an endoscopic image based on a plurality of pieces of image data which are sequentially acquired in chronological order by an imaging unit (26) included at the tip end of an insertion part (21) of an endoscope (1); a usage state determination unit (15) that determines the usage state of the endoscope on the basis of the time-series image change pattern detected by the time-series change determination unit; and a focus control unit (16) that drives and controls the imaging optical system of the imaging unit. The focus control unit switches to focus control according to the determination result of the usage state determination unit.

Description

Focus control device, focus control method, focus control program, and endoscope system

[0001] The present invention relates to a focus control device, a focus control method, a focus control program, and an endoscope system that automatically perform appropriate focus control according to the state of use of an endoscope.

[0002] Conventionally, endoscopic systems that include an endoscope that captures images of the inside of a subject and acquires image data, a processor that performs various types of image processing on the image data acquired by the endoscope, a display device that displays the image data processed by the processor as a visible image, and a storage device that records or stores the image data have been widely used in, for example, the medical field and the industrial field.

[0003] Furthermore, in the medical field, various examinations are widely performed using this type of endoscope system to observe the inside of biological organs, etc. In an examination using a medical endoscope system (hereinafter referred to as an endoscopic examination), for example, an imaging unit provided at the tip of an endoscope inserted into a lumen of a biological organ, etc., is used to capture images while the insertion section of the endoscope is advanced and retreated, such as by being withdrawn along the length of the lumen. At this time, the endoscopic images acquired by the imaging unit are displayed as moving images in real time chronologically on a display device. At the same time, the moving image data can also be stored in a storage device.

[0004] During such an endoscopic examination, the doctor or other equipment operator performs so-called screening, searching for lesions such as polyps or tumors occurring on the inner walls of organs as specific objects (hereinafter referred to as specific objects) while observing the real-time endoscopic image displayed on the display device (or, after the examination, a replayed endoscopic image based on recorded or stored endoscopic image data).

[0005] Normally, specific objects such as lesions are very small and have similar colors to the inner walls of surrounding organs, so skill is required to accurately distinguish and detect the image area of ​​the specific object from an endoscopic image. In addition, in endoscopic examinations, the operation of the endoscope itself, such as the operation of pointing the tip of the insertion part of the endoscope (the observation window of the imaging unit) at the desired object and capturing an image, requires skill.

[0006] Incidentally, the use states of an endoscope during an endoscopic examination can be broadly divided into, for example, a state in which the above-mentioned screening is being performed (referred to as the screening state), a state in which a detailed observation of a specific detected object is being performed (referred to as the detailed observation state), and a state in which the tip of the endoscope is in the process of transitioning from the screening state to a specific object to be observed in order to perform detailed observation (referred to as the approach state).

[0007] When performing endoscopic examinations using conventional medical or testing equipment such as endoscopes, it is necessary to perform appropriate focus control of the imaging unit depending on the usage conditions of the endoscope, etc., in order to always ensure a reliable observation environment.

[0008] However, in conventional devices such as endoscopes, it is common to use an imaging unit with a so-called deep focus setting, which is set to obtain a good focus state within a predetermined imaging range without performing precise focus control and relies on the characteristics of the imaging optical system.

[0009] This is because the lens of an endoscope is a wide-angle lens, making it easy to obtain images with good focus over a relatively wide range of distances, and the distance to the area observed by the endoscope only needs to be limited to a relatively short distance that does not need to include a range like infinity for a camera, so the positional relationship between the imaging optical system and the imaging element can be set so that the focus is at this normal distance (about a few centimeters).

[0010] Furthermore, in some conventional devices, for example, the focus control of the imaging unit is selectively switched by an operator through manual operation as needed.

[0011] On the other hand, conventional medical devices or testing devices have been disclosed, for example in Japanese Patent No. 6833870, which have a function of determining the usage state of the device based on a signal from an endoscope or the like.

[0012] It would be extremely convenient if the device's usage status could be determined and the image capture unit's focus control and other controls could be automatically switched in accordance with the determined usage status. However, no such device has existed until now.

[0013] An object of the present invention is to provide a focus control device, a focus control method, a focus control program, and an endoscope system that can automatically switch control so that appropriate focus control of an imaging unit can be performed depending on the usage status of the equipment in medical equipment such as endoscopes and inspection equipment (image inspection equipment, etc.).

[0014] In order to achieve the above object, a focus control device of one aspect of the present invention comprises a time series change determination unit that detects a time series image change pattern of an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of an endoscope insertion portion, a usage state determination unit that determines the usage state of the endoscope based on the time series image change pattern detected by the time series change determination unit, and a focus control unit that drives and controls the imaging optical system of the imaging unit, and the focus control unit switches to focus control according to the determination result by the usage state determination unit.

[0015] A focus control method according to one aspect of the present invention detects a time-series image change pattern of an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of an endoscope insertion portion, determines the usage state of the endoscope based on the detection result of the time-series image change pattern, and switches to focus control according to the determination result of the usage state of the endoscope.

[0016] A focus control program according to one aspect of the present invention causes a computer to execute a time series change determination process for detecting a time series image change pattern of an endoscopic image based on a plurality of image data acquired sequentially in time series, a usage state determination process for determining the usage state of the endoscope based on the detection result of the time series image change pattern, and a process for switching to focus control according to the usage state determination result of the endoscope.

[0017] An endoscopic system according to one aspect of the present invention comprises an endoscope including an imaging unit having an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system, and a processor. The processor is equipped with a time series change determination unit that detects a time series image change pattern of an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of an endoscope insertion section, a usage state determination unit that determines the usage state of the endoscope based on the time series image change pattern detected by the time series change determination unit, and a focus control unit that drives and controls the imaging optical system of the imaging unit, and the focus control unit switches to focus control according to the determination result by the usage state determination unit.

[0018] A second aspect of the focus control device of the present invention comprises a specific object image detection unit that detects an image area including a specific object from an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of an endoscope insertion portion, a vascular determination unit that determines the vascular pattern of the specific object, and a focus control unit that drives and controls an imaging optical system that forms an optical image of the object, and the focus control unit drives and controls the imaging optical system to a focus position corresponding to the distance from the pan focus position to the object in accordance with the output result determined by the vascular determination unit based on the endoscopic image acquired when the imaging optical system is set to a pan focus position.

[0019] A focus control method of a second aspect of the present invention detects an image area including a specific object from an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of an endoscope insertion portion, determines the vascular pattern of the specific object, and drives and controls the imaging optical system to a focus position corresponding to the distance from the pan-focus position to the object in accordance with the output result determined by the vascular determination unit based on the endoscopic image acquired when the imaging optical system is set to a pan-focus position.

[0020] A third aspect of the focus control method of the present invention detects a time-series image change surface pattern of an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of the endoscope insertion portion, detects an image of a specific object included in the endoscopic image, determines the usage state of the endoscope based on the detection result of the change pattern of the specific object in the time-series image change, and switches the imaging optical system of the endoscope from a pan-focus state to a different focus control.

[0021] A third aspect of the focus control device of the present invention comprises a time series change determination unit that determines the time series image change pattern of an endoscopic image based on multiple image data acquired sequentially in time series by an imaging unit included in the tip of the endoscope insertion portion, a specific object image detection unit that detects a specific object image included in the endoscopic image, a usage state determination unit that determines the usage state of the endoscope based on the detection result of the change pattern of the specific object in the time series image change pattern, and a focus control unit that switches the imaging optical system of the endoscope from a pan focus state to a different focus control.

[0022] According to the present invention, it is possible to provide a focus control program and an endoscope system that can automatically switch control so that appropriate focus control of an imaging unit can be performed depending on the usage status of the equipment in medical equipment such as endoscopes and inspection equipment (image inspection devices, etc.).

[0023] FIG. 1 is a block diagram showing an outline of the internal configuration of an endoscope system including a focus control device according to an embodiment of the present invention; FIG. 2 is a flowchart illustrating the operation of an endoscope system including a focus control device according to an embodiment of the present invention; FIG. 3 is a schematic diagram showing a first state (during normal search) when an endoscope of the endoscope system of FIG. 1 is in a first observation state (screening state); a schematic diagram showing the appearance of an endoscope of the endoscopic system when it is in the second observation state (approach state); an example of a display screen of a display device when the endoscope is in the state of Figure 8; a schematic diagram showing the appearance of an endoscope of the endoscopic system of Figure 1 when it is in the third observation state (detailed observation state); an example of a display screen of a display device when the endoscope is in the state of Figure 10; an explanatory diagram of the first and second inference models obtained as a result of learning in the learning device; an explanatory diagram of the third and fourth inference models obtained as a result of learning in the learning device; a flowchart showing a modified example of the operation of an endoscopic system including a focus control device of one embodiment of the present invention; and a schematic diagram conceptually showing the appearance of performing close autofocus control in an endoscopic system including a focus control device of one embodiment of the present invention.

[0024] The present invention will be described below with reference to illustrated embodiments, for example, as an application example to an endoscope system. The drawings used in the following description are schematic. Therefore, in these drawings, each component is shown at a size that allows it to be recognized. Therefore, the dimensional relationships and scales of each component may be shown differently for each component. The present invention is not limited to the illustrated embodiments with respect to the quantities, shapes, size ratios, relative positional relationships, and the like of each component shown in the drawings.

[0025] First, before describing the detailed configuration of a focus control device according to one embodiment of the present invention, the schematic configuration of an entire endoscope system including the focus control device of this embodiment will be described below with reference to Fig. 1. Fig. 1 is a diagram illustrating the schematic configuration of an entire endoscope system including the focus control device of one embodiment of the present invention.

[0026] 1, the endoscope system 1 is mainly configured to include a processor 10, an endoscope 20, a display device 30, a light source device 40, and a storage device 50. This endoscope system 1 illustrates a general configuration of an endoscope system used for endoscopic examinations in which the inside of organs such as the upper digestive tract (esophagus, stomach, duodenum, etc.) or the lower digestive tract (large intestine, etc.) of a subject (patient, etc.) such as a living body is observed.

[0027] The endoscope 20 is an image inspection device that includes an insertion section 21, an operation section 22, a universal cable 23, and the like.

[0028] The insertion section 21 is a component that is inserted into a subject such as a living organism. The insertion section 21 is formed by continuously connecting a distal section 21a, a bending section 21b, and a flexible tubular section 21c, in that order from the distal end. The insertion section 21 is formed in a generally elongated tubular shape. The insertion section 21 has a treatment tool insertion channel 21d therein, which is a conduit for inserting an endoscopic treatment tool (not shown). This treatment tool insertion channel 21d is provided so as to pass through the insertion section 21 from the distal end to the proximal end. The operation section 22 is connected to the proximal end side of the insertion section 21.

[0029] The tip portion 21a is a structural unit provided at the most distal end of the insertion portion 21. Various structural members (not shown in FIG. 1; see FIG. 2) such as an illumination unit 25 and an imaging unit 26 are provided inside the tip portion 21a.

[0030] Here, the illumination unit 25 is a structural unit that includes an optical element (illumination lens; not shown) that emits a light beam guided from the light source device 40 (described later) forward from the tip surface of the tip portion 21 a to illuminate an observation area including a lesion or the like within the subject.

[0031] The imaging unit 26 is an electronic device unit including an optical lens (imaging optical system 26a; see FIG. 2) that forms an optical image of a specific object (a lesion such as a tumor) to be observed inside the subject, and a photoelectric conversion element (imaging element 26b; see FIG. 2) that generates image information (still image data, moving image data, etc.) based on the optical image.

[0032] The bending portion 21b is a tubular portion that can be actively bent by a bending operation mechanism (not shown) that acts in conjunction with the operation of a bending operation member 22b (described later). The flexible tube portion 21c is a tubular member that extends from the distal end of the operation portion 22 and is connected to the proximal end of the bending portion 21b.

[0033] The operation section 22 is connected to the proximal end of the insertion section 21. The operation section 22 is configured to include an operation section main body 22a, a bending operation member 22b, a plurality of operation members 22c, a treatment tool insertion port 22d, and the like.

[0034] The operation unit main body 22a has a generally box-like shape as a whole and constitutes a gripping portion that is held with the fingers of a user (such as a doctor) of the endoscope 20. Various constituent units such as a bending operation mechanism (not shown) are provided inside the operation unit main body 22a. As described above, the insertion section 21 extends from the operation unit main body 22a.

[0035] The bending operation member 22b and the plurality of operation members 22c are operation members for performing various operations of the endoscope 20. The bending operation member 22b and the plurality of operation members 22c are respectively provided at predetermined positions on the outer surface of the operation portion main body 22a.

[0036] The treatment instrument insertion port 22d is provided at a predetermined position near the distal end of the operation portion main body 22a. The treatment instrument insertion port 22d is a proximal end opening of the treatment instrument insertion channel 21d of the insertion portion 21. The treatment instrument insertion channel 21d is connected at its distal end to a distal channel opening (not shown) of the distal end portion 21a.

[0037] With this configuration, an endoscopic treatment tool (not shown) inserted through the treatment tool insertion port 22d is configured to pass through the treatment tool insertion channel 21d and then protrude outward from the tip channel opening of the tip portion 21a.

[0038] The universal cable 23 is a connection cord for connecting the endoscope 20 to the light source device 40 and the processor 10. For this purpose, the universal cable 23 is made of a tubular member extending from the side of the operation unit body 22a of the operation unit 22. A scope connector 23a is provided at the tip of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

[0039] An electric cable 23b extends from the scope connector 23a. A connector 23c is provided at the tip of the electric cable 23b. The connector 23c is connected to the front panel of the processor 10. Various signal transmission cables, optical fiber cables, etc. (not shown) are inserted into the universal cable 23.

[0040] The light source device 40 is a device that supplies illumination light to the illumination unit 25 (see FIG. 2) provided inside the tip portion 21a of the insertion section 21 of the endoscope 20. The illumination light emitted from the light source device 40 is transmitted to the illumination unit 25 in the tip portion 21a through an optical fiber cable (not shown) or the like that is arranged passing through the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21. The illumination light then passes through an illumination lens (not shown) included in the illumination unit 25 in the tip portion 21a and is irradiated toward the observation target area in front of the tip portion 21a.

[0041] The processor 10 is a control device, a signal processing device, or a circuit unit that includes a control circuit and a signal processing circuit that control the entire endoscope system 1. The processor 10 is also configured to include a function as a focus control device of this embodiment (described in detail below; see FIG. 2).

[0042] A control circuit included in the processor 10 receives, for example, an operation instruction signal from the operation member 22c of the operation section 22 of the endoscope 20, and outputs various control signals for driving and controlling the imaging unit 26, the light source device 40, the illumination unit 25, etc. Furthermore, a signal processing circuit included in the processor 10 receives, for example, an imaging signal from the imaging unit 26, and performs predetermined image signal processing, etc.

[0043] For this purpose, the processor 10 and the imaging unit 26 are electrically connected by a signal transmission cable (not shown). The signal transmission cable is inserted and arranged from the connector 23c through the electrical cable 23b, the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21 to the imaging unit at the tip end 21a.

[0044] With this configuration, control signals output from the processor 10 and imaging signals output from the imaging unit 26 are transmitted between the imaging unit 26 and the processor 10 through the signal transmission cable. Note that one form of the signal transmission cable is, for example, a composite cable in which a plurality of cables are bundled together and covered with an outer sheath shield, an outer sheath tube, or the like.

[0045] The display device 30 receives image signals and the like output from the processor 10 and displays endoscopic images and various types of information in a predetermined format. To this end, the display device 30 and the processor 10 are electrically connected using a video cable 24. The display device 30 may be configured as a display device using, for example, a general liquid crystal panel.

[0046] The storage device 50 is a storage device that saves (records or stores) image data for storage that is generated for storage after various processes are performed in the processor 10 based on image information generated by the imaging unit 26 .

[0047] As shown in Fig. 1, the storage device 50 is configured as an integral part of the processor 10. As shown in Fig. 2, a temporary storage unit 51 is provided inside the storage device 50 in addition to a main storage unit (not shown) in the storage device 50. The temporary storage unit 51 is configured, for example, by a semiconductor memory or the like, and is a component that functions as a temporary memory area for temporarily storing predetermined image data output from the image processing unit 11, etc.

[0048] 1 illustrates a configuration in which the storage device 50 is integrally disposed inside the housing of the processor 10, but the present invention is not limited to this. For example, the storage device 50 may be configured as an external storage device configured using a housing separate from the processor 10.

[0049] 2 shows an example of a configuration in which the temporary storage unit 51 is provided inside the storage device 50, but this is not limiting. The temporary storage unit 51 can also be provided integrally inside the storage control unit 13 (see FIG. 2), which will be described later, for example.

[0050] Furthermore, the processor 10 and the light source device 40 are not limited to being configured as separate entities, as in the example configuration shown in Fig. 1. For example, the processor 10 and the light source device 40 may be configured as an integrated unit using a single housing.

[0051] The configuration of the illumination unit 25 is not limited to the above-described configuration (a configuration in which illumination light from the light source device 40 is transmitted to the distal end portion 21a through an optical fiber cable or the like). As a configuration of the illumination unit 25 other than this, for example, a light-emitting element such as an LED (Light Emitting Diode) as an illumination light source can be provided inside the distal end portion 21a, and power supply to the illumination light source (LED) and light emission control thereof can be controlled by a predetermined control circuit included in the processor 10.

[0052] The endoscope system 1 configured as described above basically has substantially the same configuration as conventional endoscope systems of the same type, and therefore, illustrations and detailed descriptions of other detailed configurations will be omitted.

[0053] In the endoscope system 1 configured as described above, the focus control device of this embodiment is configured as being included in the processor 10. Next, the internal configuration of the processor in the endoscope system including the focus control device of one embodiment of the present invention will be described in detail below with reference to FIG.

[0054] The processor 10 is configured to include an image processing unit 11, a display control unit 12, a memory control unit 13, a specific object image detection unit 14, a usage state determination unit 15, a focus control unit 16, an illumination control unit 17, and a memory device 50 including a temporary memory unit 51.

[0055] The image processing unit 11 is a structural unit or circuit unit that receives image information output from the imaging unit 26 and performs various types of information processing based on the image information. The image processing performed here includes, for example, normal image information processing, image data processing for display, image data processing for storage, and various other types of image processing.

[0056] The normal image processing includes an AD conversion process that converts the analog image signal output from the image sensor 26b into a digital image signal, and is a basic image information processing that is normally performed on image information. The normal image processing is assumed to be a conventional process, and a detailed description thereof will be omitted.

[0057] The display image data processing is a process for generating image data for normal display, such as image data representing an endoscopic image to be displayed in a predetermined display area on the display screen of the display device 30.

[0058] Fig. 5 shows an example of a display screen of the display device in the endoscope system of this embodiment. In Fig. 5, reference numeral 100 denotes a display screen frame. Reference numeral 101 in Fig. 5 denotes a display area for an endoscopic image 101a. Reference numeral 103 in Fig. 5 denotes a display area for information other than the image. Reference numeral 104 in Fig. 5 denotes a predetermined information display area (described in detail later). The storage image data processing is processing for generating image data for recording, etc.

[0059] The various types of image processing are image processing that are appropriately added to normal image data for display. Specific examples of the various types of image processing include normal image adjustment processing such as brightness adjustment processing and white balance adjustment processing, and image enhancement processing such as edge enhancement processing and texture and color enhancement imaging (TXI).

[0060] The display control unit 12 is a structural unit or circuit unit that controls the display device 30 so that an appropriate display format is displayed according to the display image data output from the image processing unit 11. The display control unit 12 controls the display of the display image data (various settings such as display position, display area, display size, etc.) as well as the display of various information other than image information (text information, icon information, etc.). Here, the various information other than image information includes examination date and time information, subject (patient) information, medical record information, etc., as well as various alarm information (notification information) that is displayed as appropriate during an endoscopic examination, for example.

[0061] Specifically, the display control unit 12 performs display control to continuously display the endoscopic image in the display area, and when a specific object is detected in the endoscopic image, the display control unit 12 performs display control to add a predetermined display (for example, a frame display) to the image area including the specific object.

[0062] The memory control unit 13 is a structural unit or circuit unit that controls the memory device 50 (including the temporary memory unit 51) so that the image data for storage output from the image processing unit 11 is saved (recorded or stored) in an appropriate storage format.

[0063] The storage control unit 13 performs storage control to permanently store image data representing an endoscopic image acquired by the endoscope 20 in a storage area of ​​the storage device 50 .

[0064] The specific object image detection unit 14 is a structural unit or circuit unit that detects an image area including a specific object from an endoscopic image based on image data acquired by the endoscope 20 .

[0065] In detail, the specific object image detection unit 14 detects a specific object (such as a lesion such as a tumor) within each frame of endoscopic image that is displayed continuously based on the image data output from the image processing unit 11.

[0066] The specific object image detection unit 14 detects specific objects, for example, by performing similar image recognition based on multiple specific object images (case images, etc.) prepared in advance, or by performing data analysis using deep learning or machine learning to detect the desired specific object.

[0067] For example, in recent years, object detection technology using AI (Artificial Intelligence) has been developed that obtains desired inference results (e.g., detection of specific objects such as tumors) through machine learning of endoscopic images acquired by an endoscope.

[0068] Machine learning is a method of obtaining inference results about unknown matters by learning the features, time-series information, spatial information, etc. of known input information and making inferences based on the learning results. That is, in machine learning, a trained model (also called an inference model) is first obtained that enables inference of a determinable output result from specific input information.

[0069] In this case, when generating a trained model, a large amount of information with known input-output relationships is used as training data to obtain highly reliable inference results. For example, in deep learning, a network is designed using a large amount of training data so that expected outputs can be obtained for known inputs. The trained model obtained through this process can be used independently of the network that was used for training.

[0070] Therefore, by providing the processor 10 with this type of inference model, it becomes possible to accurately and quickly perform various judgments or detections, such as the detection of a specific object in the specific object image detection unit 14, as well as the judgment of the imaging distance range in which an image can be captured in a focused state during pan-focus control, the judgment of the usage state of the endoscope during an endoscopic examination, and the detection of the surface blood vessel pattern of a specific object (detection of information including the position of the blood vessel pattern in the image), etc. In this case, the specific object image detection unit 14 combines the functions of a specific object detection unit, an imaging distance range judgment unit, a blood vessel judgment unit, etc.

[0071] Here, the inference model is constructed roughly as follows. For example, a large amount of image data corresponding to input and output is provided as training data to a predetermined network (not shown) for constructing the inference model. Here, the training data includes image data suitable for detection or determination, such as image data of endoscopic images acquired sequentially in time series during an endoscopic examination, image data of images (such as case images) in which a specific object is captured in the endoscopic images, etc. In this case, an annotation may be set to surround the image area of ​​the specific object with a frame or the like.

[0072] By learning using a large amount of training data, a network design is determined so that a predetermined network (not shown) can obtain an output corresponding to an input. For example, Figures 12 and 13 are explanatory diagrams of an inference model obtained as a result of learning in a predetermined learning device.

[0073] For example, as shown in Figure 12, when a group of endoscopic images (first training data group 201) containing a specific object (such as a tumor) is input, information about the specific object captured in the image and additional information such as adding a frame display surrounding the image area of ​​the specific object are obtained along with reliability information. This allows the construction of a first inference model 200 that detects the specific object from the endoscopic image.

[0074] When endoscopic image data (time-series image data) 202 taken during an endoscopic examination is input to the first inference model 200 constructed in this manner, the first inference model 200 detects an image area containing the specific object in a frame in which the specific object is captured from the endoscopic image data 202, and outputs image data (inference result image 203) in which a frame display is added to the image area of ​​the specific object. In this manner, a specific object can be detected from an endoscopic image. The image data of this image 203 can then be temporarily stored, or, if necessary, displayed in a specified display area of ​​the display device 30, providing a reference for the endoscope user during the endoscopic examination.

[0075] 12 , when a group of endoscopic images (second training data group 204) annotated by associating imaging distance range information with images of a specific object contained in the endoscopic images that are different in focus using pan focus control during an endoscopic examination is input, information about the imaging distance range of the specific object using pan focus control during the endoscopic examination is obtained together with reliability information. This allows for the construction of a second inference model 210 that determines the imaging distance range in which a well-focused endoscopic image can be obtained using pan focus control during an endoscopic examination.

[0076] When endoscopic image data (time-series image data) 202 during an endoscopic examination is input to the second inference model 210 constructed in this manner, the second inference model 210 outputs a determination that a well-focused image cannot be obtained from the endoscopic image data 202 using pan focus control.

[0077] It is known that multiple endoscopic images acquired sequentially in time series have image change patterns corresponding to the respective use states. Therefore, the image change patterns can be categorized for each use state. In other words, by detecting an image change pattern similar to a categorized image change pattern, the use state of the endoscope can be determined. Here, a group of images representing the time-series image change patterns categorized for each use state is referred to as a use state pattern image group.

[0078] As shown in Figure 13, when a group of usage state pattern images (third training data group 206) is input, time-series image change pattern information for each usage state of the endoscope during an endoscopic examination is obtained along with reliability information. This allows the construction of a third inference model (endoscope usage state inference model) 220 that determines the usage state of the endoscope during an endoscopic examination.

[0079] When endoscopic image data (time-series image data) 202 taken during an endoscopic examination is input to the third inference model 220 constructed in this manner, the third inference model 220 outputs a group of inference result images 207 from the endoscopic image data 202. In this way, the usage status of the endoscope can be determined from the group of endoscopic images.

[0080] On the other hand, it is known that specific objects such as tumors have unique surface vascular patterns depending on, for example, the type of lesion. Therefore, by categorizing the surface vascular patterns of specific objects by lesion type, it is possible to detect the lesion type and contribute to supporting diagnosis. Note that the surface vascular pattern of a specific object such as a tumor refers to the pattern of blood vessels running on the surface of the object such as a tumor.

[0081] 13, when a group of images of a specific object including a surface vascular pattern (fourth training data group 208) is input, image information of the surface vascular pattern of the specific object being observed during endoscopic examination is obtained together with reliability information, thereby constructing a fourth inference model (inference model for vascular determination) 230 that detects the surface vascular pattern of the specific object (e.g., tumor) being observed during endoscopic examination.

[0082] When endoscopic image data (time-series image data) 202 taken during an endoscopic examination is input to the fourth inference model 230 constructed in this manner, the fourth inference model 230 outputs a group of inference result images 208 from the endoscopic image data 202. In this manner, surface vascular pattern images of a specific object can be detected from the group of endoscopic images.

[0083] "Deep learning" is a multilayered version of the "machine learning" process using neural networks. A typical example is a "forward propagation neural network," which sends information from front to back and makes a judgment. In its simplest form, it requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons determined by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights, and a bias value is added between the hidden and output layers, making it easy to form logic gates. While three layers are sufficient for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, models with 9 to 152 layers have become practical due to their training time, judgment accuracy, and energy consumption.

[0084] Various well-known networks may be used for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) using CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses image features, operates with minimal processing, and is strong in pattern recognition. Furthermore, "recurrent neural networks" (fully connected recurrent neural networks) that can handle more complex information and allow information analysis whose meaning changes depending on the order or sequence of information may be used.

[0085] To realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, devices specialized for matrix calculations such as GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) may also be used. In recent years, such dedicated artificial intelligence (AI) hardware "neural network processing units (NPUs)" have been designed to be integrated and embeddable with CPUs and other circuits, and may even become part of the processing circuit.

[0086] Furthermore, inference models may be obtained by employing various well-known machine learning techniques, not limited to deep learning. For example, techniques such as support vector machines and support vector regression are available. Here, learning involves calculating the weights, filter coefficients, and offsets of a classifier; other techniques include using logistic regression processing. When a machine is to make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method for deriving an image judgment using machine learning is employed. However, a rule-based method for applying rules acquired by humans through experience or heuristics to make a specific judgment may also be used.

[0087] The usage state determination unit 15 is a structural unit or circuit unit that determines the usage state of the endoscope 20 during an endoscopic examination. The usage state determination unit 15 determines the usage state of the endoscope 20 based on a plurality of image data that are sequentially output in time series from the image processing unit 11.

[0088] For this purpose, the usage state determination unit 15 has a time-series change determination unit 15a, which is a structural unit or circuit unit that analyzes a plurality of endoscopic images displayed based on a plurality of image data sequentially output in time series from the image processing unit 11, and detects a time-series image change pattern.

[0089] Here, the detection of the time-series image change pattern is carried out, for example, by sequentially storing multiple image data outputted in time series from the image processing unit 11 in the temporary storage unit 51, and sequentially comparing the temporarily stored image with the next image inputted, thereby detecting the image change over time.

[0090] The usage state determination unit 15 then determines the usage state of the endoscope 20 based on the time-series image change pattern detected by the time-series change determination unit 15 a. Note that, here, the detection of the time-series image change pattern and the determination of the usage state of the endoscope 20 may be performed using the second inference model 210, as described above.

[0091] Here, the usage states of the endoscope 20 determined by the usage state determination unit 15 include, for example: (1) a screening state, which is a first observation state in which a specific object (such as a lesion, such as a tumor) is searched for while observing the inside of the subject's lumen; (2) an approach state, which is a second observation state in which the tip of the endoscope approaches the specified specific object from the first observation state (screening state); and (3) a detailed observation state, which is a third observation state in which detailed observation of the specified specific object is performed.

[0092] The focus control unit 16 is a structural unit or circuit unit that controls the driving of the imaging optical system 26a of the imaging unit 26. The focus control unit 16 controls the driving of the imaging optical system 26a based on the detection result of the specific object image detection unit 14, and performs, for example, a predetermined automatic focus adjustment operation (hereinafter referred to as autofocus control) or pan focus control.

[0093] In addition, the focus control unit 16 controls the driving of the imaging optical system 26a based on the determination result of the use state determination unit 15, and performs switching control to an appropriate focus control according to the determined use state of the endoscope 20.

[0094] Here, appropriate focus control according to the usage state of the endoscope 20 is, for example, as follows.

[0095] First, observation in the first observation state (screening state) is performed with the imaging unit and the object being observed at a slight distance (for example, about a few centimeters). In this case, sufficient observation can be performed within the depth of field of the imaging optical system. Therefore, the focus control of the imaging unit in this state is usually set to a so-called pan-focus control setting.

[0096] In this case, the pan focus control setting is, for example, a control setting that sets the position of the imaging optical system so that an object within a predetermined distance range (for example, within a range of approximately 1 to 3 centimeters in front of the tip surface of the lens) is roughly in focus, and maintains that position.

[0097] Here, fine focus control according to distance may be performed, or control may be performed in several stages, or two-stage switching may be performed, simply changing the position of the optical system from a pan-focus state to another position (close-distance compatible focus position).

[0098] On the other hand, when in the second observation state (approach state), it is necessary to continue observing the target specific object so as not to lose sight of it. Furthermore, in the approach state, the distal end of the endoscope (imaging unit) is gradually moved closer to the specific object. For this reason, if the endoscope is moved while in the approach state with the pan focus control setting, it becomes increasingly difficult to maintain a focused state. Therefore, in this case, it is desirable to use normal autofocus control (hereinafter referred to as normal autofocus control).

[0099] Here, normal autofocus control refers to a control that performs drive control to move the focus lens of the imaging optical system in the optical axis direction so that a focused state can always be obtained continuously on a specified object located, for example, within approximately 3 centimeters in front of the tip surface of the lens, although this does not require the same precision as control during close-up imaging.

[0100] On the other hand, observation in the third observation state (detailed observation state) is often performed when the imaging unit is close to the object being observed (for example, within about 3 millimeters of the object). In this case, it is known that the depth of field becomes extremely shallow. Therefore, it is desirable that the focus control of the imaging unit in this case be autofocus control suitable for close-up imaging (hereinafter referred to as close-up autofocus control).

[0101] Here, close-range autofocus control refers to precise drive control that moves at least some of the optical lenses that contribute to focus control (hereinafter referred to as focus lenses) among the multiple optical lenses that make up the imaging optical system in the optical axis direction, so that an appropriate focus state can be obtained for a specified object that is within a range of approximately 3 millimeters in front of the lens tip surface.

[0102] During normal autofocus control or proximity autofocus control, focus control is generally performed by targeting an object within a predetermined range (within a focus frame) in the central region of the imaging screen.

[0103] However, in recent years, there have been various other forms of autofocus control, such as a form in which autofocus control is performed by tracking a predetermined object detected within the image capture screen, a form in which autofocus control is performed by preferentially selecting either a distant object or a close object within the image capture screen, a form in which autofocus control is performed on an object included in an area within the image capture screen that is arbitrarily specified by the user, and focus control that takes into account the depth of field of the imaging optical system to ensure a good focus state within a predetermined distance range in front of the imaging unit. The focus control unit 16 can apply any of these various autofocus control forms by appropriately selecting or combining them.

[0104] The lighting control unit 17 is a structural unit or circuit unit that drives and controls the light source device 40 and the lighting unit 25. The lighting control unit 17 controls the light source device 40 or the lighting unit 25 in response to operation instruction signals from various operation members 22c provided in the operation unit 22, or instruction signals based on the detection results of the specific object image detection unit 14 or the determination results of the use state determination unit 15, and controls switching of the light source type (white light, special light, etc.).

[0105] Here, the light source types include, for example, white light that is irradiated onto the subject when performing normal observation, as well as predetermined special light that is irradiated onto the subject when performing image-enhanced observation such as narrow band imaging (NBI) or red dichromatic imaging (RDI).

[0106] It should be noted that all or part of the processor 10 includes hardware. Here, the processor 10 is configured by a well-known configuration including, for example, a central processing unit (CPU), random access memory (RAM), read only memory (ROM), non-volatile memory, non-volatile storage, and a non-transitory computer readable medium, as well as peripheral devices thereof.

[0107] Software programs to be executed by the CPU and fixed data such as data tables are stored in advance in ROM, nonvolatile memory, nonvolatile storage devices, etc. The CPU reads out the software programs stored in ROM, etc., expands them into RAM, and executes them, and the software programs refer to various data, etc. as appropriate, thereby realizing the functions of the processor 10.

[0108] The processor 10 may also be configured with a semiconductor chip such as an FPGA (Field Programmable Gate Array), etc. Furthermore, the processor 10 may also be configured with an electronic circuit.

[0109] Furthermore, the software program may be recorded or stored, in whole or in part, as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.

[0110] The operation of the endoscope system 1 including the focus control device of this embodiment configured as described above will be described below. Fig. 3 is a flowchart explaining the operation of the endoscope system including the focus control device of one embodiment of the present invention. The flowchart shown in Fig. 3 is a diagram explaining the operation when an endoscopic examination is performed using the endoscope system 1.

[0111] First, it is assumed that the endoscope system 1 has been activated and is ready to perform an endoscopic examination. The endoscope 20 in the endoscope system 1 is inserted into an organ of a subject (e.g., a patient) to be examined. The focus control setting in the initial state after activation of the endoscope 20 is, for example, a pan-focus control setting.

[0112] 3, the processor 10 performs image change analysis in the time-series change determination unit 15a based on the endoscopic images acquired sequentially in time series by the imaging unit 26 to detect a time-series image change pattern. At the same time, the use state determination unit 15 starts executing a process to determine the use state of the endoscope 20 based on the detected time-series image change pattern. The image change analysis process and use state determination process of step S1 are assumed to be executed continuously while the endoscopic examination is being performed.

[0113] Here, as described above, the usage states of the endoscope 20 include a first observation state (hereinafter referred to as the screening state), a second observation state (hereinafter referred to as the approach state), and a third observation state (hereinafter referred to as the detailed observation state).

[0114] 4 and 6 are schematic diagrams showing the state of the endoscope 20 in the screening state. Of these, Fig. 4 shows a first state representing a state in which a specific object has not been detected in the screening state (object search in progress), and Fig. 6 shows a second state representing a state in which a specific object has been detected in the screening state.

[0115] In Fig. 4, reference numeral 300 schematically indicates an organ or the like having a lumen. Fig. 4 shows the insertion section 21 (tip portion 21a) of the endoscope 20 being inserted into the lumen of the organ or the like 300. Note that reference numeral 301 in Fig. 4 indicates a specific object such as a tumor present on the inner wall of the organ or the like 300 (described in detail later).

[0116] 4, the symbol O indicates the central axis of the insertion section 21 of the endoscope 20. The insertion section 21 of the endoscope 20 is operated so that the central axis O advances and retreats in a direction along the lumen of the organ 300. The symbol V in Fig. 4 indicates the imaging range (field of view) of the imaging unit 26 built into the tip section 21a.

[0117] When the endoscope 20 is in the screening state, as shown in Fig. 4, the insertion section 21 of the endoscope 20 is arranged in a substantially straight line along the lumen of the organ 300. For example, when the tip section 21a of the endoscope 20 is in position [A] shown in Fig. 4, the endoscopic image displayed on the display screen of the display device 30 is as shown in Fig. 5.

[0118] Here, Fig. 5 shows an example of what is displayed on the display screen of the display device when the endoscope is in the screening state and the tip portion 21a of the endoscope 20 is in the position [A] in Fig. 4. Note that Fig. 6 shows an example of what is displayed on the display screen of the display device when the endoscope is in the screening state and the tip portion 21a of the endoscope 20 is in the position [B].

[0119] First, when the endoscope 20 is at position [A] in Fig. 4, the endoscopic image 101a is displayed in the display area 101 in Fig. 5, for example. The endoscopic image 101a at this time has a dark area 101c in the central area, as shown in Fig. 5. This dark area 101c is the lumen of the organ 300, and corresponds to the area in front of the endoscope 20. At the same time, the endoscopic image 101a at this time has a bright area 101d in the peripheral area, as shown in Fig. 5. At this point (when the tip 21a of the endoscope 20 is at position [A] in Fig. 4), the endoscopic image 101a does not include an image of the specific object, as shown in Fig. 5.

[0120] When the endoscope 20 is in the screening state, a lumen extends in the area in front of the endoscope 20, and the illumination light from the illumination unit 25 does not sufficiently reach this area in front of the endoscope 20. Therefore, when the endoscope 20 is in the screening state, a dark area 101c often exists within a predetermined range in the central area of ​​the endoscopic image 101a.

[0121] Next, suppose the endoscope 20 moves in the direction of arrow X in Fig. 4 from position [A] in Fig. 4 to position [B] in Fig. 6. The endoscopic image 101a at this time then takes a form such as that displayed in the display area 101 in Fig. 7, for example. The endoscopic image 101a shown in Fig. 7 also has a dark area 101c in the central area. At the same time, the endoscopic image 101a at this time also has a bright area 101d in the peripheral area, as shown in Fig. 7. At this point (when the distal end 21a of the endoscope 20 is at position [B] in Fig. 6), the endoscopic image 101a includes a specific object image 101b captured within a portion of the peripheral area, as shown in Fig. 7.

[0122] Generally, when the endoscope 20 is in the screening state, as described above, a lumen extends to the front region of the endoscope 20. In this situation, the illumination light from the illumination unit 25 does not sufficiently reach the front region of the endoscope 20.

[0123] Therefore, the endoscopic image 101a acquired when the endoscope 20 is in the screening state often has a tendency to have a dark area 101c in a predetermined range in the central area and a bright area 101d in the peripheral area. In the endoscopic image 101a, the specific object image 101b is detected in the bright area 101d in the peripheral area.

[0124] For this reason, when the time-series change determination unit 15a detects a time-series image change pattern of an image in which a dark area 101c exists in the central area of ​​the endoscopic image 101a and a bright area 101d exists in the peripheral area, the usage state determination unit 15 can determine that the endoscope 20 during the endoscopic examination is in a screening state (first observation state).

[0125] On the other hand, consider the case where a specific object image 101b is detected in the endoscopic image 101a when the endoscope 20 is in the screening state (the case where the situation shown in FIGS. 6 and 7 occurs).

[0126] Here, when a specific object image 101b is detected, it may be, for example, when it is detected by the specific object image detection unit 14, or when a user of the endoscope 20 (such as a doctor) recognizes the specific object image 101b by visually observing the endoscopic image 101a.

[0127] When the specific object image 101b is detected by the specific object image detection unit 14, the detection result is notified to the user (doctor, etc.) of the endoscope 20 in a predetermined form. In this case, the notification may be made by displaying a marking display such as a frame 106 (see FIGS. 7 and 8, etc.) surrounding the specific object image 101b in the endoscopic image 101a, by displaying a notification display in the information display area 104 notifying the user that a specific object has been detected, or by displaying a notification display such as audio. This allows the user (doctor, etc.) of the endoscope 20 to recognize the specific object image 101b in the endoscopic image 101a.

[0128] In this way, when the user (doctor, etc.) of the endoscope 20 recognizes the specific object image 101b in the endoscopic image 101a, the user (doctor, etc.) of the endoscope 20 may attempt to observe the specific object image 101b in detail. To do so, the user (doctor, etc.) of the endoscope 20 performs an operation to move the tip portion 21a of the endoscope 20 closer to the specific object image 101b. By performing this operation, the usage state of the endoscope 20 transitions from the screening state to the approach state.

[0129] FIG. 8 is a schematic diagram showing the state of the endoscope 20 when it is in the approach state. As shown in FIG. 8 , when the endoscope 20 is in the approach state, the insertion section 21 of the endoscope 20 is bent relative to the bending portion 21b, and the bending portion 21b is bent by a predetermined amount in a predetermined direction. The bending direction of the bending portion 21b at this time is set to a direction in which the distal end surface of the distal end portion 21a of the endoscope 20 is positioned to face the detected specific object 301. Then, the user (doctor, etc.) performs an operation to gradually move the distal end portion 21a of the endoscope 20 toward the specific object 301. The endoscopic image 101a displayed on the display screen of the display device 30 during this operation is as shown in FIG. 9 . Here, FIG. 9 is an example of a display screen of the display device when the endoscope is in the approach state.

[0130] When the endoscope 20 is in the state shown in Fig. 8, the endoscopic image 101a has a bright region 101d over almost the entire display area, as shown in Fig. 9. A specific object image 101b exists in the central area of ​​the endoscopic image 101a, and a part of a dark region 101c exists in a part of the peripheral area.

[0131] When the endoscope 20 is in the approach state, the inner wall of an organ 300 or the like is present in the area in front of the endoscope 20, within a range that is sufficiently reached by the illumination light from the illumination unit 25. Therefore, when the endoscope 20 is in the approach state, the endoscopic image 101a is substantially entirely a bright area 101d. When the endoscope 20 is in the approach state, the distal end surface of the distal end portion 21a of the endoscope 20 is positioned facing the specific object 301, as shown in Fig. 8. Therefore, a specific object image 101b is present near the central area of ​​the endoscopic image 101a at this time.

[0132] Generally, when the endoscope 20 is in the approach state, the inner wall of an organ or the like 300 is within the range of the illumination light from the illumination unit 25 in the forward region of the endoscope 20, and a specific object 301 is present on the inner wall surface.

[0133] Therefore, the endoscopic image 101a acquired when the endoscope 20 is in the approach state tends to have a bright area 101d over almost the entire image, a specific object image 101b within a predetermined range in the central area, and a portion of the dark area 101c in a portion of the peripheral area.

[0134] Furthermore, in the approach state, an operation is performed to gradually bring the tip 21a of the endoscope 20 closer to the specific object 301. As a result, the endoscopic images 101a acquired in time series in the approach state are displayed with the specific object image 101b in the central region gradually enlarging.

[0135] For this reason, if the time-series change determination unit 15a detects a time-series image change pattern in which a specific object image 101b is present in the central region of the endoscopic image 101a and the display size of the specific object image 101b gradually increases, the usage state determination unit 15 can determine that the endoscope 20 during endoscopic examination is in an approach state (second observation state).

[0136] Next, consider the case where the endoscope 20 transitions from the approach state to the detailed observation state. In the approach state during an endoscopic examination, after the distal end portion 21a of the endoscope 20 has reached a sufficiently close distance to the specific object 301 that is the target of observation, an operation is performed to perform detailed observation of the specific object 301 while maintaining the distance between the distal end surface of the distal end portion 21a and the specific object 301. By performing such an operation, the usage state of the endoscope 20 transitions from the approach state to the detailed observation state.

[0137] FIG. 10 is a schematic diagram showing the state of the endoscope 20 in the detailed observation state. As shown in FIG. 10 , when the endoscope 20 is in the detailed observation state, the bending section 21b of the endoscope 20 is maintained bent by a predetermined amount in a predetermined direction, and the distal end surface of the distal end section 21a is positioned in close proximity to and facing the specific object 301 being observed. Then, the user (doctor, etc.) performs operations to observe the specific object 301 from various angles while pointing the distal end section 21a of the endoscope 20 toward the specific object 301. The endoscopic image 101a displayed on the display screen of the display device 30 during such operations is as shown in FIG. 11 . Here, FIG. 11 is an example of a display screen of the display device when the endoscope is in the detailed observation state.

[0138] When the endoscope 20 is in the state shown in Fig. 10, the endoscopic image 101a has an enlarged specific object image 101b in the central region, and a bright region 101d in the peripheral region, as shown in Fig. 11. Since the specific object image 101b at this time is observed from a sufficiently close distance, for example, the blood vessel pattern on the surface of the specific object 301 can also be observed.

[0139] Generally, when the endoscope 20 is in the detailed observation state, a specific object 301 is present at a close distance opposite the distal end surface of the distal end portion 21a of the endoscope 20. The user (doctor, etc.) of the endoscope 20 observes the specific object 301 from various angles while always capturing the specific object 301 on the screen. For this reason, the endoscopic image 101a acquired in the detailed observation state tends to have a specific object image 101b always present in the central region, and there is little change in the display size of the specific object image 101b. Furthermore, when the specific object 101b is observed from various angles in this state, the specific object image 101b appears to move parallel within the display region 101 of the endoscopic image 101a while remaining within the display region 101 of the endoscopic image 101a.

[0140] For this reason, if the display size of the specific object image 101b present in the central region of the endoscopic image 101a does not change and the time-series image change pattern of the specific object image 101b remaining within the display region 101 of the endoscopic image 101a or moving within the range of the display region 101 is detected by the time-series change determination unit 15a, the usage state determination unit 15 can determine that the endoscope 20 during the endoscopic examination is in a detailed observation state (third observation state).

[0141] 3, in the next step S2, the processor 10 controls the time-series change determination unit 15a and the usage state determination unit 15 to detect a time-series image change pattern of the endoscopic image based on the multiple image data acquired sequentially in time series by the imaging unit 26. Then, the usage state determination unit 15 determines the usage state of the endoscope 20 based on the detected time-series image change pattern.

[0142] Specifically, for example, the time series change determination unit 15a detects a time series change pattern of whether or not the image in the central region of the endoscopic image is gradually enlarging based on multiple image data acquired sequentially in time series by the imaging unit 26.

[0143] In this case, if the time series change determination unit 15a detects that the image in the central region of the endoscopic image is gradually expanding, the usage state determination unit 15 determines that the endoscope 20 is in an approach state and proceeds to processing in step S3.

[0144] Furthermore, if the time series change determination unit 15a does not detect that the image in the central region of the endoscopic image is gradually expanding, the usage state determination unit 15 determines that the endoscope 20 is, for example, in a screening state, and proceeds to processing in step S11.

[0145] In step S3, the processor 10 controls the focus control unit 16 to execute a control process for switching to normal autofocus control, and then proceeds to the next step S4.

[0146] Next, in step S4, the processor 10 controls the time-series change determination unit 15a and the usage state determination unit 15 to detect a time-series image change pattern of the endoscopic image based on the multiple image data acquired sequentially in time series by the imaging unit 26. Then, the usage state determination unit 15 determines the usage state of the endoscope 20 based on the detected time-series image change pattern.

[0147] Specifically, for example, the time series change determination unit 15a detects a time series change pattern based on multiple image data acquired sequentially in time series by the imaging unit 26, to determine whether the size of the image present in the central region of the endoscopic image has not changed (within a predetermined range) and whether the image is moving within the range of the display area (i.e., whether there is a parallel movement of the image within the screen).

[0148] In this case, if the time series change determination unit 15a determines that the size of the image in the central area of ​​the endoscopic image remains almost unchanged and detects movement within the display area of ​​the image, the usage state determination unit 15 determines that the endoscope 20 is in a detailed observation state and proceeds to processing in step S5.

[0149] Furthermore, if the time series change determination unit 15a determines that there is a change in the size of the image in the central region of the endoscopic image, the usage state determination unit 15 determines that the endoscope 20 is in a state other than the detailed observation state, and returns to processing in step S2.

[0150] In step S5, the processor 10 controls the focus control unit 16 to execute a control process for switching to close-up autofocus control, and then proceeds to the next step S6.

[0151] In step S6, the processor 10 refers to the results of the image change analysis process and the use state determination process that were started in the process of step S1 described above and have been continuously executed thereafter, and checks whether or not a change has occurred in the use state of the endoscope 20. If it is confirmed that a change has occurred in the use state of the endoscope 20, the process proceeds to step S7. If no change has occurred in the use state of the endoscope 20, the confirmation process is repeated.

[0152] Next, in step S7, the processor 10 controls the focus control unit 16 to execute a control process for switching to pan focus control, and then returns to the process of step S2.

[0153] On the other hand, if it is determined in the processing of step S2 that the endoscope 20 is in the screening state and the processing proceeds to the processing of step S11, the processor 10 controls the focus control unit 16 to execute a control processing for switching to the pan focus control in this step S11, and then proceeds to the processing of the next step S12.

[0154] In step S12, the processor 10 checks the detection result by the specific object image detection unit 14 and confirms whether or not a specific object has been detected in the endoscopic image. Although not specifically mentioned, the specific object detection process by the specific object image detection unit 14 is started at an appropriate predetermined timing, such as when the endoscopic examination starts (when the endoscope 20 is started) or at the same timing as when the processing of step S1 described above is executed, and is subsequently executed continuously while the endoscopic examination is being performed.

[0155] If the detection of the specific object is confirmed, a predetermined display process is executed, and then the process proceeds to the next step S13. If the detection of the specific object is not confirmed, the process proceeds to step S15.

[0156] The predetermined display processing that is performed when the detection of a specific object is confirmed is, for example, a processing that displays a frame 106 around the specific object image 101b in the endoscopic image 101a, or a processing that displays a notification display in the information display area 104.

[0157] Next, in step S13, the processor 10 detects the surface blood vessel pattern of the specific object detected in the process of step S12. If the surface blood vessel pattern of the specific object is detected, the process proceeds to the next step S14. If the surface blood vessel pattern of the specific object is not detected, the process returns to step S2.

[0158] In step S14, the processor 10 controls the time-series change determination unit 15a and the usage state determination unit 15 to detect a time-series image change pattern of the endoscopic image based on the multiple image data acquired sequentially in time series by the imaging unit 26. Then, the usage state determination unit 15 determines the usage state of the endoscope 20 based on the detected time-series image change pattern.

[0159] Specifically, for example, the time series change determination unit 15a detects a time series change pattern of whether or not the surface blood vessel pattern of the specific object detected in the processing of step S12 described above is approaching, based on multiple image data acquired sequentially in time series by the imaging unit 26.

[0160] In this case, if it is confirmed that the surface blood vessel pattern is approaching, it is determined that the endoscope 20 has transitioned to the approach state, and the process returns to step S3. If it is not confirmed that the surface blood vessel pattern is approaching, it is determined that no operation has been performed to perform detailed observation of the specific object, and the process proceeds to the next step S15.

[0161] In step S15, the processor 10 confirms an instruction to terminate the ongoing endoscopic examination. Here, the instruction to terminate the endoscopic examination may be, for example, a predetermined examination termination instruction signal generated by operating a predetermined operating member among the processor 10 of the endoscopic system 1 or the operating members 22c provided on the operating unit 22 of the endoscope 20. In addition, the termination of the endoscopic examination can also be determined, for example, by checking image data of the endoscopic image acquired by the endoscope 20.

[0162] If the instruction to end the examination is confirmed in the process of step S15, the series of processes is ended (END). If the instruction to end the examination is not confirmed, the process returns to the process of step S12.

[0163] As described above, according to the embodiment, a time-series image change pattern is detected for a plurality of endoscopic images acquired sequentially in time series by the imaging unit 26 of the endoscope 20, and the usage state of the endoscope is determined based on the detected time-series image change pattern. Depending on the usage state of the endoscope 20 thus determined, appropriate focus control is switched to.

[0164] In this case, the endoscope 20 has three usage states: a first observation state (screening state), a second observation state (approach state), and a third observation state (detailed observation state). When the determination result is the first observation state, the endoscope 20 switches to pan focus control. When the determination result is the second observation state, the endoscope 20 switches to normal autofocus control. When the determination result is the third observation state, the endoscope 20 switches to close autofocus control.

[0165] In this way, according to the above embodiment, appropriate focus control is automatically performed according to the usage state of the endoscope 20, so that clear endoscopic images can always be obtained in a well-focused state.

[0166] Furthermore, when the endoscope 20 is in the first observation state, if an image area including a specific object is detected and the surface blood vessel pattern of the specific object is detected, and a time-series image change pattern in which the surface blood vessel pattern is displayed gradually larger is detected, it is determined that the endoscope has changed to the second observation state, and the endoscope is switched to normal autofocus control.

[0167] According to this, the detection of the surface blood vessel pattern when the endoscope 20 is in the first observation state (screening state) and pan focus control is being performed is used as the timing for switching focus control, thereby enabling more rapid control switching.

[0168] 3, the normal autofocus control that is switched to when the process moves from step S14 to step S3 may be a form of precise focus control according to distance, or may be a control that switches the focus position in multiple stages. Alternatively, it may be a two-stage switching that simply changes the position of the imaging optical system from a pan-focus position to another position (for example, a close-range focus position).

[0169] In the above-described embodiment, the time series change determination unit 15a detects a time series change pattern indicating whether or not the image in the central region of the endoscopic image is gradually enlarging, based on multiple image data acquired sequentially in time series by the imaging unit 26.

[0170] When the time series change determination unit 15a detects that the image in the central region of the endoscopic image is gradually enlarging, the usage state determination unit 15 determines that the endoscope 20 is in an approach state and performs an example of processing to switch to normal autofocus control (processing of steps S2 and S3 in Figure 3).

[0171] However, the determination of the approach state of the endoscope 20 and the timing of switching to normal autofocus control are not limited to this example. For example, if the imaging distance range in which a well-focused endoscopic image can be obtained by pan focus control during an endoscopic examination is known, it becomes possible to conveniently switch focus control when that imaging distance is reached.

[0172] For example, by having the second inference model 210 shown in Figure 11 in the processor 10 (e.g., the specific object image detection unit 14), it is possible to output a determination that a well-focused image cannot be obtained from the input endoscopic image data 202 using pan focus control.

[0173] Therefore, when the endoscope 20 is activated with, for example, a pan focus control setting, if it is confirmed that the imaging unit 26 of the endoscope 20 is gradually approaching a specific object, the focus control is not immediately switched to normal autofocus control (processing of steps S2 and S3 in FIG. 3 ), but rather, while maintaining the pan focus control setting, an imaging distance at which an endoscopic image in a well-focused state can be obtained with the pan focus control setting is determined. Then, imaging with the pan focus control setting may be continued until the determined imaging distance is reached. A portion of the processing sequence in this case is shown in FIG. 14 . FIG. 14 is a flowchart showing a modified example of the operation of this embodiment, showing a partial modification of the processing sequence in FIG. 3 .

[0174] First, in Fig. 14, the processes of steps S1, S2, and S3 are the same as those in Fig. 3. Furthermore, the processes from step S4 onwards in Fig. 3 are exactly the same, so illustration and description thereof will be omitted.

[0175] That is, in step S2, in the processor 10, the time series change determination unit 15a detects a time series change pattern of whether or not the image in the central region of the endoscopic image is gradually enlarging, based on multiple image data acquired sequentially in time series by the imaging unit 26.

[0176] In this case, if the time-series change determining unit 15a detects that the image in the central region of the endoscopic image is gradually enlarging, the process proceeds to step S2A in FIG.

[0177] In step S2A, the time-series change determination unit 15a in the processor 10 detects whether a specific object image in the central region of the endoscopic image can be acquired as a well-focused image even with the pan-focus control setting, based on multiple image data acquired sequentially in time series by the imaging unit 26. If it is detected that a well-focused image cannot be acquired with the pan-focus control setting, the process proceeds to the next step S3. If a well-focused image can be acquired with the pan-focus control setting, the process returns to step S2 and repeats the subsequent steps.

[0178] In this way, in the processing sequence shown in this modification, by detecting the imaging distance range in which a well-focused image can be obtained with the pan focus control setting, it is possible to delay the timing of switching from pan focus control to normal autofocus control when the endoscope 20 is in the approach state. This makes it possible to shorten the period during which the endoscope 20 is driven under autofocus control, thereby reducing the control processing load on the processor 10.

[0179] In the above-described embodiment, when the endoscope is in the detailed observation state (third observation state), the proximity autofocus control is performed, which always precisely controls the drive of the focus lens.

[0180] However, in close-range autofocus control, which captures images of objects at close range, the depth of field becomes extremely shallow, so for example, when focus control is performed on a certain point, there may be situations where an in-focus state cannot be obtained at positions in front of or behind that point.

[0181] Specifically, for example, polyps, tumors, etc. often protrude from the inner wall surface of an organ, etc. In such a specific object, if focus adjustment is performed near the top of the protruding shape, the base of the protruding shape (the part connected to the inner wall surface of the organ, etc.) may go out of focus.

[0182] However, when performing detailed observation of a specific object such as a detected polyp or tumor during endoscopic examination, there is a demand for detailed observation of, for example, the blood vessel pattern that connects from near the apex of the protrusion to the base.

[0183] Therefore, when performing close-up autofocus control, not only pinpoint focus control but also focus control that takes the depth of field into consideration may be performed.

[0184] Fig. 15 is a schematic diagram conceptually illustrating the state when close-range autofocus control is performed. The situation shown in Fig. 15 illustrates the state in which a specific object 301, such as a tumor present on the inner wall surface of an organ 300, is being observed in detail using an endoscope 20.

[0185] In Fig. 15, reference numeral 301a denotes the top of the specific object 301. Similarly, reference numeral 301b denotes the base of the specific object 301. Also, in Fig. 15, reference numeral D1 denotes the distance between the tip surface of the tip portion 21a and the top 301a of the specific object 301 (hereinafter referred to as the imaging distance D1). Similarly, in Fig. 15, reference numeral D2 denotes the distance between the tip surface of the tip portion 21a and the base 301b of the specific object 301 (hereinafter referred to as the imaging distance D2). Furthermore, reference numeral DOF in Fig. 15 denotes the depth of field.

[0186] Under such circumstances, when performing detailed observation of the specific object 301 using the endoscope 20, as shown in Fig. 15, the distal end surface of the distal end portion 21a of the insertion section 21 of the endoscope 20 is positioned to face the specific object 301. This allows the imaging unit 26 provided in the distal end portion 21a to capture an image of the specific object 301.

[0187] In this case, the focus control unit 16 controls the imaging optical system (not shown) of the imaging unit 26 to perform focus control on a desired position of the object within the imaging field of view.

[0188] For example, when focus adjustment is performed by focus control targeting the top 301a of the specific object 301 at the imaging distance D1, a detailed and clear image of the top 301a can be acquired. However, in this case, if the base 301b is out of the depth of field of the imaging optical system of the imaging unit 26, the base 301b will be out of focus, and a clear image may not be obtained.

[0189] Similarly, when focus adjustment is performed by focus control targeting the base 301b of the specific object 301 at the imaging distance D2, a detailed and clear image of the base 301b can be acquired. However, in this case, if the top 301a is outside the depth of field of the imaging optical system of the imaging unit 26, the top 301a will be out of focus, and a clear image may not be obtained.

[0190] Furthermore, when focus control is performed so that the apex 301a at the imaging distance D1 and the base 301b at the imaging distance D2 are included in the depth of field DOF, a clear image of the area connecting the apex 301a to the base 301b can be acquired, making it possible to observe in detail the blood vessel pattern connecting from the vicinity of the apex 301a to the base 301b.

[0191] In this way, close-range autofocus control is not limited to the conventional pinpoint precision focus control, but rather, depending on the situation, it may be better to perform the control described above (such as control that takes depth of field into account) as appropriate.

[0192] In this way, by further improving the focus control for the close focus control that is executed when the endoscope 20 is in the detailed observation state during an endoscopic examination, doctors and other medical professionals can obtain clear, well-focused endoscopic images that suit their needs and diagnostic purposes.

[0193] The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the spirit and scope of the invention. For example, while an endoscope has been described as an example, the present invention can be applied to any device that sequentially acquires and evaluates images (such as an image inspection device). Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple components. For example, if the problem to be solved by the invention can be solved and the effects of the invention can be obtained even if some components are deleted from all the components shown in one embodiment, the configuration from which these components are deleted can be extracted as an invention. Furthermore, components from different embodiments may be appropriately combined. The present invention is not limited by specific embodiments other than as limited by the appended claims.

Claims

1. A focus control device comprising: a time-series change determination unit that detects a time-series image change pattern of an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at a distal end of an endoscopic insertion portion; a use state determination unit that determines a use state of the endoscope based on the time-series image change pattern detected by the time-series change determination unit; and a focus control unit that drives and controls an imaging optical system of the imaging unit, wherein the focus control unit switches to focus control according to a determination result by the use state determination unit.

2. The use state of the endoscope determined by the use state determination unit includes at least: a first observation state of searching for a specific object while observing the inside of a lumen of a subject; a second observation state of a process of approaching the specific object from the first observation state; and a third observation state of performing a detailed observation of the specific object. The focus control unit performs pan focus control when the determination result is the first observation state, performs normal autofocus control when the determination result is the second observation state, and performs proximity autofocus control when the determination result is the third observation state. The focus control device according to claim 1, characterized in that.

3. When a time-series image change pattern of an image in which a dark region exists in a central region of the endoscopic image and a bright region exists in a peripheral region is detected, the use state determination unit determines that it is in the first observation state. When a time-series image change pattern of an image in which the specific object exists in the central region of the endoscopic image and the display size of the specific object is gradually displayed larger is detected, the use state determination unit determines that it is in the second observation state. When a time-series image change pattern of an image in which there is no change in the display size of the specific object existing in the central region of the endoscopic image and the specific object remains within the display area of the endoscopic image or moves within the range of the display area is detected, the use state determination unit determines that it is in the third observation state. The focus control device according to claim 2, characterized in that.

4. The focus control device according to claim 2, further comprising a specific object image detection unit that detects an image region including a specific object from the endoscopic image based on the image data acquired by the imaging unit. When the usage state of the endoscope is the first observation state, an image region including the specific object is detected, and the surface blood vessel pattern of the specific object is detected, the usage state determination unit determines that a change has occurred to the second observation state when a time-series image change pattern of an image in which the surface blood vessel pattern is gradually and largely displayed is detected, and the focus control unit switches to the normal autofocus control.

5. The focus control device according to claim 4, wherein the specific object image detection unit has a first inference model constructed using a large amount of specific object image data as teacher data and annotated by a frame display surrounding the image region including the specific object.

6. The focus control device according to claim 4, wherein the specific object image detection unit has a second inference model constructed using a large amount of the endoscopic images sequentially acquired by the pan-focus control during an endoscopic examination performed using the endoscope as teacher data and annotated with imaging distance range information in which an endoscopic image in a good focus state can be obtained.

7. The focus control device for an endoscope according to claim 2, wherein the usage state determination unit has a third inference model constructed using a large amount of the endoscopic images sequentially acquired in time series during an endoscopic examination performed using the endoscope as teacher data and annotated with a usage state pattern image group representing a time-series image change pattern classified for each usage state of the endoscope.

8. The focus control device according to claim 4, wherein the specific object image detection unit has a fourth inference model constructed using a large amount of specific object image data as teacher data and annotated with the surface blood vessel pattern of the specific object.

9. When the endoscope is in the third observation state, the focus control unit performs focus control such that a region from the top to the base of the specific object is included within the depth of field range, according to claim 2 of the focus control device.

10. A focus control method, comprising: detecting a time-series image change pattern of an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at a tip of an endoscopic insertion portion; determining a usage state of the endoscope based on a detection result of the time-series image change pattern; and switching to a focus control according to a determination result of the usage state of the endoscope.

11. A focus control program, causing a computer to execute: a time-series change determination process for detecting a time-series image change pattern of an endoscopic image based on a plurality of image data sequentially acquired in time series; a usage state determination process for determining a usage state of the endoscope based on a detection result of the time-series image change pattern; and a process for switching to a focus control according to a determination result of the usage state of the endoscope.

12. An endoscope system comprising: an endoscope including an imaging unit having an imaging optical system for forming an optical image of an object and an image sensor for photoelectrically converting the optical image formed by the imaging optical system; and a processor, wherein the processor includes: a time-series change determination unit for detecting a time-series image change pattern of an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at a tip of an endoscopic insertion portion; a usage state determination unit for determining a usage state of the endoscope based on the time-series image change pattern detected by the time-series change determination unit; and a focus control unit for driving and controlling the imaging optical system of the imaging unit, and the focus control unit switches to a focus control according to a determination result by the usage state determination unit.

13. A focus control device, comprising: a specific object image detection unit that detects an image region including a specific object from an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at the tip of an endoscopic insertion portion; a blood vessel determination unit that determines a blood vessel pattern of the specific object; and a focus control unit that drives and controls an imaging optical system that forms an optical image of the object, wherein the focus control unit drives and controls the imaging optical system to a focus position corresponding to the distance from the pan-focus position to the object according to an output result determined by the blood vessel determination unit based on the endoscopic image acquired when the imaging optical system is set at the pan-focus position.

14. The focus control device according to claim 13, wherein the blood vessel determination unit has an inference model for blood vessel determination that is trained to output the position of the blood vessel pattern when an endoscopic image is input, using teacher data in which the position of the blood vessel pattern to be observed in the endoscopic image is annotated.

15. The usage state determination unit determines the usage state of the endoscope by detecting a time-series image change pattern in which the display size of the image region including the blood vessel pattern in the endoscopic image is sequentially enlarged. When the usage state determination unit determines that the display size of the image region including the blood vessel pattern is sequentially enlarged, the focus control unit drives and controls the imaging optical system to a focus position corresponding to a short distance from the pan-focus position to the object. The focus control device according to claim 13.

16. A focus control method, comprising: detecting an image region including a specific object from an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at the tip of an endoscopic insertion portion; determining a blood vessel pattern of the specific object; and driving and controlling the imaging optical system to a focus position corresponding to the distance from the pan-focus position to the object according to an output result determined by the blood vessel determination unit based on the endoscopic image acquired when the imaging optical system is set at the pan-focus position.

17. A focus control method, comprising: detecting a temporal image change surface pattern of an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at a tip of an endoscopic insertion portion; detecting a specific object image included in the endoscopic image; determining a usage state of the endoscope based on a detection result of a change pattern of the specific object in the temporal image change; and switching an imaging optical system of the endoscope from a pan-focus state to a different focus control.

18. A focus control apparatus, comprising: a temporal change determination unit that determines a temporal image change pattern of an endoscopic image based on a plurality of image data sequentially acquired in time series by an imaging unit included at a tip of an endoscopic insertion portion; a specific object image detection unit that detects a specific object image included in the endoscopic image; a usage state determination unit that determines a usage state of the endoscope based on a detection result of a change pattern of the specific object in the temporal image change pattern; and a focus control unit that switches an imaging optical system of the endoscope from a pan-focus state to a different focus control.

19. The focus control apparatus according to claim 13, wherein the blood vessel determination unit has an inference model for blood vessel determination that determines a running pattern of blood vessels of a specific object based on a plurality of image data sequentially acquired in time series by an imaging unit included at a tip of the endoscopic insertion portion, and the inference model for blood vessel determination is learned to output a position of an object having a running pattern to be observed when an endoscopic image is input, by teacher data in which positions of objects having a running pattern of blood vessels to be observed in an image obtained from the endoscope are annotated.

20. The focus control apparatus according to claim 2, wherein the usage state determination unit has an inference model for endoscopic usage state determination that determines a usage state of the endoscope based on a plurality of image data sequentially acquired in time series by an imaging unit included at a tip of the endoscopic insertion portion, and the inference model for endoscopic usage state determination is learned to output the endoscopic usage state when an endoscopic image is input, by teacher data in which the endoscopic usage state at the time when a continuous temporal image group obtained from the endoscope is annotated.

Images