Focus control device, focus control method, focus control program, endoscope system, and method for creating inference model for distance distribution of endoscopic image

Abstract

The present invention comprises: a distance distribution information acquisition unit (15) that acquires, from an endoscope or the like, distance distribution information in the front-back direction of an object on the basis of output signals from an imaging unit (26) provided at the distal end of an insertion part (21) of an endoscope (20); a pattern classification unit (18) that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns; and a focus position control unit (16) that adjusts the relative position between a focus lens and an imaging element (26b) included in the imaging unit along the optical axis in accordance with the result of the classification by the pattern classification unit.

Description

Focus control device, focus control method, focus control program, endoscope system, and method for creating an inference model for distance distribution of endoscopic images

[0001] The present invention relates to a focus control device, a focus control method, a focus control program, an endoscope system, and a method for creating an inference model for distance distribution of an endoscopic image, which switch to an appropriate focus control depending on the state of use of the endoscope.

[0002] Conventionally, endoscopic systems that include an endoscope that captures images of the inside of a subject and acquires image data, a processor that performs various types of image processing on the image data acquired by the endoscope, a display device that displays the image data processed by the processor as a visible image, and a storage device that records or stores the image data have been widely used in, for example, the medical field and the industrial field.

[0003] Furthermore, in the medical field, various examinations are widely performed using this type of endoscope system to observe the inside of living organs, etc. In an examination using a medical endoscope system (hereinafter referred to as an endoscopic examination), for example, an insertion section of an endoscope inserted into the lumen of a living organ, etc., is advanced and retreated along the length of the lumen, such as by being inserted and withdrawn, while an imaging unit provided at the distal end of the insertion section captures images. At this time, the endoscopic images acquired by the imaging unit are displayed as moving images in real time chronologically on a display device. At the same time, the moving image data can also be stored in a storage device.

[0004] During such an endoscopic examination, the doctor or other equipment operator searches for lesions such as polyps or tumors occurring on the inner walls of organs as specific objects (hereinafter referred to as specific objects) while observing the real-time endoscopic image displayed on the display device (or, after the examination, a replayed endoscopic image based on recorded or stored endoscopic image data).

[0005] Generally, the interior of an organ or the like containing an object such as a tumor that is the subject of observation in an endoscopic examination performed using an endoscope or the like does not have a uniform planar shape, but rather has a complex shape with protruding and concave shapes.

[0006] Therefore, for example, if the object being observed (such as a tumor) has a large difference in distance in the front-to-back direction, adjusting the focus by targeting a specific point on the surface of the object will ensure that the object is in focus within a specific range in the front-to-back direction of that point (within the so-called depth of field), but it may not always be possible to ensure that the object is in focus within the desired range.

[0007] Furthermore, when illumination light is directed toward the object of observation, the entire screen cannot be uniformly illuminated. For example, in the front-to-back direction of the object of observation, the illumination light may not reach the rear area that is a predetermined distance or more away from the imaging unit, resulting in underexposed areas (so-called blackout). On the other hand, the illumination light may be too strong in the front area close to the imaging unit, resulting in overexposed areas (so-called whiteout). As such, differences in brightness may occur within the screen, and therefore, it is often impossible to obtain uniform exposure across the screen.

[0008] Therefore, in recent years, various technologies have been proposed that acquire distance distribution information (depth map) in the forward and backward directions within an image based on the imaging signal acquired by an imaging unit, and use the acquired distance distribution information to determine whether to perform image processing or shooting processing.

[0009] For example, the imaging device disclosed in Japanese Patent Publication No. 2015-118338 is an imaging device that performs general image acquisition, in which the captured image is divided into multiple regions based on the distance distribution within the captured image, and which of multiple bracket processes (continuous shooting, AF bracket, AE bracket, strobe bracket, etc.) to perform is determined based on focus adjustment setting conditions that have been acquired in advance.

[0010] Therefore, for example, in equipment such as endoscopes, it would be very convenient if it were possible to determine, for example, the usage status of the equipment based on the distance distribution within the captured image, and to switch the focus control in the imaging unit in accordance with the determined usage status of the equipment.

[0011] An object of the present invention is to provide a focus control device, a focus control method, a focus control program, and an endoscope system that can switch to an appropriate focus control of an imaging unit in medical equipment such as an endoscope or inspection equipment (such as an image inspection device) depending on the usage status of the equipment.

[0012] In order to achieve the above object, a focus control device according to one aspect of the present invention includes a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion, a pattern classification unit that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns, and a focus position control unit that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the classification result by the pattern classification unit.

[0013] A focus control method according to one aspect of the present invention acquires distance distribution information of an object in the forward and backward directions based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion, classifies the acquired distance distribution information into a plurality of distance distribution patterns, and adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit according to the classification results.

[0014] A focus control program according to one aspect of the present invention causes a computer to execute a distance distribution information acquisition process that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion, a pattern classification process that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns, and a focus position control process that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the classification result by the pattern classification unit.

[0015] An endoscopic system according to one aspect of the present invention comprises an endoscope including an imaging unit including an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system to obtain an image signal, an endoscope including a focus adjustment mechanism that moves some of the optical lenses included in the imaging optical system forward and backward in a direction along the optical axis, and a processor, wherein the processor comprises: a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of the object based on an output signal from the imaging unit provided at the tip of an endoscope insertion portion; a pattern classification unit that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns; and a focus position control unit that adjusts the relative position on the optical axis between a focus lens and the imaging element included in the imaging unit in accordance with a classification result by the pattern classification unit.

[0016] A focus control method according to another aspect of the present invention includes a distance distribution information acquisition step of acquiring distance distribution information of an object in front of an imaging unit provided at the tip of an endoscope insertion portion; a distance distribution pattern classification step of classifying the distance distribution information into a screening distance distribution pattern representing a screening operation for searching for a specific object and an observation distance distribution pattern representing a detailed observation operation at a close range in close proximity to the specific object; and a focus control switching step of switching between pan focus control fixed at a first focus position and observation-priority focus control fixed at a second focus position according to the classification result.

[0017] A method for creating an inference model for distance distribution of endoscopic images according to one aspect of the present invention includes a learning step in which distance distribution information of an object in front of an imaging unit provided at the tip of an endoscope insertion portion is learned using training data in which information indicating screening or detailed observation is annotated for multiple distance distribution information acquired by the imaging unit, in order to infer either a screening distance distribution pattern representing a screening operation for searching for a specific object or a detailed observation distance distribution pattern representing a detailed observation operation for observing the specific object at close range.

[0018] According to the present invention, it is possible to provide a focus control program and an endoscope system that can switch to an appropriate focus control of an imaging unit depending on the usage status of the equipment in medical equipment such as endoscopes and inspection equipment (image inspection devices, etc.).

[0019] A configuration diagram showing an overall configuration of an endoscope system including a focus control device of one embodiment of the present invention; a block configuration diagram showing an outline of the internal configuration of an endoscope system including a focus control device of one embodiment of the present invention; a flowchart explaining the operation of an endoscope system including a focus control device of one embodiment of the present invention; a schematic diagram showing a first state (during normal search) when the endoscope of the endoscope system of Figure 1 is in a first observation state (screening state); a display example of distance distribution information (depth map) when the endoscope is in the state of Figure 4; a schematic diagram showing a second state (when a specific object is detected) when the endoscope of the endoscope system of Figure 1 is in the first observation state (screening state); a display example of distance distribution information (depth map) when the endoscope is in the state of Figure 6; 10 is a schematic diagram showing the state when the endoscope of the endoscopic system of FIG. 1 is in the third observation state (detailed observation state); an example of display of distance distribution information (depth map) when the endoscope is in the state of FIG. 10; a schematic diagram conceptualizing the structure of an imaging element applied to an endoscope included in the endoscopic system of FIG. 1 and explaining the principle of image plane phase difference AF by said imaging element; a conceptual diagram showing an endoscopic image acquired during an endoscopic examination performed using the endoscopic system of FIG. 1; an explanatory diagram of first and second inference models acquired as a result of learning in a learning device; and a flowchart showing a modified example of the operation of an endoscopic system including a focus control device of one embodiment of the present invention.

[0020] The present invention will be described below with reference to illustrated embodiments, for example, as an application example to an endoscope system. The drawings used in the following description are schematic. Therefore, in these drawings, each component is shown at a size that allows it to be recognized. Therefore, the dimensional relationships and scales of each component may be shown differently for each component. The present invention is not limited to the illustrated embodiments with respect to the quantities, shapes, size ratios, relative positional relationships, and the like of each component shown in the drawings.

[0021] First, before describing the detailed configuration of a focus control device according to one embodiment of the present invention, the schematic configuration of an entire endoscope system including the focus control device of this embodiment will be described below with reference to Figures 1 and 2. Figure 1 is a schematic diagram showing the overall configuration of an endoscope system including a focus control device according to one embodiment of the present invention. Figure 2 is a block diagram showing the schematic internal configuration of the endoscope system.

[0022] 1, the endoscope system 1 is mainly configured to include a processor 10, an endoscope 20, a display device 30, a light source device 40, and a storage device 50. This endoscope system 1 illustrates a general configuration of an endoscope system used for endoscopic examinations in which the inside of organs such as the upper digestive tract (esophagus, stomach, duodenum, etc.) or the lower digestive tract (large intestine, etc.) of a subject (patient, etc.) such as a living body is observed.

[0023] The endoscope 20 is an image inspection device that includes an insertion section 21, an operation section 22, a universal cable 23, and the like.

[0024] The insertion section 21 is a component that is inserted into a subject such as a living organism. The insertion section 21 is formed by continuously connecting a distal section 21a, a bending section 21b, and a flexible tubular section 21c, in that order from the distal end. The insertion section 21 is formed in a generally elongated tubular shape. The insertion section 21 has a treatment tool insertion channel 21d therein, which is a conduit for inserting an endoscopic treatment tool (not shown). This treatment tool insertion channel 21d is provided so as to pass through the insertion section 21 from the distal end to the proximal end. The operation section 22 is connected to the proximal end side of the insertion section 21.

[0025] The tip portion 21a is a structural unit provided at the most distal end of the insertion portion 21. Various structural members (not shown in FIG. 1; see FIG. 2) such as an illumination unit 25 and an imaging unit 26 are provided inside the tip portion 21a.

[0026] Here, the illumination unit 25 is a structural unit that includes an optical element (illumination lens; not shown) that emits a light beam guided from the light source device 40 (described later) forward from the tip surface of the tip portion 21 a to illuminate an observation area including a lesion or the like within the subject.

[0027] The imaging unit 26 is an electronic device unit including an optical lens (imaging optical system 26a; see FIG. 2) that forms an optical image of a specific object to be observed inside the subject (a lesion such as a tumor, etc.), and a photoelectric conversion element (imaging element 26b; see FIG. 2) that generates image information (still image data, moving image data, etc.) based on the optical image.

[0028] The bending portion 21b is a tubular portion that can be actively bent by a bending operation mechanism (not shown) that acts in conjunction with the operation of a bending operation member 22b (described later). The flexible tube portion 21c is a tubular member that extends from the distal end of the operation portion 22 and is connected to the proximal end of the bending portion 21b.

[0029] The operation section 22 is connected to the proximal end of the insertion section 21. The operation section 22 is configured to include an operation section main body 22a, a bending operation member 22b, a plurality of operation members 22c, a treatment tool insertion port 22d, and the like.

[0030] The operation unit main body 22a has a generally box-like shape as a whole and constitutes a gripping portion that is held with the fingers of a user (such as a doctor) of the endoscope 20. Various constituent units such as a bending operation mechanism (not shown) are provided inside the operation unit main body 22a. As described above, the insertion section 21 extends from the operation unit main body 22a.

[0031] The bending operation member 22b and the plurality of operation members 22c are operation members for performing various operations of the endoscope 20. The bending operation member 22b and the plurality of operation members 22c are respectively provided at predetermined positions on the outer surface of the operation portion main body 22a.

[0032] The treatment instrument insertion port 22d is provided at a predetermined position near the distal end of the operation portion main body 22a. The treatment instrument insertion port 22d is a proximal end opening of the treatment instrument insertion channel 21d of the insertion portion 21. The treatment instrument insertion channel 21d is connected at its distal end to a distal channel opening (not shown) of the distal end portion 21a.

[0033] With this configuration, an endoscopic treatment tool (not shown) inserted through the treatment tool insertion port 22d is configured to pass through the treatment tool insertion channel 21d and then protrude outward from the tip channel opening of the tip portion 21a.

[0034] The universal cable 23 is a connection cord for connecting the endoscope 20 to the light source device 40 and the processor 10. For this purpose, the universal cable 23 is made of a tubular member extending from the side of the operation unit body 22a of the operation unit 22. A scope connector 23a is provided at the tip of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

[0035] An electric cable 23b extends from the scope connector 23a. A connector 23c is provided at the tip of the electric cable 23b. The connector 23c is connected to the front panel of the processor 10. Various signal transmission cables, optical fiber cables, etc. (not shown) are inserted into the universal cable 23.

[0036] The light source device 40 is a device that supplies illumination light to the illumination unit 25 (see FIG. 2) provided inside the tip portion 21a of the insertion section 21 of the endoscope 20. The illumination light emitted from the light source device 40 is transmitted to the illumination unit 25 in the tip portion 21a through an optical fiber cable (not shown) or the like that is arranged passing through the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21. The illumination light then passes through an illumination lens (not shown) included in the illumination unit 25 in the tip portion 21a and is irradiated toward the observation target area in front of the tip portion 21a.

[0037] The processor 10 is a control device, a signal processing device, or a circuit unit that includes a control circuit and a signal processing circuit that control the entire endoscope system 1. The processor 10 is also configured to include a function as a focus control device of this embodiment (described in detail below; see FIG. 2).

[0038] A control circuit included in the processor 10 receives, for example, an operation instruction signal from the operation member 22c of the operation section 22 of the endoscope 20, and outputs various control signals for driving and controlling the imaging unit 26, the light source device 40, the illumination unit 25, etc. Furthermore, a signal processing circuit included in the processor 10 receives, for example, an imaging signal from the imaging unit 26, and performs predetermined image signal processing, etc.

[0039] For this purpose, the processor 10 and the imaging unit 26 are electrically connected by a signal transmission cable (not shown). The signal transmission cable is inserted and arranged from the connector 23c through the electrical cable 23b, the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21 to the imaging unit at the tip end 21a.

[0040] With this configuration, control signals output from the processor 10 and imaging signals output from the imaging unit 26 are transmitted between the imaging unit 26 and the processor 10 through the signal transmission cable. Note that one form of the signal transmission cable is, for example, a composite cable in which a plurality of cables are bundled together and covered with an outer sheath shield, an outer sheath tube, or the like.

[0041] The display device 30 receives image signals and the like output from the processor 10 and displays endoscopic images and various types of information in a predetermined format. To this end, the display device 30 and the processor 10 are electrically connected using a video cable 24. The display device 30 may be configured as a display device using, for example, a general liquid crystal panel.

[0042] The storage device 50 is a storage device that saves (records or stores) image data for storage that is generated for storage after various processes are performed in the processor 10 based on image information generated by the imaging unit 26 .

[0043] As shown in Fig. 1, the storage device 50 is configured as an integral part of the processor 10. As shown in Fig. 2, a temporary storage unit 51 is provided inside the storage device 50 in addition to a main storage unit (not shown) in the storage device 50. The temporary storage unit 51 is configured, for example, by a semiconductor memory or the like, and is a component that functions as a temporary memory area for temporarily storing predetermined image data output from the image processing unit 11, etc.

[0044] 1 illustrates a configuration in which the storage device 50 is integrally disposed inside the housing of the processor 10, but the present invention is not limited to this. For example, the storage device 50 may be configured as an external storage device configured using a housing separate from the processor 10.

[0045] 2 shows an example of a configuration in which the temporary storage unit 51 is provided inside the storage device 50, but this is not limiting. The temporary storage unit 51 can also be provided integrally inside the storage control unit 13 (see FIG. 2), which will be described later, for example.

[0046] Furthermore, the processor 10 and the light source device 40 are not limited to being configured as separate entities, as in the example configuration shown in Fig. 1. For example, the processor 10 and the light source device 40 may be configured as an integrated unit using a single housing.

[0047] The configuration of the illumination unit 25 is not limited to the above-described configuration (a configuration in which illumination light from the light source device 40 is transmitted to the distal end portion 21a through an optical fiber cable or the like). As a configuration of the illumination unit 25 other than this, for example, a light-emitting element such as an LED (Light Emitting Diode) as an illumination light source can be provided inside the distal end portion 21a, and power supply to the illumination light source (LED) and light emission control thereof can be controlled by a predetermined control circuit included in the processor 10.

[0048] The endoscope system 1 configured as described above basically has substantially the same configuration as conventional endoscope systems of the same type, and therefore, illustrations and detailed descriptions of other detailed configurations will be omitted.

[0049] In the endoscope system 1 configured as described above, the focus control device of this embodiment is configured as being included in the processor 10. Next, the internal configuration of the processor in the endoscope system including the focus control device of one embodiment of the present invention will be described in detail below with reference to FIG.

[0050] The processor 10 is configured to include an image processing unit 11, a display control unit 12, a memory control unit 13, a specific object image detection unit 14, a distance distribution information acquisition unit 15, a focus control unit 16, an illumination control unit 17, a pattern classification unit 18, an illumination state determination unit 19, and a memory device 50 including a temporary memory unit 51.

[0051] The image processing unit 11 is a structural unit or circuit unit that receives an output signal (mainly image information) from the imaging unit 26 and performs various types of information processing based on the received image information. The image processing performed here includes, for example, normal image information processing, display image data processing, storage image data processing, and various other types of image processing.

[0052] The normal image processing includes an AD conversion process that converts the analog image signal output from the image sensor 26b into a digital image signal, and is a basic image information processing that is normally performed on image information. The normal image processing is assumed to be a conventional process, and a detailed description thereof will be omitted.

[0053] The display image data processing is processing for generating image data for normal display, such as processing for generating image data representing an endoscopic image to be displayed in a predetermined display area on the display screen of the display device 30. The storage image data processing is processing for generating image data for recording.

[0054] The various types of image processing are image processing that are appropriately added to normal image data for display. Specific examples of the various types of image processing include normal image adjustment processing such as brightness adjustment processing and white balance adjustment processing, and image enhancement processing such as edge enhancement processing and texture and color enhancement imaging (TXI).

[0055] The display control unit 12 is a structural unit or circuit unit that controls the display device 30 so that an appropriate display format is displayed according to the display image data output from the image processing unit 11. The display control unit 12 controls the display of the display image data (various settings such as display position, display area, display size, etc.) as well as the display of various information other than image information (text information, icon information, etc.). Here, the various information other than image information includes examination date and time information, subject (patient) information, medical record information, etc., as well as various alarm information (notification information) that is displayed as appropriate during an endoscopic examination, for example.

[0056] Specifically, the display control unit 12 performs display control to continuously display the endoscopic image in the display area, and when a specific object is detected in the endoscopic image, the display control unit 12 performs display control to add a predetermined display (for example, a frame display) to the image area including the specific object.

[0057] The memory control unit 13 is a structural unit or circuit unit that controls the memory device 50 (including the temporary memory unit 51) so that the image data for storage output from the image processing unit 11 is saved (recorded or stored) in an appropriate storage format.

[0058] The storage control unit 13 performs storage control to permanently store image data representing an endoscopic image acquired by the endoscope 20 in a storage area of ​​the storage device 50 .

[0059] The specific object image detection unit 14 is a structural unit or circuit unit that detects an image area including a specific object from an endoscopic image based on image data acquired by the endoscope 20 .

[0060] In detail, the specific object image detection unit 14 detects a specific object (such as a lesion such as a tumor) within each frame of endoscopic image that is displayed continuously based on the image data output from the image processing unit 11.

[0061] The specific object image detection unit 14 detects specific objects, for example, by performing similar image recognition based on multiple specific object images (case images, etc.) prepared in advance, or by performing data analysis using deep learning or machine learning to detect the desired specific object.

[0062] For example, in recent years, object detection technology using AI (Artificial Intelligence) has been developed that obtains desired inference results (e.g., detection of specific objects such as tumors) through machine learning of endoscopic images acquired by an endoscope.

[0063] Machine learning is a method of obtaining inference results about unknown matters by learning the features, time-series information, spatial information, etc. of known input information and making inferences based on the learning results. That is, in machine learning, a trained model (also called an inference model) is first obtained that enables inference of a determinable output result from specific input information.

[0064] In this case, when generating a trained model, a large amount of information with known input-output relationships is used as training data to obtain highly reliable inference results. For example, in deep learning, a network is designed using a large amount of training data so that expected outputs can be obtained for known inputs. The trained model obtained through this process can be used independently of the network that was used for training.

[0065] Therefore, by providing the processor 10 with this type of inference model, it becomes possible to accurately and quickly perform various judgments or detections, such as the detection of a specific object in the specific object image detection unit 14, as well as the judgment of the imaging distance range in which an image can be captured in a focused state during pan-focus control, the judgment of the usage state of the endoscope during an endoscopic examination, and the detection of the surface blood vessel pattern of a specific object (detection of information including the position of the blood vessel pattern in the image), etc. In this case, the specific object image detection unit 14 combines the functions of a specific object detection unit, an imaging distance range judgment unit, a blood vessel judgment unit, etc.

[0066] Here, the inference model is constructed roughly as follows. For example, a large amount of image data corresponding to input and output is provided as training data to a predetermined network (not shown) for constructing the inference model. Here, the training data includes image data suitable for detection or determination, such as image data of endoscopic images acquired sequentially in time series during an endoscopic examination, image data of images (such as case images) in which a specific object is captured in the endoscopic images, etc. In this case, an annotation may be set to surround the image area of ​​the specific object with a frame or the like.

[0067] By performing learning using a large amount of training data, a network design is determined for a specific network (not shown) so that an output corresponding to an input can be obtained. For example, Fig. 14 is an explanatory diagram of an inference model obtained as a result of learning in a specific learning device.

[0068] For example, as shown in Figure 14, when a group of endoscopic images (first training data group 201) containing a specific object (such as a tumor) is input, information about the specific object captured in the image and additional information such as adding a frame display surrounding the image area of ​​the specific object are obtained along with reliability information. This allows the construction of a first inference model 200 that detects the specific object from the endoscopic image.

[0069] When endoscopic image data (time-series image data) 202 taken during an endoscopic examination is input to the first inference model 200 constructed in this manner, the first inference model 200 detects an image area containing the specific object in a frame in which the specific object is captured from the endoscopic image data 202, and outputs image data (inference result image 203) in which a frame display is added to the image area of ​​the specific object. In this manner, a specific object can be detected from an endoscopic image. The image data of this image 203 can then be temporarily stored, or, if necessary, displayed in a specified display area of ​​the display device 30, providing a reference for the endoscope user during the endoscopic examination.

[0070] 14, when a group of endoscopic images (second training data group 204) annotated with endoscope usage status information is input to endoscopic images acquired during an endoscopic examination and associated with distance distribution information (depth map; described in detail below), information regarding the endoscope usage status during the endoscopic examination is obtained together with reliability information. This allows the construction of a second inference model 210 for determining the endoscope usage status during the endoscopic examination.

[0071] When endoscopic image data (time-series image data) 202 during an endoscopic examination is input to the second inference model 210 constructed in this manner, the second inference model 210 outputs the endoscope usage status from the endoscopic image data 202.

[0072] Furthermore, for example, when a group of endoscopic images (second training data group) annotated with information indicating a screening operation or a detailed observation operation is input to endoscopic images acquired during an endoscopic examination and associated with distance distribution information (depth maps), information regarding the operation state (screening operation or detailed observation operation) during the endoscopic examination can be obtained together with reliability information. This makes it possible to construct a distance distribution inference model for endoscopic images that can infer either a screening distance distribution pattern or a detailed observation distance distribution pattern from the distance distribution information of the object during the endoscopic examination.

[0073] When endoscopic image data during an endoscopic examination is input to the distance distribution inference model constructed in this manner, the inference model outputs either a screening distance distribution pattern or a detailed observation distance distribution pattern from the endoscopic image data.

[0074] In this case, if training data annotated with information indicating the appropriate state of autofocus for multiple distance distributions is used, a distance distribution inference model capable of inferring the appropriate state of autofocus can be created. Such an inference model may be configured, for example, by being included in the pattern classification unit 18 in the processor 10.

[0075] "Deep learning" is a multilayered version of the "machine learning" process using neural networks. A typical example is a "forward propagation neural network," which sends information from front to back and makes a judgment. In its simplest form, it requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons determined by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights, and a bias value is added between the hidden and output layers, making it easy to form logic gates. While three layers are sufficient for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, models with 9 to 152 layers have become practical due to their training time, judgment accuracy, and energy consumption.

[0076] Various well-known networks may be used for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) using CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses image features, operates with minimal processing, and is strong in pattern recognition. Furthermore, "recurrent neural networks" (fully connected recurrent neural networks) that can handle more complex information and allow information analysis whose meaning changes depending on the order or sequence of information may be used.

[0077] To realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, devices specialized for matrix calculations such as GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) may also be used. In recent years, such dedicated artificial intelligence (AI) hardware "neural network processing units (NPUs)" have been designed to be integrated and embeddable with CPUs and other circuits, and may even become part of the processing circuit.

[0078] Furthermore, inference models may be obtained by employing various well-known machine learning techniques, not limited to deep learning. For example, techniques such as support vector machines and support vector regression are available. Here, learning involves calculating the weights, filter coefficients, and offsets of a classifier; other techniques include using logistic regression processing. When a machine is to make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method for deriving an image judgment using machine learning is employed. However, a rule-based method for applying rules acquired by humans through experience or heuristics to make a specific judgment may also be used.

[0079] The distance distribution information acquisition unit 15 is a structural unit or circuit unit that acquires distance distribution information (hereinafter simply referred to as distance distribution information or depth map) within the imaging screen based on the output signal (image information or phase difference information; described later) from the imaging element 26b.

[0080] Here, distance distribution information (depth map) within the captured image refers to information that represents the distribution of distances in the forward and backward directions for objects, including specific objects that are the subject of observation, that are captured within the captured image.

[0081] Furthermore, the front-to-back direction of an object refers to the front-to-back direction (in other words, the depth direction) when viewing an object including a specific target from the imaging unit 26 (image sensor 26b) provided at the tip 21a of the endoscope 20, and is the direction along the direction in which the endoscope 20 moves forward and backward, and is the direction along the insertion axis of the endoscope 20.

[0082] As described above, the image pickup element 26b included in the imaging unit 26 of the endoscope 20 applied in this embodiment has a plurality of pixels that receive light beams from a subject and output image signals, and some or all of these plurality of pixels perform imaging plane phase difference detection, and an image pickup element having a configuration that can contribute to, for example, autofocus control (AF control), predetermined focus control, etc. In this embodiment, the application of this type of image pickup element 26b makes it possible to acquire image information and phase difference information.

[0083] Here, a configuration that enables image-surface phase difference detection using an image sensor will be briefly described below. Fig. 12 is a schematic diagram illustrating the principle of image-surface phase-difference AF control that performs image-surface phase difference detection using an image sensor.

[0084] 12 indicates one pixel of the image sensor 26b. The pixel 260 has a photodiode 262 and a microlens 261. Here, the photodiode 262 is formed in a two-part configuration, for example, a right photodiode 262R and a left photodiode 262L.

[0085] The microlens 261 is an optical lens that transmits a light beam that has passed through the imaging optical system 26a and forms an optical image on the light receiving surface of the photodiode 262. The photodiode 262 is a photoelectric conversion element that converts the optical image formed by the microlens 261 into an electrical signal and outputs it.

[0086] The right photodiode 262R receives a light beam LL from the left region (the region indicated by left-hand diagonal hatching in FIG. 12) and forms an image on the light-receiving surface, while the left photodiode 262L receives a light beam LR from the right region (the region indicated by right-hand diagonal hatching in FIG. 12) and forms an image on the light-receiving surface.

[0087] The pixel 260 configured as described above can calculate the distance between the two images (parallax) from the output signals (phase difference information) of the left and right photodiodes 262R and 262L to determine the defocus amount for the object in the imaging optical system. Furthermore, by combining the output signals of the left and right photodiodes 262R and 262L, the combined signals can be treated as an image signal for one pixel.

[0088] The distance distribution information acquisition unit 15 receives the phase difference information from the output signal acquired by the image sensor 26b in this manner and acquires distance distribution information (depth map) of a predetermined region within the imaging plane (e.g., a region including a specific object). The acquired depth map can be represented in the form of a graph, for example, as shown in Figures 5, 7, 9, and 11. A detailed description of this depth map will be given later.

[0089] 12, one photodiode is divided into two and the signals from the left and right photodiodes are handled separately to perform pupil division of the microlens, but the present invention is not limited to this configuration. For example, a configuration in which pupil division is performed by using a light-shielding member can be similarly realized.

[0090] In the above-described configuration example, the distance distribution information acquisition unit 15 acquires distance distribution information (depth map) from phase difference information from the image sensor 26 b having a plurality of pixels for acquiring image signals and detecting phase differences on the imaging surface, but the present invention is not limited to this configuration. For example, in a typical image sensor, all pixels provided on the imaging surface acquire imaging signals.

[0091] In an imaging unit having such a conventional imaging element 26b, for example, at least one of the optical lenses constituting the imaging optical system 26a, which is an optical lens that contributes to focus control (hereinafter referred to as the focus lens), is slightly moved back and forth along the optical axis to change the lens position. This causes a change in the contrast of objects in the image, thereby changing the focus state. At this time, distance information can be obtained from the correlation between the lens position and the focus state corresponding to the lens position. Based on the distance information thus obtained, distance distribution information (depth map) within the screen can be obtained. In other words, a configuration can be adopted in which distance distribution information (depth map) is obtained based on image signals from the imaging element 26b.

[0092] In this way, the distance distribution information acquisition unit 15 acquires a depth map based on the output signal (phase difference information or image information) of the image sensor 26 b. That is, the endoscopic image generated based on the output signal (image signal) of the image sensor 26 b and the depth map generated based on the output signal (phase difference information or image information) of the image sensor 26 b are related to and correspond to each other.

[0093] Furthermore, it is known that the multiple endoscopic images sequentially acquired in time series by the image sensor 26b have corresponding image patterns, i.e., distance distribution patterns, depending on the usage state of the device, such as the endoscope. Therefore, the image patterns or distance distribution patterns can be categorized for each usage state of the endoscope. In other words, when an image pattern or distance distribution pattern similar to a categorized image pattern or distance distribution pattern is detected, the usage state of the endoscope can be determined. Therefore, the processor 10 of this embodiment is equipped with a pattern classification unit 18.

[0094] The pattern classification unit 18 is a structural unit or circuit unit that detects and classifies distance distribution patterns based on the depth map acquired by the distance distribution information acquisition unit 15, and determines the usage state of the endoscope 20 during endoscopic examination based on the classification results. In this case, the detection and classification of distance distribution patterns and the determination of the usage state of the endoscope 20 may be performed using the second inference model 210, as described above.

[0095] Here, the distance distribution patterns classified by the pattern classification unit 18 and the distance distribution patterns corresponding to the usage state of the endoscope 20 determined in accordance with the classified distance distribution patterns include, for example: (1) a first distance distribution pattern during a screening state, which is a first observation state in which a specific object (such as a lesion, e.g., a tumor) is searched for while observing the inside of a lumen of an organ, e.g., of a subject; (2) a second distance distribution pattern during an approach state, which is a second observation state in which the tip of the endoscope approaches a specific object from the first observation state (screening state); (reduced to an approach distance distribution pattern other than the screening distance distribution pattern and the observation distance distribution pattern); and (3) a third distance distribution pattern during a detailed observation state, which is a third observation state in which a specific object is observed in detail (in other words, an observation distance distribution pattern that represents a detailed observation operation at a close range close to the specific object). The distance distribution patterns detected and classified from the depth map will be described later (see FIGS. 5, 7, 9, and 11).

[0096] The focus control unit 16 is a structural unit or circuit unit that controls the driving of the imaging optical system 26a of the imaging unit 26. The focus control unit 16 controls the driving of the imaging optical system 26a based on the detection result of the specific object image detection unit 14, and performs, for example, a predetermined focus adjustment operation (hereinafter referred to as focus control), pan focus control, and near-field priority focus control (or observation priority focus control) (described in detail later).

[0097] The focus control unit 16 also functions as a focus position control unit that controls the drive of the imaging optical system 26a based on the determination result of the pattern classification unit 18 and adjusts the relative position on the optical axis between the focus lens and the imaging element. The focus control unit 16 also controls switching to an appropriate focus control depending on the usage state of the endoscope 20 based on the determination result of the pattern classification unit 18. Here, the focus lens refers to at least some of the optical lenses that make up the imaging optical system 26a and contribute to focus control.

[0098] The usage states of an endoscope during an endoscopic examination can be broadly divided into, for example, a screening state in which lesions are searched for, a state in which a specific detected object is observed in detail (referred to as a detailed observation state), and a transition state in which the tip of the endoscope is moved closer to the specific object being observed from the screening state in order to perform detailed observation (referred to as an approach state).

[0099] When performing endoscopic examinations using conventional medical or testing equipment such as endoscopes, it is necessary to perform appropriate focus control of the imaging unit depending on the usage conditions of the endoscope, etc., in order to always ensure a reliable observation environment.

[0100] Here, appropriate focus control according to the usage state of the endoscope 20 is, for example, as follows.

[0101] First, observation in the first observation state (screening state) is performed with the imaging unit and the object being observed at a slight distance (for example, about a few centimeters). In this case, sufficient observation can be performed within the depth of field of the imaging optical system. Therefore, the focus control of the imaging unit in this state is usually so-called pan-focus control.

[0102] In this case, the deep focus control is, for example, a focus control in which the imaging optical system is positioned so that an object within a predetermined distance range (for example, within a range of about 1 to 3 centimeters in front of the lens tip surface) is generally in focus, and the position is maintained, etc. In this case, the position of the imaging optical system (for example, the focus lens) set during deep focus control is referred to as the first focus position.

[0103] Furthermore, the pan focus control performed here may set a predetermined focus lens position after performing fine focus control according to distance, or it may be focus control that selects one of several predetermined position settings, or it may be two-stage switching control that simply changes from the pan focus state to another predetermined focus lens position (close-distance compatible focus position).

[0104] On the other hand, when in the second observation state (approach state), it is necessary to continue observing the target specific object so as not to lose sight of it. Furthermore, in the approach state, the distal end of the endoscope (imaging unit) is gradually moved closer to the specific object. For this reason, if the endoscope is moved while using pan focus control in the approach state, it becomes increasingly difficult to maintain a focused state. Therefore, in this case, it is desirable to use autofocus control.

[0105] The autofocus control performed here is a control that drives the focus lens of the imaging optical system to move in the optical axis direction so that a predetermined object located within approximately 3 centimeters in front of the lens tip can always be continuously in focus. In this case, focus control that takes into account, for example, the depth of field is required so that a predetermined range in the forward and backward directions of the target specific object can always be in focus.

[0106] On the other hand, observation in the third observation state (detailed observation state) is often performed when the imaging unit is close to the object being observed (for example, within about 3 millimeters of the object). In this case, it is known that the depth of field becomes extremely shallow. Therefore, it is desirable that the focus control of the imaging unit in this state be focus control suitable for so-called close-range imaging (observation) (hereinafter referred to as close-range priority focus control or observation priority focus control) (described in detail below).

[0107] In addition to these focus controls, various other forms of autofocus control have recently become available, such as a form in which autofocus control is performed by tracking a predetermined object detected within the imaging screen, a form in which autofocus control is performed by preferentially selecting either a distant object or a close object within the imaging screen, a form in which autofocus control is performed on an object included in an area within the imaging screen that is arbitrarily specified by the user, and focus control that ensures a good focus state within a predetermined distance range in front of the imaging unit by taking into account the depth of field of the imaging optical system. The focus control unit 16 can apply any of these various autofocus control forms by appropriately selecting or combining them.

[0108] The illumination control unit 17 is a structural unit or circuit unit that drives and controls the light source device 40 and the illumination unit 25. The illumination control unit 17 controls the light source device 40 or the illumination unit 25 in response to operation instruction signals from various operation members 22c provided in the operation unit 22, or instruction signals based on the detection results of the specific object image detection unit 14 or the determination results of the pattern classification unit 18, and performs switching control of the light source type (white light, special light, etc.).

[0109] Here, the light source types include, for example, white light that is irradiated onto the subject when performing normal observation, as well as predetermined special light that is irradiated onto the subject when performing image-enhanced observation such as narrow band imaging (NBI) or red dichromatic imaging (RDI).

[0110] The illumination condition determination unit 19 is a structural unit or circuit unit that determines the illumination condition based on the exposure distribution within the screen of the endoscopic image generated based on the output signal (image information) from the imaging unit 26. Here, the illumination conditions in the endoscopic image include, for example, a first illumination condition that is an appropriate (good) exposure condition, a second illumination condition that is an overexposed condition, and a third illumination condition that is an underexposed condition. In this case, the second illumination condition causes so-called blown-out highlights, while the third illumination condition causes so-called crushed shadows.

[0111] Therefore, in the focus control device of this embodiment, the illumination condition determination unit 19 determines a proper exposure area in the endoscopic image screen that will be in a first illumination condition, and performs focus control so as to obtain a focused state in an area limited to the range of the proper exposure area (described in detail below). This completes the configuration of the endoscope system 1 including the focus control device of this embodiment. Other configurations that are not described here are similar to those of conventional endoscope systems with a general configuration.

[0112] It should be noted that all or part of the processor 10 includes hardware. Here, the processor 10 is configured by a well-known configuration including, for example, a central processing unit (CPU), random access memory (RAM), read only memory (ROM), non-volatile memory, non-volatile storage, and a non-transitory computer readable medium, as well as peripheral devices thereof.

[0113] Software programs to be executed by the CPU and fixed data such as data tables are stored in advance in ROM, nonvolatile memory, nonvolatile storage devices, etc. The CPU reads out the software programs stored in ROM, etc., expands them into RAM, and executes them, and the software programs refer to various data, etc. as appropriate, thereby realizing the functions of the processor 10.

[0114] The processor 10 may also be configured with a semiconductor chip such as an FPGA (Field Programmable Gate Array), etc. Furthermore, the processor 10 may also be configured with an electronic circuit.

[0115] Furthermore, the software program may be recorded or stored, in whole or in part, as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.

[0116] The operation of the endoscope system 1 including the focus control device of this embodiment configured as described above will be described below. Fig. 3 is a flowchart explaining the operation of the endoscope system including the focus control device of one embodiment of the present invention. The flowchart shown in Fig. 3 is a diagram explaining the operation when an endoscopic examination is performed using the endoscope system 1.

[0117] First, it is assumed that the endoscope system 1 is activated and ready to perform an endoscopic examination. At this time, it is assumed that the endoscope 20 in the endoscope system 1 is inserted into the organ of a subject (e.g., a patient) to be examined. It is assumed that the focus control setting in the initial state after activation of the endoscope 20 is, for example, a pan-focus control setting.

[0118] When the endoscopic system 1 of this embodiment is in this state, in step S1 of Figure 3, the image processing unit 11 of the processor 10 acquires an output signal (mainly image information) from the imaging unit 26 and starts predetermined image processing based on the output signal (image information).

[0119] In addition, the illumination condition determination unit 19 of the processor 10 starts to acquire exposure distribution information within the screen of the endoscopic image based on the endoscopic image data generated based on the image data processed by the image processing unit 11.

[0120] At the same time, the distance distribution information acquisition unit 15 of the processor 10 starts acquiring distance distribution information (depth map) based on the output signal (mainly phase difference information) from the imaging unit 26 .

[0121] Next, in step S2, the processor 10 detects a distance distribution pattern based on the various information (image information, depth map, exposure distribution information, etc.) acquired in the processing of step S1 described above. Then, the pattern classification unit 18 executes processing to determine the usage state of the endoscope 20 based on the various detected information. Note that the processing of steps S1 and S2 is executed continuously while the endoscopic examination is being performed. Then, the processing proceeds to step S3.

[0122] Here, as described above, the usage states of the endoscope 20 include a first observation state (hereinafter referred to as the screening state), a second observation state (hereinafter referred to as the approach state), and a third observation state (hereinafter referred to as the detailed observation state).

[0123] 4 and 6 are schematic diagrams showing the state of the endoscope 20 in the screening state. Of these, Fig. 4 shows a first state (during normal search) representing a state in which a specific object has not been detected in the screening state (during object search). Also, Fig. 6 shows a second state (during specific object detection) representing a state in which a specific object has been detected in the screening state.

[0124] In Fig. 4, reference numeral 300 schematically indicates an organ or the like having a lumen. Fig. 4 shows the insertion section 21 (tip portion 21a) of the endoscope 20 being inserted into the lumen of the organ or the like 300. Note that reference numeral 301 in Fig. 4 indicates a specific object such as a tumor present on the inner wall of the organ or the like 300 (details will be described later). Note that under the circumstances shown in Fig. 4, the specific object 301 is in an undetected state. That is, at this time, the specific object 301 is outside the field of view of the endoscope 20.

[0125] 4, the symbol O indicates the central axis of the insertion portion 21 of the endoscope 20 (or may be considered as the optical axis of the imaging optical system 26a of the imaging unit 26; hereinafter, simply referred to as the central axis). The insertion portion 21 of the endoscope 20 is operated so that this central axis O advances and retreats in a direction along the lumen of the organ 300, etc. The symbol V in FIG. 4 indicates the imaging range (field of view) of the imaging optical system 26a of the imaging unit 26 built into the tip portion 21a.

[0126] When the endoscope 20 is in the screening state, as shown in Fig. 4, the insertion section 21 of the endoscope 20 is arranged in a substantially linear form along the lumen of the organ 300. For example, when the tip section 21a of the endoscope 20 is in position [A] shown in Fig. 4, a depth map generated based on the output signal (phase difference information) from the imaging unit 26 is expressed in the form shown in Fig. 5.

[0127] Here, Fig. 5 is a display example of distance distribution information (depth map) when the endoscope is in the screening state and the tip portion 21a of the endoscope 20 is at the position [A] in Fig. 4. Note that Fig. 7 is a display example of distance distribution information (depth map) when the endoscope is in the screening state and the tip portion 21a of the endoscope 20 is at the position [B] in Fig. 6.

[0128] First, when the endoscope 20 is at position [A] in Fig. 4, the depth map takes the form shown in Fig. 5, for example. Here, the vertical axis in the depth map in Fig. 5 represents the reciprocal (1 / D) of the distance D. This means that in Fig. 5, the further upward from the origin, the closer the distance distribution. Furthermore, the horizontal axis in the depth map in Fig. 5 corresponds to the directions of arrows Y1 and Y2 in Fig. 4. This means that in Fig. 5, the further left or right from the origin, the closer the position on the screen that corresponds to a position farther away in the Y1 direction or the Y2 direction in Fig. 4.

[0129] Note that the arrow Y1 in Fig. 4 indicates the downward direction as seen from the endoscope 20 in Fig. 4. Therefore, the direction of the arrow Y1 on the horizontal axis in Fig. 5 indicates the distance distribution in the downward region as seen from the endoscope 20. Similarly, the arrow Y2 in Fig. 4 indicates the upward direction as seen from the endoscope 20 in Fig. 4. Therefore, the direction of the arrow Y2 on the horizontal axis in Fig. 5 indicates the distance distribution in the upward region as seen from the endoscope 20. In other words, each end of the arrows Y1 and Y2 in Fig. 5 indicates the distance distribution in the vertical screen periphery of the endoscopic image. These symbols are the same in Figs. 7, 9, and 11.

[0130] The depth map when the endoscope 20 is in the state shown in FIG. 4 (specific object undetected state) shows an overall flat distance distribution pattern, as shown in FIG. 5. In this case, the central region corresponds to the front region of the endoscope 20 and shows a dark region where the illumination light has not reached. This dark region shows a long-distance distance distribution. In addition, the peripheral region (the ends of the horizontal axis Y1 and Y2) shows a short-distance distribution (bright region) that shows the unevenness of the inner wall surface of the organ 300, etc. Therefore, in the distance distribution pattern shown in FIG. 5, some distance distribution can be seen in both end regions of the horizontal axis.

[0131] Next, assume that the endoscope 20 moves in the direction of the arrow X in Figure 4 from position [A] in Figure 4 (see the dotted line in Figure 6) to position [B] in Figure 6 (see Figure 6). Here, when the endoscope 20 is in the state of Figure 6 (specific object detection state), the depth map will have a form such as that shown in Figure 7, for example. In the depth map shown in Figure 7, a dark area is shown in the central region as a long-distance distribution where the illumination light has not reached. In addition, in the peripheral region, the irregularities of the inner wall surface of the organ 300 are shown as a bright area as a short-distance distribution. Furthermore, in the example of Figure 7, a distance distribution corresponding to the specific object 301 is shown as a bright area as a short-distance distribution in the peripheral region in the Y1 direction of the depth map.

[0132] Therefore, when the pattern classification unit 18 detects a distance distribution pattern from the depth map in which a long-distance distribution (dark area) is present in the central area and a short-distance distribution (bright area) is shown in the peripheral area (as shown in Figures 5 and 7), it classifies this as a first distance distribution pattern.

[0133] Generally, when the endoscope 20 is in the screening state, as described above, a lumen extends to the front region of the endoscope 20. In this situation, the illumination light from the illumination unit 25 does not sufficiently reach the front region of the endoscope 20.

[0134] Therefore, when the endoscope 20 is in the screening state, the depth map often has a tendency to have a long-distance distribution (dark area) in a predetermined range in the central area and a short-distance distribution (bright area) in the peripheral area. In this case, the specific object 301 in the depth map generally has a distance distribution in the bright area of ​​the peripheral area.

[0135] For this reason, when the pattern classification unit 18 detects a first distance distribution pattern in which a long-distance distribution (dark area) exists in the central area of ​​the depth map and a short-distance distribution (bright area) exists in the peripheral area, it can be determined that the endoscope 20 during the endoscopic examination is in a screening state (first observation state).

[0136] On the other hand, consider the case where a specific object is detected when the endoscope 20 is in the screening state (the situation shown in FIGS. 6 and 7).

[0137] Here, when a specific object 301 is detected, it may be detected by the specific object image detection unit 14, or when a user of the endoscope 20 (such as a doctor) recognizes a specific object in the image by visually observing the endoscopic image.

[0138] When the specific object 301 is detected by the specific object image detection unit 14, the detection result is notified to the user (doctor, etc.) of the endoscope 20 in a predetermined format. In this case, the notification is displayed in a predetermined format on the display screen of the display device 30. Alternatively, a notification display such as audio may be used. This allows the user (doctor, etc.) of the endoscope 20 to recognize that the specific object 301 has been detected.

[0139] In this way, when the user (doctor, etc.) of the endoscope 20 recognizes the specific object 301, the user (doctor, etc.) may wish to perform a detailed observation of the specific object 301. To do so, the user (doctor, etc.) of the endoscope 20 performs an operation to bring the distal end 21a of the endoscope 20 closer to the specific object 301. By performing this operation, the usage state of the endoscope 20 transitions from the screening state to the approach state.

[0140] 8 is a schematic diagram showing the state of the endoscope 20 when it is in the approach state. As shown in Fig. 8, when the endoscope 20 is in the approach state, the insertion section 21 of the endoscope 20 is bent with respect to the bending portion 21b, and the bending portion 21b is bent by a predetermined amount in a predetermined direction. The bending direction of the bending portion 21b at this time is set to a direction in which the distal end surface of the distal end portion 21a of the endoscope 20 is positioned to face the detected specific object 301.

[0141] That is, at this time, the insertion portion 21 of the endoscope 20 is positioned so that the tip surface faces the specific object 301, and the central axis O of the insertion portion 21 extends from the tip surface of the insertion portion 21 of the endoscope 20 toward the specific object 301.

[0142] In this state, the user (doctor, etc.) performs an operation to gradually bring the tip portion 21a of the endoscope 20 closer to the specific object 301. The depth map during this operation is, for example, as shown in Fig. 9. Fig. 9 shows an example of the display of distance distribution information (depth map) when the endoscope is in the approach state.

[0143] 9, the depth map when the endoscope 20 is in the state shown in Fig. 8 has a distance distribution pattern in which there is a large amount of short-distance distribution on the side of the central region where the specific object 301 exists (the peripheral region side in the direction of the arrow Y1), and there is a small amount of short-distance distribution on the side where the specific object 301 does not exist (the peripheral region side in the direction of the arrow Y2). The depth map in this case shows that the specific object 301 exists in one peripheral region (Y1 side) from the central region, and a long-distance distribution (dark region) exists in the other peripheral region (Y2 side).

[0144] When the endoscope 20 is in the approach state, a specific object 301 is present in the forward region of the endoscope 20 within a range sufficiently reached by illumination light from the illumination unit 25. At this time, the illumination light is irradiated onto the inner wall of the organ 300 near the specific object 301, forming a bright region. However, at the end on the side where the specific object 301 is not present, there is a dark region where the illumination light does not reach. Therefore, when the endoscope 20 is in the approach state, the depth map tends to have a short-distance distribution (bright region) present over almost the entire area, but a long-distance distribution (dark region) present in part of the peripheral region. It has been found that the closer the tip 21 a of the endoscope 20 is to the specific object 301, the gradually decreasing long-distance distribution (dark region) in the peripheral region.

[0145] Furthermore, when the endoscope 20 is in the approach state, the distal end surface of the distal end portion 21a of the endoscope 20 is positioned facing the specific object 301, as shown in Fig. 8. Therefore, the central region of the depth map at this time shows a distance distribution indicating the presence of the specific object 301.

[0146] Therefore, when the endoscope 20 is in the approach state, the depth map often shows a tendency for a short-distance distribution (bright area) to exist in one peripheral area (Y1 side) from the central area, and a long-distance distribution (dark area) to exist in a portion of the peripheral area (Y2 side).

[0147] Therefore, when the pattern classification unit 18 detects a distance distribution pattern from the depth map in which a short-distance distribution (bright area) is shown from the central area to the peripheral area, and a long-distance distribution (dark area) is shown in a part of the peripheral area (as shown in Figure 9), it classifies this as a second distance distribution pattern.

[0148] During an endoscopic examination, the state change when the endoscope 20 transitions from the screening state to the approach state can be determined by detecting the change over time in the output signal from the imaging unit 26, which is acquired sequentially in chronological order.

[0149] As described above, the depth map in the screening state can be determined by a distance distribution pattern (first distance distribution pattern) in which a long-distance distribution (dark area) exists in the central region. The state when transitioning from the screening state to the approach state can be determined by detecting a state in which the long-distance distribution (dark area) in the central region of the depth map gradually changes over time to a short-distance distribution (bright area).

[0150] Therefore, if a time series change is detected in the depth map during the screening state, in which the long-distance distribution (dark area) in the central area gradually decreases and becomes a short-distance distribution (bright area), it can be determined that the state has transitioned from the screening state (first distance distribution pattern) to the approach state.

[0151] Note that although the situation in which the long-distance distribution (dark area) in the central area changes to the short-distance distribution (bright area) is exemplified as a "gradual" change, there may also be situations in which the change occurs "suddenly and all at once," for example.

[0152] Furthermore, during the approach state, some areas of the specific object 301 may have an extremely short-distance distribution. In such areas, the illumination light is strongly irradiated, resulting in excessive exposure distribution and what is called blown-out highlights. Meanwhile, in the depth map during the approach state, some long-distance distributions (dark areas) may have insufficient exposure distribution and what is called crushed shadows. Figure 13 is a conceptual diagram of an endoscopic image obtained under such circumstances.

[0153] In Fig. 13, reference numeral 100 denotes a display screen frame for an endoscopic image. The endoscopic image is displayed within this display screen frame 100. Here, reference numeral 101a in Fig. 13 denotes a main portion of a specific object captured in the endoscopic image. Reference numeral 101b denotes an organ wall surface at the periphery of the specific object. Reference numeral 101c denotes a blown-out highlight portion of the organ wall surface located on the close side of the imaging unit 26. Reference numeral 101d denotes a crushed-black portion located on the far side of the imaging unit 26.

[0154] 13 shows a situation in which such an endoscopic image is acquired, in which a specific object is in focus and the exposure distribution is appropriate. Also, the peripheral portion 101b is in focus and the exposure distribution is appropriate.

[0155] On the other hand, in the close distance region 101c, the exposure distribution is excessive, resulting in overexposure and unclear focus. Furthermore, in the long distance region 101d, the exposure distribution is insufficient, resulting in underexposure and unclear focus. Endoscopic images captured during the approach state often exhibit such conditions.

[0156] Furthermore, in the approach state, the distal end 21a of the endoscope 20 is gradually moved closer to the specific object 301. As a result, in the approach state, the distance distribution corresponding to the specific object 301 in the central region tends to gradually shift toward closer distances. At this time, the long-distance distribution (dark region) present in a portion of the screen gradually decreases.

[0157] Taking these factors into consideration, if the pattern classification unit 18 detects a distance distribution pattern (second distance distribution pattern) in which a short-distance distribution (bright area) exists from the central area of ​​the depth map to the peripheral area, a long-distance distribution (dark area) exists in a portion of the peripheral area, and a specific object 301 exists in approximately the central area, it can be determined that the endoscope 20 during endoscopic examination is in an approach state (second observation state).

[0158] Next, consider the case where the endoscope 20 transitions from the approach state to the detailed observation state. In the approach state during an endoscopic examination, after the distal end portion 21a of the endoscope 20 has reached a sufficiently close distance to the specific object 301 that is the target of observation, an operation is performed to perform detailed observation of the specific object 301 while maintaining the distance between the distal end surface of the distal end portion 21a and the specific object 301. By performing such an operation, the usage state of the endoscope 20 transitions from the approach state to the detailed observation state.

[0159] FIG. 10 is a schematic diagram showing the state of the endoscope 20 when it is in the detailed observation state. As shown in FIG. 10 , when the endoscope 20 is in the detailed observation state, the bending section 21b of the endoscope 20 is maintained bent by a predetermined amount in a predetermined direction, and the distal end surface of the distal end section 21a is positioned in a position facing and close to the specific object 301 that is the observation target. Then, a user (such as a doctor) performs operations to observe the specific object 301 from various angles while pointing the distal end section 21a of the endoscope 20 toward the specific object 301. The depth map during such operations is as shown in FIG. 11 . Here, FIG. 11 is a display example of distance distribution information (depth map) when the endoscope is in the detailed observation state.

[0160] When the endoscope 20 is in the state shown in Figure 10, the depth map has a short-distance distribution (bright area) over almost the entire area, with a specific object 301 located in the central area, as shown in Figure 11, for example.

[0161] Generally, when the endoscope 20 is in the detailed observation state, a specific object 301 is present at a close distance in a position facing the distal end surface of the distal end portion 21a of the endoscope 20. The user (doctor, etc.) of the endoscope 20 observes the specific object 301 from various angles while always capturing the specific object 301 on the screen. For this reason, the depth map in the detailed observation state tends to always have a specific object with a close distance distribution in the central region, and the distance distribution of the specific object 301 tends to change little over time.

[0162] For this reason, when the pattern classification unit 18 detects a distance distribution pattern in which the distance distribution of an object in the central region of the depth map is closer than a predetermined distance (for example, 3 millimeters), it can be determined that the endoscope 20 during the endoscopic examination is in a detailed observation state (third observation state), and the distance distribution pattern at this time is classified as the third distance distribution pattern.

[0163] 3 , in the next step S3, the processor 10 (pattern classification unit 18) checks whether the distance distribution pattern determination result obtained by the processing in the above-mentioned step S2 is the first distance distribution pattern (screening distance distribution pattern) (distance distribution pattern classification step). If the determination result is the first distance distribution pattern, it is determined that the state is in the screening state, and the processing proceeds to step S12. If the determination result is other than the first distance distribution pattern, the processing proceeds to step S4.

[0164] Generally, when an endoscope is in a screening state, observation is performed while the endoscope is moved within the lumen of an organ, etc. In this case, although the endoscope is actually moving, it can be said that an object present on the inner wall surface of the organ, etc., is moving relative to the endoscope. If focus control is performed each time on an object that moves relative to the endoscope (imaging unit), the distance measurement operation and focus lens movement become complicated, which may make it difficult to obtain accurate and stable images. Therefore, during screening operations, it is preferable to fix the focus lens in a predetermined position and use pan-focus control that can ensure a focused state within a predetermined imaging distance range.

[0165] 3, if it is determined that the imaging optical system is in the screening state, the processor 10 causes the focus control unit 16 to perform pan focus control in step S12, fixing the imaging optical system at the first focus position (focus control switching step).Then, the process proceeds to step S11.

[0166] On the other hand, when the process proceeds from step S3 to step S4, the processor 10 checks whether a specific object has been detected in step S4. If the detection of a specific object is confirmed, the process proceeds to step S5. If the specific object is not detected, the process proceeds to step S11.

[0167] In step S5, the processor 10 checks whether the distance distribution pattern determination result is the second distance distribution pattern (approach distance distribution pattern) (distance distribution pattern classification step). If the determination result is the second distance distribution pattern, it is determined that the vehicle is in an approach state, and the process proceeds to step S6. If the determination result is not the second distance distribution pattern, the process proceeds to step S9.

[0168] Furthermore, when an endoscope is in the approach state, it is generally desired to observe not only the object to be observed (imaged) but also a predetermined range including the peripheral portion of the object in a single field of view. Therefore, it is preferable to perform focus control during the approach operation taking into account the object and its peripheral portion.

[0169] Typically, when in the approach state, imaging is performed from a position slightly farther away than from a close-up position. In this case, a deeper depth of field can be ensured. Therefore, by utilizing a deep depth of field, it becomes easier to ensure a focused state within a predetermined range even when there is a difference in distance between the front and rear of the object. In this way, by performing focus control that takes into account the object and its peripheral areas, it becomes possible to observe a wide range including the object in a focused state from a bird's-eye view.

[0170] Here, focus control that takes into account the object and its peripheral areas is assumed to be, for example, focus control such as a focus adjustment operation that sets a focus point at a predetermined position between the main part of the object being observed and the peripheral area on the front side of the object.

[0171] Returning to Figure 3, if a second distance distribution pattern (approach distance distribution pattern) is detected in the processing of step S5 described above, in step S6, the processor 10 determines the focusing distance L1, which is the imaging distance at which the main part of the specific object is in focus, based on the depth map.

[0172] Next, in step S7, the processor 10 detects a proper exposure area (first lighting condition) based on the exposure distribution information, and determines a focus distance L2 that can obtain a focus state of a well-exposed area in the surrounding area including the specific object based on the depth map.

[0173] Next, in step S8, focus control is performed so that both focus distances L1 and L2 are included in the in-focus state. In other words, in the processing of steps S6 to S8 (focus control switching step) that is performed when the second distance distribution pattern (approach distance distribution pattern) is detected, focus position control is performed to continuously adjust the focus to a position (target) corresponding to a specific distance included in the acquired distance distribution information. Then, the processing proceeds to step S11.

[0174] In step S9, the processor 10 checks whether the distance distribution pattern determination result is the third distance distribution pattern (observation distance distribution pattern) (distance distribution pattern classification step). If the determination result is the third distance distribution pattern, it is determined that the detailed observation state is in progress, and the process proceeds to step S10. If the determination result is not the third distance distribution pattern, the process proceeds to step S11.

[0175] Generally, when the endoscope is in the detailed observation state, the distal end 21 a (imaging unit 26) of the endoscope 20 is positioned close to the object to be observed. At this time, the distance between the imaging unit 26 and the object to be observed is approximately constant in the central region and peripheral region of the image. For this reason, it is preferable that the focus control in the detailed observation state be, for example, a closest object priority focus control.

[0176] Here, the near-point priority focus control is a focus control in which, for example, when the imaging unit 26 is brought close to an object located in the central area of ​​the screen of the endoscopic image, the focus lens position is fixed at the closest position at which a focused state can be ensured.

[0177] Specifically, this control moves the focus lens in the optical axis direction and maintains a predetermined position so that a focused state can be obtained on a predetermined object within a close distance range, for example, within approximately 3 millimeters in front of the lens tip surface. In this case, the imaging optical system (focus lens) position set during close-distance priority focus control is referred to as the second focus position. Note that close-distance (observation) priority focus control performs control with the focus lens fixed on the close-distance side of focus control unless the distance distribution pattern changes.

[0178] Normally, the closer the imaging distance, the shallower the depth of field. In this case, if the imaging unit sways back and forth relative to the object, for example, and focus control is performed to control the movement of the focus lens each time, a stably focused image cannot be obtained.

[0179] Therefore, as described above, in the detailed observation state, the focus lens position is fixed by performing the closest object priority focus control, thereby ensuring a focused state within a predetermined imaging distance range and enabling a stable observation image to be obtained.

[0180] Another example of a method for providing close-range priority focus control is focus control that performs precise focus adjustment within a close-range range (for example, within an imaging distance of 3 millimeters). In this case, the focus control is, for example, focus control that performs extremely fine and rapid movement control of the focus lens to ensure accurate focusing at all times.

[0181] 3, if the third distance distribution pattern is detected and it is determined that the detailed observation state is in effect, the process proceeds to step S10. In step S10, the processor 10 executes near-field priority focus control with the focus control unit 16 fixed at the second focus position (focus control switching step), and then proceeds to step S11.

[0182] In step S11, the processor 10 confirms an instruction to terminate the ongoing endoscopic examination. Here, the instruction to terminate the endoscopic examination may be, for example, a predetermined examination termination instruction signal generated by operating a predetermined operating member among the processor 10 of the endoscopic system 1 or the operating members 22c provided on the operating unit 22 of the endoscope 20. In addition, the termination of the endoscopic examination can also be determined, for example, by checking image data of the endoscopic image acquired by the endoscope 20.

[0183] If the instruction to end the examination is confirmed in the process of step S11, the series of processes is ended (END). If the instruction to end the examination is not confirmed, the process returns to the process of step S1.

[0184] As described above, the focus control device of the above embodiment includes a distance distribution information acquisition unit 15 that acquires distance distribution information (depth map) in the forward and backward directions of the object based on an output signal (e.g., phase difference information) from the imaging unit 26, a pattern classification unit 18 that classifies the distance distribution information (depth map) acquired by the distance distribution information acquisition unit 15 into a plurality of distance distribution patterns, and a focus position control unit (focus control unit 16) that adjusts the position of the focus lens on the optical axis in accordance with the classification result by the pattern classification unit 18.

[0185] The focus position control unit (focus control unit 16) performs pan focus control when the distance distribution pattern is the first distance distribution pattern (screening state) (step S12 in FIG. 3), performs focus control taking into account the object and its surroundings when the distance distribution pattern is the second distance distribution pattern (approach state) (steps S6 to S8 in FIG. 3), and performs near-point priority focus control when the distance distribution pattern is the third distance distribution pattern (detailed observation state) (step S9 in FIG. 3).

[0186] With this configuration, this embodiment can determine the state of use of the endoscope based on the distance distribution of the endoscopic image and switch focus control in accordance with the result of this determination, thereby enabling clear endoscopic images to be acquired at all times, thereby contributing to supporting image diagnosis during endoscopic examinations.

[0187] Furthermore, the system is provided with an illumination condition determination unit 19 that determines the illumination condition within the imaging screen based on the output signal from the imaging unit 26, and when the distance distribution pattern is the second distance distribution pattern (approach condition), the focus position control unit (focus control unit 16) performs focus control on the image area determined by the illumination condition determination unit 19 to be in the first illumination condition (appropriate exposure area) (step S7 in Figure 3).

[0188] This allows for determining not only the distance distribution (usage state) but also the exposure distribution (illumination state), and for focus control to be performed by focusing only on the area of ​​the first illumination state, which is the appropriate (good) exposure state. This makes it possible to limit the area that is the target of focus adjustment processing by focus control, thereby contributing to reducing the processing load and increasing the processing speed.

[0189] In the above-described embodiment, a distance distribution pattern is detected from a depth map (distance distribution information) of an endoscopic image, the usage state of the endoscope is determined based on the detected distance distribution pattern, and processing is performed to switch to an appropriate focus control depending on the determined usage state.

[0190] In the processing flow of a modified example of this embodiment, which will be described next, a distance distribution pattern is determined based on the depth maps (distance distribution information) of the central region, long distance region, and short distance region of the endoscopic image, and focus control is switched depending on the determination result.

[0191] 15 is a flowchart illustrating a modified example of the operation of an endoscope system including a focus control device according to an embodiment of the present invention. Similar to FIG. 3, the flowchart shown in FIG. 15 is a diagram illustrating the operation when an endoscopic examination is performed using the endoscope system 1.

[0192] The state of the endoscope system 1 when the flowchart of FIG. 15 starts is the same as the state when the processing sequence of FIG. 3 starts.

[0193] When the endoscopic system 1 is in a predetermined state, in step S21 of Figure 15, the image processing unit 11 of the processor 10 acquires an output signal (mainly image information) from the imaging unit 26 and starts predetermined image processing based on the output signal (image information).

[0194] In addition, the illumination condition determination unit 19 of the processor 10 starts to acquire exposure distribution information within the screen of the endoscopic image based on the endoscopic image data generated based on the image data processed by the image processing unit 11.

[0195] At the same time, the distance distribution information acquisition unit 15 of the processor 10 starts acquiring distance distribution information (depth map) based on the output signal (mainly phase difference information) from the imaging unit 26 .

[0196] Next, in step S22, the processor 10 performs processing to detect a distance distribution pattern based on the various information (image information, depth map, exposure distribution information, etc.) acquired in the processing of step S1 described above. Note that the processing of steps S21 and S22 is assumed to be continuously executed while the endoscopic examination is being performed. Then, the processing proceeds to step S23.

[0197] In step S23, the processor 10 checks whether the distance distribution pattern detected in the processing of step S2 described above is, for example, a distance distribution pattern (first distance distribution pattern) in which a long-distance distribution (dark area) exists in the central area and a short-distance distribution (bright area) exists in the peripheral area. Here, if the distance distribution pattern in the depth map is a long-distance distribution in the central area and a short-distance distribution in the peripheral area, it is determined that the depth map is in a screening state, for example, and the processing proceeds to step S33. On the other hand, if the distance distribution pattern is not a long-distance distribution in the central area and a short-distance distribution in the peripheral area, the processing proceeds to step S24.

[0198] If it is determined in step S23 that the state is the screening state and the process proceeds to step S33, the processor 10 executes pan focus control by the focus control unit 16 in step S33. Then, the process proceeds to step S34.

[0199] On the other hand, when the process proceeds from step S23 to step S24, the processor 10 checks whether the difference in distance in the front-to-back direction within the captured image is greater than a predetermined distance based on the depth map. If it is determined that the difference in distance in the front-to-back direction is greater than the predetermined distance, the process proceeds to step S25. If the difference in distance in the front-to-back direction is smaller than the predetermined distance, the process proceeds to step S30.

[0200] In step S25, the processor 10 detects the distance distribution of a predetermined region (region from the center to the near side) in the endoscopic image.

[0201] Next, in step S26, the processor 10 detects the distance distribution of another predetermined region (region on the far side from the center) in the same endoscopic image.

[0202] In step S27, the processor 10 checks, based on the depth map, whether there is a large change in distance between the long distance side and the short distance side and whether the exposure on the short distance side is good. Specifically, an endoscopic image such as that shown in FIG. 13 is assumed.

[0203] If it is determined that there is a large change in distance between the long distance side and the short distance side and that the exposure on the short distance side is good, the process proceeds to step S29. If it is determined that the conditions of a large change in distance between the long distance side and the short distance side and good exposure on the short distance side are not met, the process proceeds to step S33. In this case, for example, if the exposure on the short distance side is not good, it is determined that the user (doctor, etc.) does not want to observe the short distance side, and the process proceeds to step S33, where deep focus control is performed to ensure a deep depth of field.

[0204] In step S29, the processor 10 performs focus control by the focus control unit 16, taking into account the object and its peripheral area. Specifically, this focus control involves a focus adjustment operation or the like, in which a focus point is set at a predetermined position between the central area of ​​the endoscopic image (the area where the main part of the specific object is present) and the close-distance area (the well-exposed portion of the close-distance area). Then, the process proceeds to step S34.

[0205] On the other hand, if the process proceeds from step S24 to step S30, the difference in distance between the front and rear directions is smaller than the predetermined distance. In this case, if a deep depth of field is ensured, focus control can be performed to ensure a focused state within the range of the small difference in distance.

[0206] In this case, in step S30, the processor 10 checks whether the imaging distance is closer than a predetermined distance (for example, within 3 millimeters). If it is determined that the imaging distance is closer than the predetermined distance, the process proceeds to step S32. If it is determined that the imaging distance is not yet the predetermined distance (farther), the process proceeds to step S31.

[0207] In step S32, the processor 10 performs closest focus priority control using the focus control unit 16. After that, the process proceeds to step S34.

[0208] In step S31, the processor 10 performs normal focus control with the central region (main part of the specific object) as the focus point using the focus control unit 16. Thereafter, the process proceeds to step S34.

[0209] In step S34, the processor 10 checks for an instruction to end the ongoing endoscopic examination. If an instruction to end the examination is confirmed, the series of processes ends (END). If an instruction to end the examination is not confirmed, the process returns to the process of step S21 and repeats the subsequent processes.

[0210] As described above, in this modified example, the distance distribution pattern is determined based on the depth map and exposure distribution information of each of the central, long distance, and short distance regions in the depth map (distance distribution information) of the endoscopic image, and focus control is switched depending on the determination result.

[0211] This allows switching to appropriate focus control without determining the endoscope usage state from a depth map (distance distribution information) of the endoscopic image, thereby ensuring that clear and well-focused endoscopic images can always be obtained during endoscopic examinations.

[0212] The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the spirit and scope of the invention. For example, while an endoscope has been described as an example, the present invention can be applied to any device that sequentially acquires and evaluates images (such as an image inspection device). Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple components. For example, if the problem to be solved by the invention can be solved and the effects of the invention can be obtained even if some components are deleted from all the components shown in one embodiment, the configuration from which these components are deleted can be extracted as an invention. Furthermore, components from different embodiments may be appropriately combined. The present invention is not limited by specific embodiments other than as limited by the appended claims.

Claims

1. A focus control device comprising: a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; a pattern classification unit that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns; and a focus position control unit that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the classification results by the pattern classification unit.

2. The focus control device according to claim 1, wherein the distance distribution information acquisition section acquires the distance distribution information based on phase difference information or image information from the output signal from the imaging unit.

3. The focus control device according to claim 1, characterized in that the plurality of distance distribution patterns classified by the pattern classification unit are at least a first distance distribution pattern representing a screening operation for searching for a specific object, a second distance distribution pattern representing a process of approaching the specific object detected from the screening operation, and a third distance distribution pattern representing a detailed observation operation at a close range close to the specific object, and the focus position control unit performs pan focus control when the classification result is the first distance distribution pattern, performs focus control taking into account the object and its surroundings when the classification result is the second distance distribution pattern, and performs close-range priority focus control when the classification result is the third distance distribution pattern.

4. The focus control device according to claim 3, characterized in that, among the focus controls performed by the focus position control unit, the pan focus control is control for setting the position of the focus lens to a position where a focused state can be obtained within a preset distance range and maintaining the focus lens position, the focus control taking into account the object and its periphery is focus adjustment control for setting the focus point to a position between a main part of a specific object and a peripheral part of the specific object, and the close-range priority focus control is control for setting the position of the focus lens to a position where a focused state can be obtained for an object at a predetermined close distance from the imaging element and maintaining the focus lens position.

5. A focus control device as described in claim 4, further comprising an illumination condition determination unit that determines the illumination condition in the captured image based on the output signal from the imaging unit, wherein the illumination condition determination unit determines at least a first illumination condition that is a proper exposure condition, a second illumination condition that is an overexposed condition, and a third illumination condition that is an underexposed condition.

6. A focus control device as described in claim 5, characterized in that, when the classification result by the pattern classification unit is the second distance distribution pattern, the focus position control unit performs focus control on an image area determined to be in the first lighting state by the lighting state determination unit.

7. A focus control device as described in claim 5, characterized in that the focus control taking into account the object and its surroundings is further a focus adjustment control that sets a focus point at a position between the central area of the endoscopic image or a main part of the specific object and the image area determined to be in the first lighting condition.

8. The focus control device according to claim 5, wherein the illumination condition determining section determines the illumination condition in the captured image based on image information in the output signal from the imaging unit.

9. A focus control device as described in claim 3, further comprising a specific object image detection unit that detects an image area including the specific object from an image based on image data acquired by the imaging unit, and wherein the focus control performed by the focus position control unit is performed taking into account the main part of the detected specific object.

10. The focus control device described in claim 9, characterized in that the specific object image detection unit has a first inference model constructed using a large amount of specific object image data as training data and annotated by displaying a frame surrounding the image area containing the specific object.

11. The focus control device described in claim 1, characterized in that the pattern classification unit has a second inference model constructed using a large amount of specific object image data as training data, annotated with distance distribution information within an image area containing the specific object, and outputs the result of inferring the endoscope usage state in response to an input.

12. The focus control device according to claim 1, characterized in that the imaging unit is configured to include an imaging element, the imaging element having a plurality of pixels that receive light beams from a subject and output image signals, and some or all of these pixels are configured to be able to perform imaging plane phase difference detection, and image information and phase difference information can be obtained.

13. A focus control method comprising the steps of: acquiring distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; classifying the acquired distance distribution information into a plurality of distance distribution patterns; and adjusting the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit according to the classification results.

14. A focus control program that causes a computer to execute the following: a distance distribution information acquisition process that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; a pattern classification process that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns; and a focus position control process that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the classification results by the pattern classification unit.

15. An endoscopic system comprising: an endoscope including an imaging unit including an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system to obtain an image signal; a focus adjustment mechanism that moves some of the optical lenses included in the imaging optical system forward and backward in a direction along the optical axis; and a processor, wherein the processor comprises: a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of the object based on an output signal from the imaging unit provided at the tip of the endoscope insertion part; a pattern classification unit that classifies the distance distribution information acquired by the distance distribution information acquisition unit into a plurality of distance distribution patterns; and a focus position control unit that adjusts the relative position on the optical axis between a focus lens included in the imaging unit and the imaging element in accordance with the classification result by the pattern classification unit.

16. A focus control method for an endoscope, comprising: a distance distribution information acquisition step for acquiring distance distribution information of an object in front of an imaging unit provided at the tip of an endoscope insertion portion; a distance distribution pattern classification step for classifying the distance distribution information into a screening distance distribution pattern representing a screening operation for searching for a specific object, and an observation distance distribution pattern representing a detailed observation operation at a close range close to the specific object; and a focus control switching step for switching between pan focus control fixed at a first focus position or observation-priority focus control fixed at a second focus position according to the classification result.

17. A focus control method according to claim 16, wherein the observation-priority focus control is performed with the focus fixed to the close-up side unless the distance distribution pattern changes.

18. A focus control method as described in claim 16, characterized in that in the distance distribution pattern classification step, an approach distance distribution pattern other than the screening distance distribution pattern and the observation distance distribution pattern is determined, and if the approach distance distribution pattern is determined, in the focus control switching step, focus position control is performed to continuously adjust the focus to a position corresponding to a specific distance included in the acquired distance distribution information.

19. A method for creating an inference model for distance distribution of endoscopic images, characterized by comprising a learning step of learning using teacher data in which information indicating screening or detailed observation is annotated for multiple distance distribution information acquired by an imaging unit provided at the tip of the endoscope insertion part, in order to infer either a screening distance distribution pattern representing a screening operation to search for a specific object, or a detailed observation distance distribution pattern representing a detailed observation operation to observe the specific object at close range, from the distance distribution information of an object in front of the imaging unit provided at the tip of the endoscope insertion part.

20. A method for creating an inference model for an endoscopic distance distribution as described in claim 19, characterized in that the learning step further learns using training data annotated with information indicating the appropriate state of autofocus for the multiple distance distributions in order to infer the appropriate state of autofocus.

Images