Endoscope image generation device, endoscope image generation method, endoscope image generation program, and endoscope system

Abstract

The present invention comprises: a distance distribution information acquisition unit (15) for acquiring distance distribution information in the front-rear direction of an observation target, on the basis of an output signal from an imaging unit (26) provided at the tip of an endoscope (20) insertion unit (21), in order to generate an endoscope image that can be displayed in a size appropriate for the target and is always clear and in high-definition in an endoscope or the like; a focus position control unit (16) for adjusting the relative position of a focus lens, included in the imaging unit, on the optical axis according to the distance distribution information; and an electronic zoom processing unit (11a) for performing image processing for limiting the region of an image signal, acquired from the imaging unit, in order to generate an image to be displayed on the display unit.

Description

Endoscopic image generating device, endoscopy image generating method, endoscopy image generating program, and endoscopy system

[0001] The present invention relates to an endoscopic image generating device, an endoscopic image generating method, an endoscopic image generating program, and an endoscopic system that can generate clear images with a wider area in focus when capturing an image including an object to be observed.

[0002] Conventionally, endoscopic systems that include an endoscope that captures images of the inside of a subject and acquires image data, a processor that performs various types of image processing on the image data acquired by the endoscope, a display device that displays the image data processed by the processor as a visible image, and a storage device that records or stores the image data have been widely used in, for example, the medical field and the industrial field.

[0003] Furthermore, in the medical field, various examinations are widely performed using this type of endoscope system to observe the inside of biological organs, etc. In an examination using a medical endoscope system (hereinafter referred to as an endoscopic examination), for example, an imaging unit provided at the tip of an endoscope inserted into a lumen of a biological organ, etc., is used to capture images while the insertion section of the endoscope is advanced and retreated, such as by being withdrawn along the length of the lumen. At this time, the endoscopic images acquired by the imaging unit are displayed as moving images in real time chronologically on a display device. At the same time, the moving image data can also be stored in a storage device.

[0004] During such an endoscopic examination, the doctor or other equipment operator searches for lesions such as polyps or tumors occurring on the inner walls of organs as specific objects (hereinafter referred to as specific objects) while observing the real-time endoscopic image displayed on the display device (or, after the examination, a replayed endoscopic image based on recorded or stored endoscopic image data).

[0005] It is generally known that specific objects such as polyps and tumors that are the subject of observation in endoscopic examinations are often formed with irregularities on the inner wall surfaces of organs and the like.

[0006] This means that the distance from the imaging optical system in the imaging unit of the endoscope to each part of the object (hereinafter referred to as imaging distance) varies in the front-to-rear direction relative to the object.

[0007] It is also well known that the depth of field of an imaging optical system tends to become shallower as the imaging distance becomes shorter, and tends to become deeper as the imaging distance becomes longer.

[0008] Therefore, when an image is captured with the imaging optical system close to the object, the depth of field becomes shallow, and only a portion of the object is in focus. In this case, the areas before and after the in-focus area of ​​the object may become out of focus (so-called out-of-focus).

[0009] In this case, although detailed observation of a focused area of ​​the object can be performed using a clear image, there is a problem in that it is difficult to perform detailed observation of a wide area of ​​the same object.

[0010] Therefore, for example, Japanese Patent Publication No. 2006-235060 discloses an imaging device that has a focus determination means for determining the focus state on the imaging plane of an optical image, and when the determination result of this focus determination means is out of focus, moves the zoom lens position to the side where the depth of field becomes deeper and performs electronic zoom processing to electronically enlarge image data.

[0011] However, in the conventional imaging device disclosed in the above-mentioned Japanese Patent Publication No. 2006-235060, the zoom lens control and the electronic zoom processing are performed on the premise that the result of the determination by the focus determination means is that the image is out of focus.

[0012] Therefore, the conventional means disclosed in the above publications, etc., were not suitable for application to equipment used in endoscopic examinations, etc., which require continuous, uninterrupted, high-quality image display based on continuously acquired image data (so-called live view display or through-image display).

[0013] Meanwhile, in recent years, imaging elements (imaging sensors) included in imaging units used in endoscopes, etc. have become increasingly multi-pixel (higher resolution) and, for example, electronic zoom processing has been put into practical use, which includes image cropping (also called trimming) for cropping a partial area of ​​an image displayed based on image data acquired by the imaging element and image enlargement interpolation for enlarging the cropped image area.

[0014] Images displayed based on image data generated by such electronic zoom processing can display sufficiently high-resolution enlarged images without getting close to the subject, just like images captured using a high-magnification imaging optical system.

[0015] An object of the present invention is to provide an endoscopic image generating device, an endoscopic image generating method, an endoscopic image generating program, and an endoscopic system that can display an appropriate size according to the object to be imaged (observed) when performing detailed observation of a specific object such as a tumor in medical equipment such as an endoscope or inspection equipment (image inspection device, etc.), and that always generates clear, high-definition endoscopic images.

[0016] In order to achieve the above object, an endoscopic image generating device according to one aspect of the present invention includes a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion, a focus position control unit that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the distance distribution information, and an electronic zoom processing unit that performs image processing to limit the range of the image signal acquired from the imaging unit in order to generate an image to be displayed on a display unit.

[0017] A second endoscopic image generating device according to one aspect of the present invention includes a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions relative to a specific object to be observed based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion, a focus position control unit that adjusts the positions of a focus lens and a zoom lens on the optical axis in accordance with the distance distribution information, and an electronic zoom processing unit that performs image processing to limit the range of the image signal acquired from the imaging unit.

[0018] An endoscopic image generating method according to one aspect of the present invention acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion, adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the distance distribution information, and performs image processing to limit the range of the image signal acquired from the imaging unit in order to generate an image to be displayed on a display unit.

[0019] An endoscopic image generating program according to one aspect of the present invention causes a computer to execute the following: a distance distribution information acquisition process for acquiring distance distribution information in the forward / backward direction of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; a lens position control process for adjusting the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the distance distribution information; an image cropping process for cropping a partial area of ​​an image based on image data acquired by the imaging unit in order to generate an image to be displayed on a display unit; and an electronic zoom image processing for limiting the area of ​​image data acquired by the imaging unit; a specific object image detection process for detecting an image area including a specific object to be observed from an image based on the image data acquired by the imaging unit; an object unevenness range determination process for determining an unevenness range of the object to be observed based on the distance distribution information; a depth of field determination process for determining a depth of field according to the object unevenness range; and an imaging distance setting process for setting an imaging distance according to the depth of field.

[0020] An endoscopic system according to one aspect of the present invention comprises an endoscope including an imaging unit including an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system to obtain an image signal, an endoscope including a focus adjustment mechanism that moves some of the optical lenses included in the imaging optical system forward and backward in a direction along the optical axis, and a processor, wherein the processor comprises: a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of the object based on an output signal from the imaging unit; a focus position control unit that adjusts the relative position on the optical axis between a focus lens and the imaging element included in the imaging unit in accordance with the distance distribution information; and an electronic zoom processing unit that performs image processing to limit the range of the image signal acquired from the imaging unit in order to generate an image to be displayed on a display unit.

[0021] According to the present invention, it is possible to provide an endoscopic image generating device, an endoscopic image generating method, an endoscopic image generating program, and an endoscopic system that can display an appropriate size according to the object to be imaged (observed) when performing detailed observation of a specific object such as a tumor in medical equipment such as an endoscope or inspection equipment (image inspection device, etc.), and that always generate clear, high-definition endoscopic images.

[0022] a schematic diagram illustrating the overall configuration of an endoscopic system including an endoscopic image generating device of one embodiment of the present invention; a block diagram illustrating the general internal configuration of an endoscopic system including an endoscopic image generating device of one embodiment of the present invention; a schematic diagram conceptualizing the structure of an imaging element applied to an endoscope included in the endoscopic system of Figure 1 and explaining the principle of imaging surface phase difference AF by the imaging element; a graph conceptually showing the relationship between imaging distance and depth of field in the imaging optical system of an endoscope; a schematic diagram illustrating a first observation mode (during close-up observation) when observing a specific object in detail using an endoscope of the endoscopic system of Figure 1; an example of a display of an endoscopic image when the endoscope is in the state of Figure 5; a schematic diagram illustrating a second observation mode (during close-up observation in combination with electronic zoom) when observing a specific object in detail using an endoscope of the endoscopic system of Figure 1; an example of a display of an endoscopic image when the endoscope is in the state of Figure 7; a flowchart illustrating the operation of an endoscopic system including an endoscopic image generating device of one embodiment of the present invention; an explanatory diagram of first and second inference models obtained as a result of learning in a learning device; and a flowchart showing a modified example of the operation of an endoscopic system including an endoscopic image generating device of one embodiment of the present invention.

[0023] The present invention will be described below with reference to illustrated embodiments, for example, as an application example to an endoscope system. The drawings used in the following description are schematic. Therefore, in these drawings, each component is shown at a size that allows it to be recognized. Therefore, the dimensional relationships and scales of each component may be shown differently for each component. The present invention is not limited to the illustrated embodiments with respect to the quantities, shapes, size ratios, relative positional relationships, and the like of each component shown in the drawings.

[0024] First, before describing the detailed configuration of one embodiment of the present invention, the schematic configuration of the entire endoscope system including the endoscopic image generating device of this embodiment will be described below with reference to Fig. 1. Fig. 1 is a configuration diagram that schematically shows the overall configuration of an endoscope system including the endoscopic image generating device of one embodiment of the present invention.

[0025] 1, the endoscope system 1 is mainly configured to include a processor 10, an endoscope 20, a display device 30, a light source device 40, and a storage device 50. The endoscope system 1 shown in Fig. 1 illustrates a general configuration of an endoscope system used in an endoscopic examination in which the inside of an organ, such as the upper digestive tract (esophagus, stomach, duodenum, etc.) or the lower digestive tract (large intestine, etc.) of a subject (patient, etc.), such as a living body, is observed.

[0026] In an endoscopic examination using such an endoscopic system, a doctor or other equipment operator performs so-called screening, searching for specific lesions such as polyps or tumors occurring on the inner walls of organs while observing the endoscopic image displayed on the display device.

[0027] However, in normal cases, specific objects such as lesions are very small and have similar colors to the inner walls of surrounding organs, etc., so it takes skill to accurately distinguish and detect the image area of ​​the specific object from an endoscopic image. In addition, in endoscopic examinations, the operation of the endoscope itself, such as the operation of pointing the tip of the insertion part of the endoscope (the observation window of the imaging unit) at the desired object and capturing an image, requires skill.

[0028] Generally, the use states of an endoscope during an endoscopic examination can be broadly divided into, for example, a state in which the above-mentioned screening is being performed (referred to as the screening state), a state in which a detected specific object is being observed in detail (referred to as the detailed observation state), and a state in which the tip of the endoscope is in the process of transitioning from the screening state to the specific object to be observed in order to perform detailed observation (referred to as the approach state).

[0029] The state of the endoscope can be determined by analyzing the speed at which the distal end of the endoscope is moved, the characteristics of the object, or changes in the object over time. For example, in the screening state, a portion of the endoscopic image will characteristically change from moment to moment. For example, an endoscopic image acquired when the distal end of the endoscope is moved along a lumen inside an organ will exhibit a flowing image change from the peripheral region of the screen to the central region of the screen, or from the central region of the screen to the peripheral region, like the tunnel wall seen at the periphery of the field of view when a vehicle traveling through a tunnel looks in the direction of travel.

[0030] Therefore, the screening state can be likened to a situation where a moving vehicle is searching for abnormalities on the wall of a tunnel. In other words, the presence or absence of abnormalities (such as polyps or lesions such as tumors) is determined by sequentially moving the observation position while determining the image characteristics of the surrounding area on the endoscopic image display screen.

[0031] Furthermore, for example, in the approach state, the distal end of the endoscope moves toward a specific object. In this case, the endoscopic image displayed in the central area of ​​the screen does not exhibit any distinctive changes, and the display of the object expands as the object approaches. Therefore, in the approach state, the state of the endoscope can be determined based on the image of the object displayed on the screen. In this case (approach state), the state of the endoscope may be determined based on, for example, the operating status of the endoscope operator, such as the output signals of various operating switches.

[0032] Conventionally, when performing endoscopic examinations using medical or inspection equipment such as endoscopes, it has been necessary to perform appropriate focus control of the imaging unit according to the usage conditions of the endoscope, etc., in order to always ensure a reliable observation environment and continue to display good images.

[0033] For example, when an endoscope or the like is in a screening state, a so-called deep focus control setting is often used, in which the position of the imaging optical system of the imaging unit is set so as to obtain a good focus state within a predetermined imaging range, without performing precise focus control, depending on the characteristics of the imaging optical system.

[0034] Furthermore, for example, when an endoscope or the like is in a detailed observation state or an approach state, it is required to appropriately perform focus control to bring the object to be imaged or observed into focus so that the object can be observed in detail with a clear image.

[0035] Therefore, for example, when performing detailed observation, in order to display the object of observation as large as possible, the tip surface of the imaging unit (the front surface of the imaging optical system) is often brought close to the surface of the object of observation (a specific object such as a tumor) to capture the image.

[0036] As shown in FIG. 1, an endoscope 20 is an image inspection device that includes an insertion section 21, an operation section 22, a universal cable 23, and the like.

[0037] The insertion section 21 is a component that is inserted into a subject such as a living organism. The insertion section 21 is formed by continuously connecting a distal section 21a, a bending section 21b, and a flexible tubular section 21c, in that order from the distal end. The insertion section 21 is formed in a generally elongated tubular shape. The insertion section 21 has a treatment tool insertion channel 21d therein, which is a conduit for inserting an endoscopic treatment tool (not shown). This treatment tool insertion channel 21d is provided so as to pass through the insertion section 21 from the distal end to the proximal end. The operation section 22 is connected to the proximal end side of the insertion section 21.

[0038] The tip portion 21a is a structural unit provided at the most distal end of the insertion portion 21. Various structural members (not shown in FIG. 1; see FIG. 2) such as an illumination unit 25 and an imaging unit 26 are provided inside the tip portion 21a.

[0039] Here, the illumination unit 25 is a structural unit that includes an optical element (illumination lens; not shown) that emits a light beam guided from the light source device 40 (described later) forward from the tip surface of the tip portion 21 a to illuminate an observation area including a lesion or the like within the subject.

[0040] The imaging unit 26 is an electronic device unit including an optical lens (imaging optical system 26a; see FIG. 2) that forms an optical image of a specific object (a lesion such as a tumor) to be observed (imaged) inside the subject, and a photoelectric conversion element (imaging element 26b; see FIG. 2) that generates image information (still image data, moving image data, etc.) based on the optical image. Although not shown, the imaging unit 26 also includes a drive mechanism (focus adjustment mechanism, magnification change mechanism, etc.) that moves some of the optical lenses (e.g., a focus lens, a zoom lens, etc.) included in the imaging optical system 26a forward and backward along the optical axis.

[0041] The bending portion 21b is a tubular portion that can be actively bent by a bending operation mechanism (not shown) that acts in conjunction with the operation of a bending operation member 22b (described later). The flexible tube portion 21c is a flexible tubular member that extends from the tip of the operation portion 22 and is connected to the base end of the bending portion 21b.

[0042] The operation section 22 is connected to the proximal end of the insertion section 21. The operation section 22 is configured to include an operation section main body 22a, a bending operation member 22b, a plurality of operation members 22c, a treatment tool insertion port 22d, and the like.

[0043] The operation unit main body 22a has a generally box-like shape as a whole and constitutes a gripping portion that is held with the fingers of a user (such as a doctor) of the endoscope 20. Various constituent units such as a bending operation mechanism (not shown) are provided inside the operation unit main body 22a. As described above, the insertion section 21 extends from the operation unit main body 22a.

[0044] The bending operation member 22b and the plurality of operation members 22c are operation members for performing various operations of the endoscope 20. The bending operation member 22b and the plurality of operation members 22c are respectively provided at predetermined positions on the outer surface of the operation portion main body 22a.

[0045] The treatment instrument insertion port 22d is provided at a predetermined position near the distal end of the operation portion main body 22a. The treatment instrument insertion port 22d is a proximal end opening of the treatment instrument insertion channel 21d of the insertion portion 21. The treatment instrument insertion channel 21d is connected at its distal end to a distal channel opening (not shown) of the distal end portion 21a.

[0046] With this configuration, an endoscopic treatment tool (not shown) inserted through the treatment tool insertion port 22d is configured to pass through the treatment tool insertion channel 21d and then protrude outward from the tip channel opening of the tip portion 21a.

[0047] The universal cable 23 is a connection cord for connecting the endoscope 20 to the light source device 40 and the processor 10. For this purpose, the universal cable 23 is made of a tubular member extending from the side of the operation unit body 22a of the operation unit 22. A scope connector 23a is provided at the tip of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

[0048] An electric cable 23b extends from the scope connector 23a. A connector 23c is provided at the tip of the electric cable 23b. The connector 23c is connected to the front panel of the processor 10. Various signal transmission cables, optical fiber cables, etc. (not shown) are inserted into the universal cable 23.

[0049] The light source device 40 is a device that supplies illumination light to the illumination unit 25 (see FIG. 2) provided inside the tip portion 21a of the insertion section 21 of the endoscope 20. The illumination light emitted from the light source device 40 is transmitted to the illumination unit 25 in the tip portion 21a through an optical fiber cable (not shown) or the like that is arranged passing through the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21. The illumination light then passes through an illumination lens (not shown) included in the illumination unit 25 in the tip portion 21a and is irradiated toward the observation target area in front of the tip portion 21a.

[0050] The processor 10 is a control device, a signal processing device, or a circuit unit that includes a control circuit and a signal processing circuit that control the entire endoscope system 1. The processor 10 is also configured to include the functions of the endoscopic image generating device of this embodiment (described in detail below; see FIG. 2).

[0051] A control circuit included in the processor 10 receives, for example, an operation instruction signal from the operation member 22c of the operation section 22 of the endoscope 20, and outputs various control signals for driving and controlling the imaging unit 26, the light source device 40, the illumination unit 25, etc. Furthermore, a signal processing circuit included in the processor 10 receives, for example, an imaging signal from the imaging unit 26, and performs predetermined image signal processing, etc.

[0052] For this purpose, the processor 10 and the imaging unit 26 are electrically connected by a signal transmission cable (not shown). The signal transmission cable is inserted and arranged from the connector 23c through the electrical cable 23b, the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21 to the imaging unit at the tip end 21a.

[0053] With this configuration, control signals output from the processor 10 and imaging signals output from the imaging unit 26 are transmitted between the imaging unit 26 and the processor 10 through the signal transmission cable. Note that one form of the signal transmission cable is, for example, a composite cable in which a plurality of cables are bundled together and covered with an outer sheath shield, an outer sheath tube, or the like.

[0054] The display device 30 receives image signals and the like output from the processor 10 and displays endoscopic images and various types of information in a predetermined format. To this end, the display device 30 and the processor 10 are electrically connected using a video cable 24. The display device 30 may be configured as a display device using, for example, a general liquid crystal panel.

[0055] The storage device 50 is a storage device that saves (records or stores) image data for storage that is generated for storage after various processes are performed in the processor 10 based on image information generated by the imaging unit 26 .

[0056] As shown in Fig. 1, the storage device 50 is configured as an integral part of the processor 10. As shown in Fig. 2, a temporary storage unit 51 is provided inside the storage device 50 in addition to a main storage unit (not shown) in the storage device 50. The temporary storage unit 51 is configured, for example, by a semiconductor memory or the like, and is a component that functions as a temporary memory area for temporarily storing predetermined image data output from the image processing unit 11, etc.

[0057] 1 illustrates a configuration in which the storage device 50 is integrally disposed inside the housing of the processor 10, but the present invention is not limited to this. For example, the storage device 50 may be configured as an external storage device configured using a housing separate from the processor 10.

[0058] 2 shows an example of a configuration in which the temporary storage unit 51 is provided inside the storage device 50, but this is not limiting. The temporary storage unit 51 can also be provided integrally inside the storage control unit 13 (see FIG. 2), which will be described later, for example.

[0059] Furthermore, the processor 10 and the light source device 40 are not limited to being configured as separate entities, as in the example configuration shown in Fig. 1. For example, the processor 10 and the light source device 40 may be configured as an integrated unit using a single housing.

[0060] The configuration of the illumination unit 25 is not limited to the above-described configuration (a configuration in which illumination light from the light source device 40 is transmitted to the distal end portion 21a through an optical fiber cable or the like). As a configuration of the illumination unit 25 other than this, for example, a light-emitting element such as an LED (Light Emitting Diode) as an illumination light source can be provided inside the distal end portion 21a, and power supply to the illumination light source (LED) and light emission control thereof can be controlled by a predetermined control circuit included in the processor 10.

[0061] The endoscope system 1 configured as described above basically has substantially the same configuration as conventional endoscope systems of the same type, and therefore, illustrations and detailed descriptions of other detailed configurations will be omitted.

[0062] In the endoscopic system 1 configured as described above, the endoscopic image generating device of this embodiment is configured to be included in the processor 10. Next, the internal configuration of the processor in the endoscopic system including the endoscopic image generating device of one embodiment of the present invention will be described in detail below with reference to FIG.

[0063] The processor 10 is configured to include an image processing unit 11, a display control unit 12, a memory control unit 13, a specific object image detection unit 14, a distance distribution information acquisition determination unit 15, an imaging distance setting unit 18, a focus position control unit 16, an illumination control unit 17, and a memory device 50 including a temporary memory unit 51.

[0064] The image processing unit 11 is a structural unit or circuit unit that receives image information output from the imaging unit 26 and performs various types of information processing based on the image information. The image processing performed here includes, for example, normal image information processing, image data processing for display, image data processing for storage, and various other types of image processing.

[0065] The normal image processing includes an AD conversion process that converts the analog image signal output from the image sensor 26b into a digital image signal, and is a basic image information processing that is normally performed on image information. The normal image processing is assumed to be a conventional process, and a detailed description thereof will be omitted.

[0066] The display image data processing is processing for generating image data for normal display, such as processing for generating image data representing an endoscopic image to be displayed in a predetermined display area on the display screen of the display device 30. The storage image data processing is processing for generating image data for recording.

[0067] The various types of image processing are image processing that are appropriately added to normal image data for display. Specific examples of the various types of image processing include normal image adjustment processing such as brightness adjustment processing and white balance adjustment processing, and image enhancement processing such as edge enhancement processing and texture and color enhancement imaging (TXI).

[0068] Furthermore, the image processing unit 11 in the processor 10 including the endoscopic image generating device of this embodiment is configured to include an electronic zoom processing unit 11a. This electronic zoom processing unit 11a performs electronic zoom processing including image cropping processing and image enlargement interpolation processing.

[0069] Here, the image cutting process is an image process (so-called image cutting process or trimming process) that cuts out a desired area (e.g., an image area including a specific object) from an image based on image data acquired by the imaging element 26b of the imaging unit 26.

[0070] Furthermore, the image enlargement interpolation process is an image process in which an image of a predetermined partial area cut out by the image cutout process is enlarged and interpolated to a predetermined image size at a predetermined enlargement magnification. In this case, the predetermined image size is preferably set to, for example, a magnification magnification that allows the specific object being observed to be displayed to fill the display screen. However, in order to generate a diagnostic image, it is desirable to always display a clear image. In consideration of this, the magnification magnification setting for the image enlargement interpolation process may be limited to avoid extreme magnifications that degrade the displayed image (e.g., magnifications that make the pixels visible).

[0071] The image processing in the electronic zoom processing section 11 a is mainly performed on an image area including a specific object detected by the specific object image detection section 14 from the image data acquired by the imaging unit 26 , for example.

[0072] The display control unit 12 is a structural unit or circuit unit that controls the display device 30 so that an appropriate display format is displayed according to the display image data output from the image processing unit 11. The display control unit 12 controls the display of the display image data (various settings such as display position, display area, display size, etc.) as well as the display of various information other than image information (text information, icon information, etc.). Here, the various information other than image information includes examination date and time information, subject (patient) information, medical record information, etc., as well as various alarm information (notification information) that is displayed as appropriate during an endoscopic examination, for example.

[0073] Specifically, the display control unit 12 performs display control to continuously display the endoscopic image in the display area, and when a specific object is detected in the endoscopic image, the display control unit 12 performs display control to add a predetermined display (for example, a frame display) to the image area including the specific object.

[0074] Furthermore, the display control unit 12 includes a guide display unit 12a. The guide display unit 12a is a structural unit or circuit unit that displays various alarm information (notification information), guidance information, and the like at appropriate predetermined timing to a device operator such as a doctor during an endoscopic examination.

[0075] The memory control unit 13 is a structural unit or circuit unit that controls the memory device 50 (including the temporary memory unit 51) so that the image data for storage output from the image processing unit 11 is saved (recorded or stored) in an appropriate storage format.

[0076] The storage control unit 13 performs storage control to permanently store image data representing an endoscopic image acquired by the endoscope 20 in a storage area of ​​the storage device 50 .

[0077] The specific object image detection unit 14 is a structural unit or circuit unit that detects an image area including a specific object from an endoscopic image based on image data acquired by the endoscope 20 .

[0078] In detail, the specific object image detection unit 14 detects a specific object (such as a lesion such as a tumor) within each frame of endoscopic image that is displayed continuously based on the image data output from the image processing unit 11.

[0079] The specific object image detection unit 14 detects specific objects, for example, by performing similar image recognition (pattern matching processing, etc.) based on multiple specific object images (case images, etc.) prepared in advance, or by performing data analysis using deep learning or machine learning to detect the desired specific object.

[0080] For example, in recent years, object detection technology using AI (Artificial Intelligence) has been developed that obtains desired inference results (e.g., detection of specific objects such as tumors) through machine learning of endoscopic images acquired by an endoscope.

[0081] Machine learning is a method of obtaining inference results about unknown matters by learning the features, time-series information, spatial information, etc. of known input information and making inferences based on the learning results. That is, in machine learning, a trained model (also called an inference model) is first obtained that enables inference of a determinable output result from specific input information.

[0082] In this case, when generating a trained model, a large amount of information with known input-output relationships is used as training data to obtain highly reliable inference results. For example, in deep learning, a network is designed using a large amount of training data so that expected outputs can be obtained for known inputs. The trained model obtained through this process can be used independently of the network that was used for training.

[0083] Therefore, by providing the processor 10 with this type of inference model, it becomes possible to accurately and quickly perform various judgments or detections, such as the detection of specific objects in the specific object image detection unit 14, as well as the judgment of the imaging distance range in which images can be captured in a focused state during pan-focus control, the judgment of the usage status of the endoscope during endoscopic examination, and the detection of the surface vascular pattern of specific objects.

[0084] Here, the inference model is constructed roughly as follows. For example, a large amount of image data corresponding to input and output is provided as training data to a predetermined network (not shown) for constructing the inference model. Here, the training data includes image data suitable for detection or determination, such as image data of endoscopic images acquired sequentially in time series during an endoscopic examination, image data of images (such as case images) in which a specific object is captured in the endoscopic images, etc. In this case, an annotation may be set to surround the image area of ​​the specific object with a frame or the like.

[0085] By performing learning using a large amount of training data, a network design is determined for a specific network (not shown) so that an output corresponding to an input can be obtained. For example, Fig. 10 is an explanatory diagram of an inference model obtained as a result of learning in a specific learning device.

[0086] 10, when a group of endoscopic images (first training data group 201) containing a specific object (such as a tumor) is input, information about the specific object captured in the image and additional information such as adding a frame display surrounding the image area of ​​the specific object are obtained along with reliability information. This allows the construction of a first inference model 200 that detects the specific object from the endoscopic image.

[0087] When endoscopic image data 202 obtained during an endoscopic examination is input to the first inference model 200 constructed in this manner, the first inference model 200 detects an image area containing the specific object in the frame in which the specific object is captured from the endoscopic image data 202, and outputs image data (inference result image 203) in which a frame display is added to the image area of ​​the specific object. In this manner, the specific object can be detected from the endoscopic image. The image data of this image 203 can then be temporarily stored, or, if necessary, displayed in a specified display area of ​​the display device 30, providing a reference for the endoscope user during the endoscopic examination.

[0088] Furthermore, when a large number of images (second training data 204) annotated with characteristic information such as the type and size of the specific object captured by a doctor or other doctor for diagnostic purposes during a past endoscopic examination or the like are input, additional information such as an appropriate display size (target display size) according to the tumor type of the specific object can be obtained along with reliability information. This allows the construction of a second inference model 210 for detecting specific objects from endoscopic images.

[0089] When endoscopic image data 202 taken during an endoscopic examination is input to the second inference model 210 constructed in this manner, the second inference model 210 detects the tumor type of the specific object from the endoscopic image data 202 and outputs target size information corresponding to the detected tumor type. In this way, display target size information of the specific object can be obtained from the endoscopic image.

[0090] "Deep learning" is a multilayered version of the "machine learning" process using neural networks. A typical example is a "forward propagation neural network," which sends information from front to back and makes a judgment. In its simplest form, it requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons determined by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights, and a bias value is added between the hidden and output layers, making it easy to form logic gates. While three layers are sufficient for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, models with 9 to 152 layers have become practical due to their training time, judgment accuracy, and energy consumption.

[0091] Various well-known networks may be used for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) using CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses image features, operates with minimal processing, and is strong in pattern recognition. Furthermore, "recurrent neural networks" (fully connected recurrent neural networks) that can handle more complex information and allow information analysis whose meaning changes depending on the order or sequence of information may be used.

[0092] To realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, devices specialized for matrix calculations such as GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) may also be used. In recent years, such dedicated artificial intelligence (AI) hardware "neural network processing units (NPUs)" have been designed to be integrated and embeddable with CPUs and other circuits, and may even become part of the processing circuit.

[0093] Furthermore, inference models may be obtained by employing various well-known machine learning techniques, not limited to deep learning. For example, techniques such as support vector machines and support vector regression are available. Here, learning involves calculating the weights, filter coefficients, and offsets of a classifier; other techniques include using logistic regression processing. When a machine is to make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method for deriving an image judgment using machine learning is employed. However, a rule-based method for applying rules acquired by humans through experience or heuristics to make a specific judgment may also be used.

[0094] The distance distribution information acquisition determination unit 15 is a structural unit or circuit unit that acquires distance distribution information (hereinafter simply referred to as distance distribution information or depth data) within the imaging screen based on the output signal (image information or phase difference information; described later) from the imaging element 26b.

[0095] Here, the distance distribution information within the imaging screen refers to information that represents the distribution of distances in the front-to-rear direction of an object, including a specific object to be observed, that is reflected in the imaging screen. The front-to-rear direction of an object refers to the front-to-rear direction (in other words, the depth direction) when the object, including the specific object, is viewed from the imaging unit 26 (image sensor 26 b) provided at the tip 21 a of the endoscope 20, and refers to the direction of advancement and retreat of the endoscope 20, and the direction along the insertion axis of the endoscope 20.

[0096] As described above, the image sensor 26b included in the imaging unit 26 of the endoscope 20 used in this embodiment has a plurality of pixels that receive light beams from a subject and output image signals, and some of these pixels also function as focus detection pixels, so that the image sensor 26b is configured to perform so-called image-surface phase difference detection and contribute to autofocus control (AF control). In this embodiment, the use of the image sensor 26b of this type makes it possible to acquire image information and phase difference information.

[0097] Here, a configuration that enables image-surface phase difference detection using an image sensor will be briefly described below. Fig. 3 is a schematic diagram illustrating the principle of image-surface phase difference AF control that performs image-surface phase difference detection using an image sensor.

[0098] As described above, the image sensor has multiple pairs of focus detection pixels, each of which is a pair of focus detection pixels 260L and 260R. As shown in FIG. 3 , each of the focus detection pixels 260L and 260R has a microlens 261, a pixel 262, and a light blocking member 263.

[0099] The microlens 261 is an optical lens that transmits a light beam that has passed through an imaging optical system (not shown in FIG. 3 ) and forms an image on the imaging surface of the pixel 262. The pixel 262 is, for example, a photodiode, and is a photoelectric conversion element that converts the optical image formed by the microlens 261 into an electrical signal and outputs it.

[0100] The light blocking member 263 is a member that blocks light from the right region or the left region with respect to the optical axis O of the microlens 261. For example, in the focus detection pixel 260L, the light blocking member 263 is disposed in a position that blocks light from the right region. As a result, in the focus detection pixel 260L, only the light beam LL that is incident from the left region of the microlens 261 is imaged on the imaging surface of the pixel 262.

[0101] Furthermore, for example, in the focus detection pixel 260R, the light blocking member 263 is disposed in a position that blocks light from the left region. As a result, in the focus detection pixel 260R, only the light beam LR that is incident from the right region of the microlens 261 is imaged on the imaging surface of the pixel 262.

[0102] By calculating the distance between the two images (parallax) from the output signals (phase difference information) of each pixel 262 of the pair of focus detection pixels 260L, 260R configured in this manner, the defocus amount for the object in the imaging optical system can be determined.

[0103] The distance distribution information acquisition determination unit 15 receives the phase difference information acquired by the image sensor 26 b in this manner and acquires distance distribution information (depth data) of a predetermined region within the imaging surface (e.g., a region including a specific object). In this case, the distance distribution information acquisition determination unit 15 functions as a distance distribution information acquisition unit.

[0104] Furthermore, the distance distribution information acquisition determination unit 15 determines the unevenness range of the specific object based on the acquired distance distribution information (depth data), and determines the required depth of field according to the unevenness range. In this case, the distance distribution information acquisition determination unit 15 functions as an object unevenness range determination unit and also functions as a depth of field determination unit.

[0105] Here, the unevenness range of a specific object refers to the range of distance difference in the front-to-back direction (depth direction) relative to the imaging unit 26, for example, when the specific object is formed in a shape that protrudes from the inner wall surface of an organ, etc., or when the surface of the specific object is formed in an uneven shape.

[0106] When functioning as an object unevenness range determiner, the distance distribution information acquisition and determination unit 15 determines the distance difference between the unevenness of the object in the front-to-rear direction relative to the imaging unit. When functioning as a depth-of-field determiner, the distance distribution information acquisition and determination unit 15 determines the depth of field that can include the determined distance difference between the unevenness of the object in the focus range.

[0107] 3, the pupil of the microlens is divided using a light-shielding member, but the present invention is not limited to this configuration. For example, a similar configuration can be realized by dividing the pixel 262 into two.

[0108] In the above-described configuration example, the distance distribution information acquisition determination unit 15 acquires the distance distribution information from the phase difference information from the image sensor 26 b having focus detection pixels, but the present invention is not limited to this configuration. For example, in a typical image sensor, all pixels provided on the imaging surface acquire image signals.

[0109] In an imaging unit having such a conventional imaging element 26b, for example, an optical lens (e.g., a focus lens), which is one of the lenses included in the imaging optical system 26a and contributes to focus adjustment, is slightly moved back and forth along the optical axis to change the lens position. This causes a change in the contrast of objects in the image, thereby changing the focus state. At this time, distance information can be obtained from the correlation between the lens position and the focus state corresponding to the lens position. Based on the distance information thus obtained, distance distribution information within the screen can be obtained. In other words, a configuration can be adopted in which distance distribution information is obtained based on image signals from the imaging element 26b.

[0110] The imaging distance setting unit 18 is a structural unit or circuit unit for setting an imaging distance that can obtain the depth of field based on the depth of field corresponding to the unevenness range determined by the distance distribution information acquisition determination unit 15.

[0111] The focus position control unit 16 is a structural unit or circuit unit that drives and controls the imaging optical system 26a of the imaging unit 26. The focus position control unit 16 controls the driving of the imaging optical system 26a based on the detection results of the specific object image detection unit 14, the distance distribution information acquisition and determination unit 15, and the imaging distance setting unit 18, and adjusts the relative position on the optical axis between the focus lens and the imaging element. This allows for, for example, performing a predetermined automatic focus adjustment operation (hereinafter referred to as autofocus control or AF control) or focus control that takes the depth of field into consideration (so-called pan focus control), etc.

[0112] For example, observation in the screening state is performed with a slight distance (e.g., a few centimeters) between the front of the imaging unit and the object being observed (imaging distance). In this case, sufficient observation can be performed within the depth of field of the imaging optical system. Therefore, the focus control in this case is usually set to the so-called pan-focus control setting.

[0113] In this case, the pan focus control setting is, for example, a control setting that sets the position of the imaging optical system so that an object within a predetermined distance range (for example, within a range of approximately 1 to 3 centimeters in front of the tip surface of the lens) is roughly in focus, and maintains that position.

[0114] On the other hand, detailed observation is generally performed at a close imaging distance (for example, within approximately 3 millimeters of the object).

[0115] Generally, the depth of field of an imaging optical system exhibits the following tendency with respect to the imaging distance. For example, as shown in FIG. 4 , if the depth of field at an imaging distance Ln is represented by DOFn, in this case, the depth of field DOFm at an imaging distance Lm closer than the imaging distance Ln is shallower than the depth of field DOFn at the imaging distance Ln. Similarly, it is known that the depth of field DOFo at an imaging distance Lo farther than the imaging distance Ln is deeper than the depth of field DOFn at the imaging distance Ln. In this way, the depth of field of an imaging optical system tends to become shallower as the imaging distance becomes shorter and deeper as the imaging distance becomes longer.

[0116] Therefore, for example, when focus adjustment is performed on an arbitrary point on a specific object to be observed (the most notable point, such as the top of the specific object or an area near the center of the screen), the area before and after the focus point can only be focused within an extremely narrow range.

[0117] For example, the situation at this time is schematically shown in Fig. 5. The situation shown in Fig. 5 is a situation when a specific object such as a tumor is observed in detail using the endoscope 20 of the endoscope system 1 of this embodiment, and in particular, a situation during close observation in which the imaging distance is set close to the specific object. In the following description, the situation shown in Fig. 5 will be referred to as the first observation mode during close observation.

[0118] As shown in Fig. 5, the distal end surface of the distal end portion 21a of the insertion section 21 of the endoscope 20 is disposed facing a specific object 301 detected on the inner wall of an organ or the like 300. Here, reference numeral 301a in Fig. 5 indicates the top of the specific object 301. Reference numeral 301b in Fig. 5 indicates the base of the specific object 301. At this time, it is assumed that the imaging optical system 26a of the imaging unit 26 is set to focus on the top 301a of the specific object 301 by AF control.

[0119] 5 indicates a first imaging distance from the tip surface of the tip portion 21a (the front surface of the imaging optical system 26a of the imaging unit 26) to the top 301a of the specific object 301. At this time, it is assumed that the top 301a is located at a position substantially on an extension of the optical axis of the imaging optical system 26a.

[0120] 5 indicates a second imaging distance from the tip surface of the tip portion 21a (the front surface of the imaging optical system 26a of the imaging unit 26) to the base portion 301b of the specific object 301. In this case, the second imaging distance L2 is the distance on an extension of the optical axis of the imaging optical system 26a.

[0121] The symbol An1 in Fig. 5 indicates the angle of view of the imaging optical system 26a. The symbol DOF1 in Fig. 5 indicates the depth of field corresponding to the currently set imaging distance. In the example of Fig. 5, the imaging distance is the first imaging distance L1 where the focus point is located. The symbol [A1] (thick line portion) in Fig. 5 indicates the area where the surface of the specific object 301 is in focus under the circumstances (the area included in the depth of field DOF1).

[0122] 5, the first imaging distance L1 is, for example, within approximately 3 millimeters. If the depth of field in this case is within the range indicated by the symbol DOF1 (see FIG. 4), the in-focus area on the surface of the specific object 301 will be only the area indicated by the symbol [A1], as shown in FIG.

[0123] In the situation shown in Fig. 5, an endoscopic image displayed based on image data acquired by the image sensor 26b may appear as shown in Fig. 6. Here, reference numeral 101 in Fig. 6 indicates the display area of ​​the display device 30. Reference numeral 101a in Fig. 6 indicates the entire endoscopic image. Reference numeral 101b in Fig. 6 indicates an image of the specific object 301 (referred to as the specific object image). Reference numeral 101c in Fig. 6 (the cross-hatched area) indicates the in-focus area corresponding to the in-focus area [A1] in Fig. 5. Reference numeral 101d in Fig. 6 indicates the out-of-focus area, which is the area other than the in-focus area 101c in the endoscopic image 101a.

[0124] 5 (in the first observation mode), the in-focus area 101c on the image acquired is only a narrow area near the top 301a of the specific object 301. In this case, the out-of-focus area 101d exists over a wide area compared to the in-focus area 101c of the detected specific object 301, and detailed observation using the image in this state is considered to be time-consuming and difficult.

[0125] Specifically, for example, under these circumstances, when attempting to perform detailed observation of the surface of the specific object 301, first, focus control is performed to the vicinity of the apex 301a to observe the surface near the apex 301a. Then, focus control is again performed to the vicinity of the base 301b to observe the surface near the base 301b, requiring complicated repeated operations and controls. Even so, in this case, detailed observation is performed for each small region, and it is not possible to observe the entire region from the apex 301a to the base 301b in a focused state at once.

[0126] Therefore, the endoscope system 1 of this embodiment has a second observation mode for detailed observation. The second observation mode for detailed observation is realized by a series of image generation processes as shown below.

[0127] First, the imaging distance is set to an imaging distance (for example, within 3 mm) that is a predetermined distance away from the imaging distance used for close-up observation (for example, within 3 mm), and an image of the specific object is captured using AF control.

[0128] Next, an image area including the specific object is extracted from the captured image, and electronic zoom processing such as image enlargement interpolation is performed on the extracted image area.

[0129] The image data thus generated is stored, and an image is displayed based on the generated image data.

[0130] For example, the situation at this time is schematically shown in Fig. 7. The situation shown in Fig. 7 shows the state in the second observation mode when a specific object such as a tumor is observed in detail using the endoscope 20 of the endoscope system 1 of this embodiment.

[0131] As shown in Figure 7, the tip surface of the tip portion 21a of the insertion section 21 of the endoscope 20 is positioned facing the specific object 301 detected on the inner wall surface of the organ, etc. 300, just like in the first observation mode (Figure 5).

[0132] 7 indicates a third imaging distance from the tip surface of the tip portion 21a (the front surface of the imaging optical system 26a of the imaging unit 26) to the top 301a of the specific object 301. This third imaging distance L3 corresponds to the first imaging distance L1 in the first observation mode (FIG. 5).

[0133] 7 indicates a fourth imaging distance from the tip surface of the tip portion 21a (the front surface of the imaging optical system 26a of the imaging unit 26) to the base portion 301b of the specific object 301. This fourth imaging distance L4 corresponds to the second imaging distance L2 in the first observation mode (FIG. 5).

[0134] In this case, the third imaging distance L3 in the second observation mode is set to be greater than the first imaging distance L1 in the first observation mode (L3>L1), as shown in Fig. 7. In this case, if the specific object 301 protrudes from the inner wall surface of an organ or the like, the fourth imaging distance L4 in the second observation mode will also be greater than the second imaging distance L2 in the first observation mode (L4>L2). Note that under the conditions shown in Fig. 7 (in the second observation mode), the third imaging distance L3 is expected to be, for example, approximately 3 to 5 millimeters.

[0135] The symbol DOF2 in Fig. 7 indicates the depth of field corresponding to the currently set imaging distance. In the example of Fig. 7, the imaging distance is the third imaging distance L3 where the focus point is located. The symbol [A2] (thick line portion) in Fig. 7 indicates the area where the surface of the specific object 301 is in focus under the given circumstances (the area included in the depth of field DOF2).

[0136] The depth of field DOF2 is determined according to the uneven range of the specific object 301 being observed. Here, the uneven range of the specific object 301 is found based on distance distribution information of the image area including the specific object 301. As described above, the distance distribution information of the image area is found based on the output from the image sensor 26b.

[0137] Then, once the depth of field DOF2 (the depth of field required in the second observation mode) defined according to the unevenness range of the specific object 301 being observed is determined, the imaging distance L3 that can ensure the depth of field DOF2 can be determined (see Figure 4).

[0138] When capturing an image using the imaging unit 26 with an imaging distance L3 set to ensure the depth of field DOF2, autofocus control is performed so that the surface from the top 301a to the base 301b of the specific object 301 (focus area [A2]) falls within the depth of field DOF2. In this case, for example, focus control is performed by placing the focus point at a predetermined location between the top 301a and the base 301b of the specific object 301 (between imaging distances L3 and L4).

[0139] Note that the symbol An2 in Fig. 7 indicates the apparent angle of view of the image generated by electronic zoom processing. The apparent angle of view An2 is narrower than the angle of view An1 of the imaging optical system that actually captures an image under the conditions shown in Fig. 7. The other symbols in Fig. 7 are the same as those in Fig. 5 described above.

[0140] In the situation shown in FIG. 7, an endoscopic image displayed based on image data acquired by the image pickup device 26b is, for example, as shown in FIG.

[0141] 8 indicate images displayed based on image data acquired with the imaging distance set to L3 in the second observation mode. Of these, image 101A is an image before electronic zoom processing is performed. This image 101A is shown for reference purposes and is not actually displayed by the display device 30 in the endoscope system 1 of this embodiment.

[0142] An image 101 is an image after electronic zoom processing has been performed. This image 101 is actually displayed by the display device 30. Note that the reference symbols in Fig. 8 are the same as those in Fig. 6. Reference symbol 101e in Fig. 8 indicates a frame (electronic zoom target area frame) that indicates an image area that includes the specific object 301 and is the target of electronic zoom processing in the second observation mode.

[0143] An image displayed based on actual image data (image data before electronic zoom processing) acquired in the second observation mode is, for example, image 101A shown in Fig. 8. As described above, the imaging distance at this time is "L3>L1," so the display size (display area) of the specific object image 101b in image 101A is smaller (Fig. 8) than in the display in the first observation mode (Fig. 6).

[0144] In the second observation mode, electronic zoom processing is performed, such as image cropping of the electronic zoom target area frame 101e from this image 101A and image enlargement interpolation of the cropped image area. As a result, the image displayed in the second observation mode becomes image 101 shown in Fig. 8. In this image 101, the specific object image 101b has been subjected to image enlargement interpolation processing, so that the display size (display area) is approximately the same as that in the first observation mode (Fig. 6) (see Fig. 8).

[0145] In addition, in the image 101 after electronic zoom processing, the focused region 101c of the specific object 301 is in focus over almost the entire surface, from the apex 301a to the base 301b of the specific object 301. Therefore, in the image 101, a wide range including the specific object 301 can be displayed in a single image, and the high-resolution image allows for more detailed observation. Therefore, the endoscope system 1 of this embodiment can support easy, reliable, and accurate image diagnosis.

[0146] The illumination control unit 17 is a structural unit or circuit unit that drives and controls the light source device 40 and the illumination unit 25. The illumination control unit 17 controls the light source device 40 or the illumination unit 25 in response to operation instruction signals from various operation members 22c provided in the operation unit 22, or instruction signals based on the detection results of the specific object image detection unit 14 or the determination results of the distance distribution information acquisition and determination unit 15, and performs switching control of the light source type (white light, special light, etc.).

[0147] Here, the light source types include, for example, white light that is irradiated onto the subject when performing normal observation, as well as predetermined special light that is irradiated onto the subject when performing image-enhanced observation such as narrow band imaging (NBI) or red dichromatic imaging (RDI).

[0148] It should be noted that all or part of the processor 10 includes hardware. Here, the processor 10 is configured by a well-known configuration including, for example, a central processing unit (CPU), random access memory (RAM), read only memory (ROM), non-volatile memory, non-volatile storage, and a non-transitory computer readable medium, as well as peripheral devices thereof.

[0149] Software programs to be executed by the CPU and fixed data such as data tables are stored in advance in ROM, nonvolatile memory, nonvolatile storage devices, etc. The CPU reads out the software programs stored in ROM, etc., expands them into RAM, and executes them, and the software programs refer to various data, etc. as appropriate, thereby realizing the functions of the processor 10.

[0150] The processor 10 may also be configured with a semiconductor chip such as an FPGA (Field Programmable Gate Array), etc. Furthermore, the processor 10 may also be configured with an electronic circuit.

[0151] Furthermore, the software program may be recorded or stored, in whole or in part, as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.

[0152] The operation of the endoscope system 1 including the endoscopic image generating device of this embodiment configured as described above will be described below. Fig. 9 is a flowchart explaining the operation of the endoscope system including the endoscopic image generating device of one embodiment of the present invention. The flowchart shown in Fig. 9 is a processing sequence explaining the operation when an endoscopic examination is performed using the endoscope system 1.

[0153] First, it is assumed that the endoscope system 1 is activated and in a state where an endoscopic examination can be performed. The endoscope 20 in the endoscope system 1 is inserted into an organ of a subject (e.g., a patient) to be examined. It is also assumed that the focus control setting in the initial state after activation of the endoscope 20 is, for example, a pan focus control setting.

[0154] When the endoscopic system 1 of this embodiment is in this state, in step S1 of Figure 9, the specific object image detection unit 14 of the processor 10 performs image detection processing of a specific object such as a tumor based on the endoscopic images sequentially acquired by the imaging unit 26, and confirms whether or not a specific object to be observed has been detected.

[0155] If the detection of a specific object is confirmed, the process proceeds to the next step S2. If the detection of a specific object is not confirmed, the process proceeds to step S10. Note that the specific object image detection process of step S1 is executed continuously while the endoscopic examination is being performed.

[0156] Furthermore, although the detection of the specific object in the processing of step S1 described above is performed by the specific object image detection unit 14, this is not limited to this case. For example, the specific object may be recognized by a device operator such as a doctor by visually observing the display on the display device 30.

[0157] In this case, whether or not a specific object has been detected may be confirmed by, for example, checking an output signal output when a device operator such as a doctor operates a predetermined operating member. When a device operator such as a doctor recognizes a specific object, in most cases, the device operator will operate the distal end face of the endoscope insertion portion toward the recognized specific object, and will also perform operations such as moving the distal end face of the endoscope insertion portion closer to the specific object. By detecting this series of operations, it can be determined that the device operator such as a doctor has recognized the specific object.

[0158] In the next step S2, the focus position control unit 16 of the processor 10 starts predetermined autofocus control for the detected specific object. At the same time, the distance distribution information acquisition determination unit 15 of the processor 10 starts processing to acquire distance distribution information (depth data) of the image region including the specific object based on the acquired image data.

[0159] Next, in step S3, the distance distribution information acquisition determination unit 15 of the processor 10 determines the unevenness range of the specific object based on the depth data acquired in the processing of the above-mentioned step S2, and determines the required depth of field according to the unevenness range.

[0160] Next, in step S4, the imaging distance setting unit 18 of the processor 10 sets an imaging distance L that can obtain the depth of field based on the depth of field determined in the processing of step S3 described above.

[0161] Next, in step S5, the processor 10 checks whether the current imaging distance Lp is approaching the set imaging distance L (i.e., whether the state has shifted to the detailed observation state). Here, the current imaging distance Lp can be detected from distance information acquired based on the output signals (image information, phase difference information) continuously output from the imaging element 26b.

[0162] First, it is confirmed whether the current imaging distance Lp is in a state before reaching the set imaging distance L. For example, the distance to a predetermined position before the set imaging distance L is expressed as [L+α] (α is the predetermined distance). Then, the predetermined distance [L+α] is compared with the current imaging distance Lp.

[0163] That is, it is confirmed whether the current imaging distance Lp has changed from a state in which it is greater than the predetermined distance [L+α] (Lp>[L+α]) to a state in which it is approximately equal to the predetermined distance [L+α] (Lp≒[L+α]).

[0164] In this case, if it is confirmed that the state in which the current imaging distance Lp is greater than the predetermined distance [L+α] (Lp>[L+α]) has been maintained for a predetermined time or longer, it can be determined that the distal end 21a of the endoscope 20 is not moving in a direction approaching the specific object (i.e., detailed observation is not being attempted). In this case, the process proceeds to step S10.

[0165] On the other hand, if it is confirmed that the state has changed from (Lp>[L+α]) to (Lp≈[L+α]), it can be determined that the distal end 21a of the endoscope 20 is moving in a direction approaching the specific object (i.e., attempting to perform detailed observation). In this case, the process proceeds to step S6.

[0166] The confirmation in this process also takes into consideration the case where the current imaging distance Lp exceeds the set imaging distance L and becomes too close to the specific object (Lp<L).

[0167] Next, in step S6, the display control unit 12 of the processor 10 controls the display device 30 to display a predetermined guide display in a predetermined area on the display screen. In this case, the predetermined guide display may be, for example, a text message such as "Watch out for approaching objects" or a voice notification, or both.

[0168] Next, in step S7, the processor 10 checks whether the current imaging distance Lp is approximately equal to the set imaging distance L (Lp≈L). If the current imaging distance Lp and the set imaging distance L are approximately equal, the process proceeds to step S8. Note that this process is repeated until the current imaging distance Lp is greater than the set imaging distance L and the two are approximately equal.

[0169] In step S8, the display control unit 12 of the processor 10 controls the display device 30 to display a predetermined guide display in a predetermined area on the display screen. In this case, the predetermined guide display may be, for example, a text message or a voice notification such as "Please stop the insertion portion operation" that calls attention to stopping the endoscope insertion operation that is being performed, or both of these.

[0170] Subsequently, in step S9, the focus position control unit 16 of the processor 10 stops the autofocus control process that was started in the process of step S2 described above. At the same time, the processor 10 controls the electronic zoom processing unit 11a of the image processing unit 11 to perform a predetermined electronic zoom process.

[0171] Here, electronic zoom refers to electronically switching the image range when displaying the range of an object captured by the image sensor 26b at approximately the same size on the display unit. When the image range is wide or narrow, the displayed area appears relatively smaller or larger, which is equivalent to an optical zoom operation.

[0172] The electronic zoom processing performed here includes image cropping and image enlargement interpolation. In this case, the magnification ratio for the image enlargement interpolation is preferably set to a value appropriate for diagnosis, taking into account, for example, the size and type of the specific object, or the size and resolution of the image sensor 26b. In other words, it is preferable to set the magnification ratio so that a clear image can be displayed when performing image diagnosis.

[0173] The image data generated by this electronic zoom processing is transmitted to the display control unit 12 and then sent to the display device 30. Then, an endoscopic image based on the generated image data is displayed on the display screen of the display device 30.

[0174] In this case, various control forms are possible for the electronic zoom processing and display control processing, in addition to a form of display control that instantly switches from the display state of a normal endoscopic image to an image (enlarged image) resulting from the image enlargement interpolation processing of the electronic zoom processing.

[0175] For example, another form of electronic zoom processing and display control processing may involve, for example, generating a progress image that gradually enlarges the image from the display size of a normal endoscopic image during image enlargement interpolation processing in the electronic zoom processing, and performing display control to gradually enlarge the display of the normal endoscopic image. By using such a display form, even though the distal end 21 a (imaging unit 26) of the endoscope 20 is not actually moving, it is possible to make the display appear as if the imaging unit 26 is moving toward the specific object and gradually enlarging the specific object in the endoscopic image.

[0176] Then, in the process of step S9, for example, when it is confirmed that the detailed observation operation has ended, the execution of the electronic zoom process ends. In this case, for example, it may be assumed that the detailed observation state has been switched to the screening state, and the focus control setting may be returned to the pan focus control.

[0177] Here, the end of the detailed observation operation can be confirmed based on the acquired image, for example, by detecting that the image of the specific object being displayed gradually becomes smaller.

[0178] In addition, the display format that may be used when the execution of electronic zoom processing is completed may include a format in which the enlarged image display is instantly changed to a normal captured image, as well as a format in which the object is gradually displayed smaller in a stepwise or seamless manner from the enlarged image display.

[0179] Next, in step S10, the processor 10 confirms an instruction to terminate the endoscopic examination currently being performed. Here, the instruction to terminate the endoscopic examination may be, for example, a predetermined examination termination instruction signal generated by operating a predetermined operating member among the operating members 22c provided in the processor 10 of the endoscopic system 1 or the operating unit 22 of the endoscope 20. In addition, the termination of the endoscopic examination can also be determined, for example, by checking image data of the endoscopic image acquired by the endoscope 20.

[0180] If an instruction to end the test is confirmed in the process of step S10, the series of processes is ended (END). If an instruction to end the test is not confirmed, the process returns to the process of step S1 described above, and the subsequent processes are repeated.

[0181] As described above, the endoscopic system including the endoscopic image generating device of the embodiment is configured to have a second observation mode for detailed observation, in which distance distribution information in the front-to-rear direction of the object is acquired based on an output signal from an imaging unit provided at the tip of the endoscope insertion portion, and focus adjustment is performed by adjusting the position of the focus lens on the optical axis in accordance with the acquired distance distribution information, and image processing is performed to limit the range of the image signal acquired from the imaging unit.

[0182] That is, during detailed observation in the second observation mode, imaging is performed with a longer imaging distance than during close-up imaging, which has a shallower depth of field. At this time, the imaging distance is set to ensure a depth of field that can encompass the uneven range of the object. Therefore, the endoscopic image based on the image data acquired with this setting ensures a wider range of focus in the forward and backward directions in accordance with the uneven range of the specific object.

[0183] Then, the endoscopic image is subjected to an image cropping process to crop an image region including the specific object, and an electronic zoom process to enlarge and interpolate the image region cropped by the image cropping process. The display image based on the image data thus generated is displayed at approximately the same display size as a display image based on image data captured at a position close to the specific object.

[0184] In other words, according to this embodiment, it is possible to acquire images in focus over a wider range in the front-to-rear direction while maintaining a predetermined imaging distance without bringing the imaging unit extremely close to the object being observed (imaged). Then, by performing electronic zoom processing on the image data acquired in this manner, including image cropping of the required image region (i.e., the image region including the specific object) and image enlargement interpolation of the cropped image region, it is possible to ensure a magnification ratio approximately equivalent to that achieved during close-up imaging. This makes it possible to display an image of approximately the same size as that displayed during close-up imaging using conventional endoscopes, while maintaining high-definition image quality.

[0185] As described above, according to this embodiment, it is possible to ensure a wide range of focus within the screen, and at the same time, it is possible to generate a clear, high-definition enlarged image at all times. Therefore, such endoscopic images can realize a less invasive endoscopic examination and contribute to the realization of a more reliable and accurate diagnosis.

[0186] In the endoscope system 1 of the present embodiment, when a specific object is detected and detailed observation is to be performed, detailed observation is performed in the second observation mode, but the present invention is not limited to this control configuration. For example, when performing detailed observation of a detected specific object, the device operator, such as a doctor, may be able to select between the first observation mode and the second observation mode.

[0187] In the above-described embodiment, the processing control sets the imaging distance in the second observation mode according to the depth of field taking into account the range of unevenness in the forward and backward directions of the detected specific object (step S4 in Figure 9).

[0188] However, the setting of the imaging distance in the second observation mode is not limited to the example of the processing control described above. For example, it would be convenient if the imaging distance that can ensure a display size suitable for observation or a target display size could be determined based on the characteristics (type, etc.) and size, etc., of a specific object such as a polyp or tumor detected during an endoscopic examination. Therefore, processing control that sets the imaging distance taking into account the characteristics, such as the type, and size information of the specific object, for example, may also be considered.

[0189] For example, by having the second inference model 210 shown in Figure 10 in the processor 10 (e.g., the specific object image detection unit 14), it is possible to estimate a display ``target size'' suitable for diagnosing a specific object detected from the input endoscopic image data 202.

[0190] Therefore, a modified example of the operation when performing an endoscopic examination using the endoscope system 1 will be described with reference to the processing sequence of Fig. 11. Fig. 11 is a flowchart showing a modified example of the operation of this embodiment, which shows a part of the processing sequence of Fig. 9 modified.

[0191] In Fig. 11, the processes of steps S1, S3, S4, S5, and S6 are the same as those in Fig. 9. Furthermore, the processes from step S8 onwards in Fig. 9 are exactly the same as those in Fig. 9, so illustration and description thereof will be omitted. Only the process steps that differ from the process sequence in Fig. 9 will be described in detail below.

[0192] That is, in the process of step S1, for example, if a specific object is detected by the specific object image detection unit 14, the process proceeds to step S2A.

[0193] In step S2A, the processor 10 starts autofocus control for the detected specific object and acquires feature information, size information, etc. of the specific object based on the acquired image data. At the same time, it starts acquiring distance distribution information (depth data) of the image area including the specific object based on the image data. Then, the process proceeds to step S3.

[0194] After the imaging distance L is set in step S4, the process proceeds to step S4B. In step S4B, the specific object image detection unit 14 of the processor 10 performs inference on the "target size" using the second inference model 210. Then, the process proceeds to step S5.

[0195] If it is confirmed in the process of step S5 that the current imaging distance Lp is approaching the set imaging distance L (that is, the state has shifted to the detailed observation state), the process proceeds to step S5B.

[0196] Next, in step S5B, the processor 10 calculates an imaging distance Lx at which the specific object in the image displayed on the display screen of the display device 30 can be displayed at the "target size" estimated in the processing of step S4 described above. Then, the processor 10 resets the calculated imaging distance Lx as the new set distance L.

[0197] Next, in step S5C, the processor 10 checks whether the current imaging distance Lp is approaching the new set imaging distance Lx. That is, it checks whether there is a change from the state (Lp > Lx) to the state (Lp ≈ Lx). If a change is confirmed, the process proceeds to step S6. If a change is confirmed but the state (Lp > Lx) continues, the process proceeds to step S6B.

[0198] In step S6B, the display control unit 12 of the processor 10 controls the display device 30 to display a predetermined guide display in a predetermined area on the display screen. In this case, the predetermined guide display may be, for example, a text message such as "Please come closer," a voice notification, or both.

[0199] In the next step S7A, the processor 10 checks whether the current imaging distance Lp and the set target imaging distance Lx are approximately equal (Lp≈Lx). If the current imaging distance Lp and the target imaging distance Lx are approximately equal, the process proceeds to step S8. Note that the current imaging distance Lp is greater than the target imaging distance Lx, and this process is repeated until the two are approximately equal. The subsequent process is the same as that shown in FIG.

[0200] According to the processing sequence of this modified example, it is possible to display the "target size" estimated based on the characteristic information such as the type of the detected specific object, thereby contributing to more reliable and accurate support for image diagnosis.

[0201] Incidentally, some imaging optical systems of imaging units applied to endoscopes are configured so that, for example, when the focus lens position is changed for focus control, the focal length changes, and as a result, the angle of view changes.

[0202] For example, there is known an imaging optical system in which, when focus adjustment control is performed on an object located at close range by changing the focus lens position, the focal length of the entire imaging optical system shifts, for example, toward a shorter focal length, thereby widening the angle of view of the image obtained.

[0203] The present invention can also be applied to an endoscopic image generating device in an endoscopic system that uses an imaging unit including such an imaging optical system. To this end, the imaging optical system is configured to include at least a focus lens that moves along the optical axis to adjust the focus, and a zoom lens that moves along the optical axis to adjust the zoom.

[0204] The focus position control unit adjusts the positions of the focus lens and the zoom lens on the optical axis in accordance with the distance distribution information acquired by the distance distribution information acquisition unit. The electronic zoom processing unit performs an image cropping process to crop a partial area of ​​the image based on image data acquired by the imaging unit, and an enlargement interpolation process to enlarge the image area cropped by the image cropping process.

[0205] By configuring in this manner, even in an endoscope that uses an imaging optical system in which the focal length and angle of view change as the focus lens position is changed, it is possible to obtain the same functions and effects as in the above-described embodiment.

[0206] The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the spirit and scope of the invention. For example, while an endoscope has been described as an example, the present invention can be applied to any device that sequentially acquires and evaluates images (such as an image inspection device). Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple components. For example, if the problem to be solved by the invention can be solved and the effects of the invention can be obtained even if some components are deleted from all the components shown in one embodiment, the configuration from which these components are deleted can be extracted as an invention. Furthermore, components from different embodiments may be appropriately combined. The present invention is not limited by specific embodiments other than as limited by the appended claims.

Claims

1. An endoscopic image generating device comprising: a distance distribution information acquiring unit that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; a focus position control unit that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the distance distribution information; and an electronic zoom processing unit that performs image processing to limit the area of the image signal acquired from the imaging unit in order to generate an image to be displayed on a display unit.

2. The endoscopic image generating device according to claim 1, characterized in that the focus position control section adjusts the position of the focus lens on the optical axis based on changes in the image signal output from the imaging unit.

3. The endoscopic image generating device according to claim 1, characterized in that the focus position control unit detects whether the endoscope is in a screening operation state or a detailed observation operation state by determining changes in the image signal and the operating status of the endoscope, and switches the position of the focus lens on the optical axis according to the detected operating status.

4. The endoscopic image generating device of claim 1, characterized in that when the characteristics of the change in the image signal accompanying the movement of the imaging unit indicate a screening operation in order to search for the object to be observed, the focus position control unit executes pan-focus control, and when it is determined that the characteristics of the change in the image signal accompanying the movement of the imaging unit for detailed observation of the object to be observed have transitioned to an operation for performing detailed observation, the focus position control unit starts auto-focus control.

5. The endoscopic image generating device according to claim 4, characterized in that, when the focus position control unit starts autofocus control, the distance distribution information acquisition unit simultaneously starts acquiring distance distribution information.

6. An endoscopic image generating device as described in claim 4, further comprising a specific object image detection unit that detects an image area including a specific object to be observed from an image based on image data acquired by the imaging unit, and wherein the autofocus control executed by the focus position control unit is performed on the detected object to be observed.

7. An endoscopic image generating device as described in claim 6, further comprising: an object unevenness range determination unit that determines the unevenness range of the observed object based on the distance distribution information; a depth of field determination unit that determines the depth of field according to the object unevenness range; and an imaging distance setting unit that sets an imaging distance according to the depth of field, wherein the current position of the tip of the endoscope insertion unit is determined based on the distance distribution information, and when the tip of the endoscope insertion unit reaches the set imaging distance, the focus position control unit stops autofocus control for the observed object, and the electronic zoom processing unit starts the image processing.

8. The endoscopic image generating device according to claim 7, characterized in that the electronic zoom processing unit performs at least an image cropping process for cropping a partial area of an image based on image data acquired by the imaging unit, and an enlargement interpolation process for the image area cropped by the image cropping process.

9. The endoscopic image generating device according to claim 8, wherein the electronic zoom processing section performs image cutting out processing to cut out the image area detected by the specific object image detection section.

10. The endoscopic image generating device according to claim 7, characterized in that, when the end of the operation of performing detailed observation of the object of observation is detected, the electronic zoom processing unit ends the image processing, and the focus position control unit switches to the pan focus control.

11. The endoscopic image generating device according to claim 7, further comprising a guide display unit, which displays a guide message to warn the user when the tip of the endoscope insertion portion approaches the set imaging distance or when the tip of the endoscope insertion portion reaches a position equal to the set imaging distance.

12. The endoscopic image generating device described in claim 6, characterized in that the specific object image detection unit has a first inference model constructed using a large amount of specific object image data as training data and annotated by displaying a frame surrounding the image area containing the specific object.

13. The endoscopic image generating device described in claim 6, characterized in that the specific object image detection unit has a second inference model that is constructed using a large amount of specific object image data as training data, is constructed by annotating the size of the specific object within the image area containing the specific object, and outputs the result of inferring a display size appropriate for diagnosis in response to input.

14. An endoscopic image generating device as described in claim 7, further comprising a guide display unit, which displays a guide message to warn the user when the tip of the endoscope insertion portion approaches an imaging distance at which a display size appropriate for the diagnosis can be realized.

15. The endoscopic image generating device according to claim 7, characterized in that the electronic zoom processing unit enlarges and interpolates a partial area cut out from an image based on image data acquired by the imaging unit to a display size suitable for the diagnosis.

16. An endoscopic image generating device comprising: a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions relative to a specific object to be observed based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; a focus position control unit that adjusts the positions of a focus lens and a zoom lens on the optical axis in accordance with the distance distribution information; and an electronic zoom processing unit that performs image processing to limit the range of image signals acquired from the imaging unit.

17. An endoscopic image generating method comprising the steps of: acquiring distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; adjusting the relative position on the optical axis between a focus lens and an imaging element included in said imaging unit according to said distance distribution information; and performing image processing to limit the range of image signals acquired from said imaging unit in order to generate an image to be displayed on a display unit.

18. An endoscopic image generating program that causes a computer to execute the following: distance distribution information acquisition processing that acquires distance distribution information in the forward and backward directions of an object based on an output signal from an imaging unit provided at the tip of an endoscope insertion portion; lens position control processing that adjusts the relative position on the optical axis between a focus lens and an imaging element included in the imaging unit in accordance with the distance distribution information; image cropping processing that crops out a partial area of an image based on image data acquired by the imaging unit in order to generate an image to be displayed on a display unit, and electronic zoom image processing that limits the area of image data acquired by the imaging unit; specific object image detection processing that detects an image area including a specific object to be observed from an image based on image data acquired by the imaging unit; object unevenness range determination processing that determines the unevenness range of the object to be observed based on the distance distribution information; depth of field determination processing that determines the depth of field according to the object unevenness range; and imaging distance setting processing that sets an imaging distance according to the depth of field.

19. An endoscopic system comprising: an endoscope including an imaging unit including an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system to obtain an image signal; a focus adjustment mechanism that moves some of the optical lenses included in the imaging optical system forward and backward in a direction along the optical axis; and a processor, wherein the processor comprises: a distance distribution information acquisition unit that acquires distance distribution information in the forward and backward directions of the object based on an output signal from the imaging unit; a focus position control unit that adjusts the relative position on the optical axis between the focus lens and the imaging element included in the imaging unit in accordance with the distance distribution information; and an electronic zoom processing unit that performs image processing to limit the range of the image signal acquired from the imaging unit in order to generate an image to be displayed on a display unit.

20. The endoscopic system described in claim 19, further comprising a specific object image detection unit that detects an image area including a specific object to be observed from an endoscopic image based on image data acquired by the imaging unit, and when a specific object is detected by the specific object image detection unit, the focus position control unit starts a focus adjustment operation targeting the detected object to be observed, and the distance distribution information acquisition unit starts an operation to acquire the distance distribution information.

Images