Focus control device, focus control method, endoscope system

Abstract

The present invention is a focus control device (10) that, based on an output signal from an imaging unit (26), moves a focus lens (26ab) included in an imaging optical system (26a) in a direction along an optical axis (O) to perform focus adjustment, and is provided with a processor that detects distance information from a front end of the imaging optical system to an imaging subject, and in a case where the distance information is contained within a prescribed distance range, performs an automatic focus control that adjusts a position of the focus lens on the optical axis in accordance with a change in imaging distance.

Description

Technical Field

[0001] This invention relates to a focusing control device, a focusing control method, and an endoscope system, which are used, for example, in medical devices such as endoscopes for observing the interior of organs of living organisms, to perform focusing control of the imaging optical system. Background Technology

[0002] In the past, endoscope systems have been widely used in fields such as medicine and industry. An endoscope system consists of an endoscope that captures images of the inside of a subject to obtain image data, a processor that performs various image processing on the image data obtained by the endoscope, a display device that displays the image data processed by the processor as a visually recognizable image, and a storage device that records or stores the image data.

[0003] Furthermore, in the medical field, various examinations using this type of endoscope system are widely conducted to observe the interior of biological organs and the like. In examinations using a medical endoscope system (hereinafter referred to as endoscopy), for example, while performing operations such as inserting and withdrawing the insertion portion of an endoscope into the lumen of a biological organ or the like along the length of the lumen, an imaging unit located at the tip of the insertion portion captures images. At this time, the endoscopic images acquired by the imaging unit are displayed in real-time, sequentially in a time sequence, as dynamic images on a display device. Simultaneously, this dynamic image data can also be stored in a storage device.

[0004] In such endoscopic examinations, doctors and other equipment operators observe real-time endoscopic images displayed on a display device (or reproduced endoscopic images based on recorded or stored endoscopic image data after the examination) while searching for lesions such as polyps or tumors that have formed on the inner walls of organs as specific objects.

[0005] Furthermore, during endoscopic examinations, when identifying and observing specific objects such as tumors, endoscopic instruments (hereinafter referred to as instruments) are used. In this case, the instruments are positioned near the object to be observed (on the camera unit side of the endoscope). In such situations, for example, when autofocus control of the camera optical system of the camera unit is performed for more detailed observation, the instrument may sometimes be focused. Therefore, in such cases, the object to be observed may sometimes not be focused.

[0006] Therefore, in a camera focusing control device disclosed, for example, in Japanese Patent Publication No. 2006-106356, when there is an unwanted object (such as a mesh screen) between the camera and the subject, the distance from the focusing position of the camera lens to the closest point of the depth of field is used as a determination distance. If this determination distance is below a preset limit (near), AF control is prohibited. Furthermore, if a limit value is set at a distance farther than the specific subject (such as a mesh screen) from which focusing is prohibited, focusing on the specific subject (such as a mesh screen) can be prevented by AF control. Moreover, for subjects located in the area where AF control is prohibited, focusing can be achieved through manual focus (MF) operation.

[0007] However, in endoscopic examinations, the imaging distance from the camera unit to the object is set to an extremely short range (e.g., a few centimeters to a few millimeters). Therefore, the depth of field of the camera optical system used in endoscopic examinations is typically very narrow.

[0008] Therefore, it is considered that the prior art disclosed in Japanese Patent Publication No. 2006-106356, which sets the focusing prohibition area by distance, is not suitable for direct application in focusing control devices used in medical equipment such as endoscopes and image inspection devices.

[0009] The purpose of this invention is to provide a focusing control device, focusing control method, and endoscope system that can switch to appropriate focusing control according to the usage status of medical devices such as endoscopes and image examination devices. Summary of the Invention

[0010] Methods for solving problems

[0011] To achieve the above objectives, one aspect of the present invention provides a focus control device that adjusts the focus by moving a focusing lens included in a camera optical system back and forth along the optical axis based on an output signal from a camera unit. The focus control device includes a processor that acquires distance information from the front end of the camera optical system to the object being photographed. If the distance information is within a predetermined distance range, the processor performs autofocus control, which adjusts the position of the focusing lens on the optical axis according to changes in the imaging distance.

[0012] One aspect of the present invention provides a focus control method that adjusts the focus by moving a focusing lens included in a camera optical system back and forth along the optical axis based on an output signal from a camera unit. The method involves detecting distance information from the front end of the camera optical system to the object being photographed. If the distance information is within a predetermined distance range, autofocus control is performed. This autofocus control adjusts the position of the focusing lens on the optical axis according to changes in the camera distance. If the distance information is greater than a first distance, focus control is performed to maintain the focusing lens at a first position or a first' position farther than the first position. If the distance information is equal to or less than a second distance, focus control is performed to maintain the focusing lens at a second position or a second' position closer to the second position.

[0013] An endoscope system according to one aspect of the present invention includes an endoscope and a processor. The endoscope includes a camera unit and a focus adjustment mechanism. The camera unit includes a camera optical system for forming an optical image of an object and an imaging element for photoelectric conversion of the optical image formed by the camera optical system to obtain an image signal. The focus adjustment mechanism causes a portion of the optical lens included in the camera optical system to move back and forth along the optical axis. The processor performs the following processing: Detect the distance information from the front end of the camera optical system to the object being photographed. When the distance information is within a specified distance range, autofocus control is performed. This autofocus control adjusts the position of the focusing lens on the optical axis according to the change in the camera distance. When the distance information deviates from the specified distance range, if the distance information is greater than a first distance, autofocus control is performed to maintain the focusing lens at a first position or a first' position farther than the first position. If the distance information is equal to or less than a second distance, autofocus control is performed to maintain the focusing lens at a second position or a second' position closer to the second position.

[0014] According to the present invention, a focusing control device, a focusing control method, and an endoscope system are provided that can switch to appropriate focusing control according to the usage status of medical devices such as endoscopes and image examination devices. Attached Figure Description

[0015] Figure 1 This is a structural diagram that schematically illustrates the overall structure of an endoscope system incorporating a focusing control device according to one embodiment of the present invention. Figure 2 This is a schematic structural block diagram illustrating the internal structure of an endoscope system incorporating a focusing control device according to one embodiment of the present invention. Figure 3 This is an explanatory diagram of the reasoning model used to obtain learning results in a learning device. Figure 4 It is a conceptual illustration of the application. Figure 1 The diagram illustrates the structure of the imaging element of the endoscope in the endoscope system and explains the principle of the phase difference (AF) of the imaging surface of the imaging element. Figure 5 It is shown Figure 1 A schematic diagram of the first scenario (normal search) when the endoscope of the endoscopic system is in the first observation state (screening state). Figure 6 It is shown Figure 1 A schematic diagram of the second scenario (when a specific object is detected) when the endoscope of the endoscope system is in the first observation state (screening state). Figure 7 It is shown Figure 1 A schematic diagram of the endoscope in the second observation state (approach state) of the endoscopic system. Figure 8 It is shown Figure 1 A schematic diagram showing the endoscope of an endoscopic system in the third observation state (detailed observation state). Figure 9 This is a flowchart illustrating the operation of an endoscope system according to one embodiment of the present invention. Figure 10 It is shown Figure 9 A flowchart detailing step S3 (endoscope status determination process) is provided. Figure 11 This is a graph showing the relationship between the light emission L of the light source and the distance D in the endoscope system of this embodiment. Figure 12 This is an example of an endoscopic image shown when the endoscope in the endoscopic system of this embodiment is in the first observation state (screening state). Figure 13 This is a conceptual diagram of an endoscopic image illustrating the fogging situation during endoscopic examination using the endoscopic system of this embodiment. Figure 14 This is a schematic diagram illustrating the concept of consecutive adjacent pixels in an endoscopic image. Figure 15 This is a conceptual diagram illustrating a situation where a large area of ​​brightness exists within the endoscopic image during an endoscopic examination using the endoscopic system of this embodiment. Figure 16 This is a conceptual diagram illustrating the vector changes that occur within the endoscopic image during endoscopic examination using the endoscopic system of this embodiment. Figure 17This is a conceptual diagram illustrating the situation where multiple fine bright spots appear in the endoscopic image during endoscopic examination using the endoscopic system of this embodiment. Figure 18 This is a conceptual diagram illustrating a situation where three relatively large bright spots appear in the endoscopic image during an endoscopic examination using the endoscopic system of this embodiment. Figure 19 yes Figure 18 The rectangle F shows an enlarged view of the area containing at least one bright spot. Detailed Implementation

[0016] The present invention will now be described, for example, through illustrated embodiments as application examples of an endoscope system. The accompanying drawings used in the following description are schematic representations. Therefore, in these drawings, each component is represented by a size that is readily identifiable on the drawing. Thus, sometimes the dimensional relationships of the components in the drawings, scale bars, etc., are represented differently for each component. The present invention is not limited to the illustrated representation regarding the quantity, shape, size ratios, relative positional relationships, etc., of each component shown in the drawings.

[0017] First, before describing the detailed structure of the focusing control device according to one embodiment of the present invention, the following will use... Figure 1 , Figure 2 The general structure of an endoscope system including the focusing control device of this embodiment will be described. Figure 1 This is a structural diagram that schematically illustrates the overall structure of an endoscope system incorporating a focusing control device according to one embodiment of the present invention. Figure 2 This is a schematic structural block diagram showing the internal structure of the endoscope system.

[0018] like Figure 1 , Figure 2 As shown, the endoscope system 1 is mainly composed of a processor 10, an endoscope 20, a display device 30, a light source device 40, and a storage device 50. Figure 1 The endoscopic system shown illustrates the general structure of an endoscopic system used in endoscopic examinations of subjects (patients, etc.) where the internal organs of the upper digestive tract (esophagus, stomach, duodenum, etc.) or lower digestive tract (large intestine, etc.) are observed.

[0019] The endoscope 20 is an image inspection device configured to have an insertion part 21, an operation part 22, and a universal cable 23.

[0020] The insertion section 21 is a component inserted into a biological object or other subject for examination. The insertion section 21 is formed by sequentially connecting a front end portion 21a, a bend portion 21b, and a flexible tube portion 21c from the front end side. The insertion section 21 is generally formed into a long and narrow tube shape. Internally, the insertion section 21 has a passageway 21d for inserting an endoscopic instrument 27. This instrument insertion channel 21d is configured to extend from the front end to the base end of the insertion section 21. Furthermore, an operating section 22 is connected to the base end side of the insertion section 21.

[0021] The front end portion 21a is a structural unit located at the very front end of the insertion portion 21. For example... Figure 2 As shown, various components such as a lighting unit 25 and a camera unit 26 are provided inside the front end 21a. Figure 1 (Not shown in the image).

[0022] Here, as Figure 2 As shown, the illumination unit 25 is a structural unit including an optical fiber cable 25a and an illumination lens 25b, etc. The optical fiber cable 25a guides the light beam from the light source device 40 to the front end 21a; the illumination lens 25b is a plurality of optical elements that allow the light beam guided from the light source device 40 to pass through and then be emitted forward from the front end face of the front end 21a to illuminate the observation area of ​​the subject body, including lesions, etc. Additionally, as... Figure 1 As shown, a plurality of illumination lenses 25b are provided on the front end face of the front end portion 21a of the insertion portion 21 of the endoscope 20. Furthermore, in the endoscope 20 illustrated in this embodiment, a structural example with three illumination lenses 25b is shown (see reference). Figure 1 ).

[0023] Additionally, the camera unit 26 consists of the camera optical system 26a (see reference 26a). Figure 2 ), photoelectric conversion element (camera element 26b; see reference) Figure 2 The imaging optical system 26a comprises an electronic device unit including an observation window 26aa and a focusing lens 26ab, etc., which forms an optical image of a specific object (including biological tissue such as lesions such as tumors; hereinafter referred to as the specific object) that is being observed inside the subject body. The photoelectric conversion element generates image information (static image data and dynamic image data, etc.) based on the optical image. The focusing lens driving mechanism 26c drives the focusing lens 26ab included in the imaging optical system 26a. Additionally, as... Figure 1 As shown, the observation window 26aa included in the camera optical system 26a is disposed on the front end face of the front end portion 21a of the insertion portion 21 of the endoscope 20.

[0024] Figure 1The bending portion 21b shown is a tubular portion configured to be actively bent freely by a bending operation mechanism (not shown) that operates in conjunction with the bending operation member 22b described later. The flexible tube portion 21c is a tubular member that extends from the front end of the operation portion 22 and is connected to the base end of the bending portion 21b.

[0025] The operating part 22 is connected to the base end of the insertion part 21. The operating part 22 is configured to have an operating part body 22a, a bending operating member 22b, multiple operating members 22c, and a treatment device insertion port 22d, etc.

[0026] The main body 22a of the operating section is generally box-shaped and serves as a grip for the endoscope 20 to be held by the user (doctor, etc.). Various structural units, such as a bending operating mechanism (not shown), are provided inside the main body 22a. As described above, an insertion section 21 extends from the main body 22a.

[0027] The bending operation component 22b and the plurality of operation components 22c are operation components used to perform various operations on the endoscope 20. These bending operation components 22b and the plurality of operation components 22c are respectively disposed at predetermined positions on the outer surface of the operation part body 22a.

[0028] The treatment device insertion port 22d is located at a predetermined position near the front end of the main body 22a of the operating section. The treatment device insertion port 22d is the base end side opening of the treatment device insertion channel 21d of the insertion section 21. The treatment device insertion channel 21d is connected at the front end to the front end channel opening 21da provided on the front end face of the front end 21a.

[0029] With this structure, the endoscope treatment device 27 inserted from the treatment device insertion port 22d is configured such that, after being inserted into the treatment device insertion channel 21d, it can protrude outward from the front end channel opening 21da of the front end 21a.

[0030] In addition, Figure 1 In the illustration, a biopsy forceps in the shape of a clip is shown as an example of an endoscopic treatment instrument 27. However, the endoscopic treatment instrument 27 is not limited to the illustrated form. Other examples of endoscopic treatment instruments 27 include, for instance, drug dispensing catheters, loop-shaped snares, injection needles, high-frequency knives, holding forceps, basket-shaped holding forceps, hemostatic clips, and various other types of instruments. Therefore, in the following description, the endoscopic treatment instrument 27 will be simply referred to as treatment instrument 27.

[0031] The universal cable 23 is a connection cable used to connect the endoscope 20 to the light source device 40 and the processor 10. Therefore, the universal cable 23 is composed of a tubular component extending from the side of the operation unit body 22a of the operation unit 22. A scope connector 23a is provided at the front end of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

[0032] Cable 23b extends from mirror connector 23a. Connector 23c is located at the front end of cable 23b. Connector 23c connects to the front panel of processor 10. Figure 2 As shown, various signal transmission cables 10a, fiber optic cables 25a, etc. are inserted into the general cable 23.

[0033] like Figure 2 As shown, the light source device 40 includes a light source 41 that supplies illumination light to the illumination unit 25, which is disposed inside the front end portion 21a of the insertion portion 21 of the endoscope 20. The illumination beam emitted from the light source 41 of the light source device 40 is transmitted from the endoscope connector 23a through an optical fiber cable 25a arranged to pass through the universal cable 23, the operation portion 22, and the insertion portion 21, to the illumination unit 25 at the front end portion 21a. Furthermore, the illumination beam passes through the illumination lens 25b included in the illumination unit 25 at the front end portion 21a and illuminates the observation area in front of the front end portion 21a.

[0034] The processor 10 is a control device or circuit unit that includes the overall control circuitry and signal processing circuitry for controlling the endoscope system 1. Furthermore, the processor 10 is configured to include functions as a focusing control device (details will be described later; see reference). Figure 2 ).

[0035] The control circuitry included in the processor 10 receives, for example, operation instruction signals from the operation unit 22c of the operation section 22 of the endoscope 20, and outputs various control signals for driving and controlling the camera unit 26, the light source device 40, or the illumination unit 25. Furthermore, the signal processing circuitry included in the processor 10 receives, for example, camera signals from the camera unit 26 and performs prescribed image signal processing.

[0036] Therefore, as Figure 2 As shown, the processor 10 and the camera unit 26 are electrically connected via a signal transmission cable 10a. This signal transmission cable 10a is configured to pass from the connector 23c through the cable 23b, the lens connector 23a, the universal cable 23, the operating part 22, and the insertion part 21 to the camera unit at the front end 21a. Furthermore, as... Figure 2As shown, the processor 10 and the light source device 40 are also electrically connected via a signal transmission cable 10a. In this case, the signal transmission cable 10a is connected from the connector 23c to the light source device 40 via the cable 23b from the mirror connector 23a.

[0037] With this structure, control signals output from processor 10 and imaging signals output from camera unit 26 are transmitted between camera unit 26 and processor 10 via signal transmission cable 10a. Similarly, control signals output from processor 10 are transmitted between light source device 40 and light source control unit 17 of processor 10 via signal transmission cable 10a. Furthermore, as one method of using signal transmission cable 10a, a composite cable, for example, is used where multiple cables are bundled together and covered with an outer shielding member, outer tubing, etc.

[0038] The display device 30 is a display device that receives image signals and the like output from the processor 10 and displays endoscopic images and various information in a prescribed manner. Therefore, the display device 30 is electrically connected to the processor 10 using a video cable 24. Furthermore, the display device 30 may be, for example, a display device constructed using a general liquid crystal panel.

[0039] Storage device 50 is a storage device for storing (recording or storing) image data and other information files. The image data is generated for storage by various processing in processor 10 based on the image information generated by camera unit 26.

[0040] like Figure 1 , Figure 2 As shown, the storage device 50 and the processor 10 are integrally formed. Figure 2 As shown, inside the storage device 50, a temporary storage unit 51 is provided separately from the main storage unit (not shown) in the storage device 50. This temporary storage unit 51 is, for example, composed of a semiconductor memory, and functions as a temporary storage area for specified image data output from the image processing unit 11 (described later).

[0041] In addition, Figure 1 The illustrated structural example shows a configuration in which the storage device 50 is integrally disposed within the housing of the processor 10, but this configuration is not limited to this. For example, the storage device 50 may also be configured as an external storage device using a housing separate from the processor 10.

[0042] In addition, Figure 2 The example shown illustrates a structure in which the temporary storage unit 51 is disposed inside the storage device 50, but this configuration is not limited to this. For example, the temporary storage unit 51 may also be integrally disposed within the storage control unit 13 described later (see [reference]). Figure 2 (The interior of)

[0043] Furthermore, the processor 10 and the light source device 40 are not limited to Figure 1 , Figure 2 The illustrated structure is a separate structure. For example, the processor 10 and the light source device 40 can also be integrally constructed using a single housing.

[0044] As an example of the structure of the lighting unit 25, it is not limited to the above-described structure (the way in which the illumination light from the light source device 40 is transmitted to the front end 21a through the optical fiber cable 25a, etc.). As a structure other than this, the lighting unit 25 can also be configured as follows: a light-emitting element such as an LED (Light Emitting Diode) as a light source is provided inside the front end 21a, and the power supply to the light source (LED) and its light emission control are controlled by a predetermined control circuit (light source control unit 17, etc.) included in the processor 10.

[0045] The endoscope system 1 constructed as described above has essentially the same structure as existing endoscope systems of the same type. Therefore, for other detailed structures, illustrations and detailed descriptions are omitted.

[0046] The endoscope system 1 constructed in this way is configured to include a focusing control device function in the processor 10. Therefore, in the endoscope system of one embodiment of the present invention, the following will use... Figure 2 The internal structure of the processor is described in great detail.

[0047] The processor 10 is configured to include an image processing unit 11, a display control unit 12, a storage control unit 13, an image detection unit 14, a distance distribution information acquisition unit 15, a focus control unit 16, a light source control unit 17, an imager drive unit 18, and a storage device 50 including a temporary storage unit 51.

[0048] The image processing unit 11 is a structural unit or circuit unit that acquires the output signal (mainly image information) from the imaging unit 26 and performs various information processing based on the acquired image information. In other words, the image processing unit 11 is a signal processing device that generates an image based on the imaging signal from the imaging element 26b. The image processing performed here includes, for example, general image information processing, display image data processing, storage image data processing, and various other image processing techniques.

[0049] Image processing typically includes analog-to-digital (AD) conversion, which receives analog image signals from the imaging element 26b and converts them into digital image signals. It is the basic image information processing usually performed on image information. Since conventional image processing methods are generally used, detailed descriptions are omitted.

[0050] Image data processing for display is the process of generating image data for general display. For example, image data processing for display may include generating image data representing an endoscopic image displayed within a designated display area on the display screen of the display device 30. Image data processing for storage may include generating image data for recording.

[0051] Various image processing techniques are image processing methods that appropriately append to ordinary display image data. Specific examples of various image processing techniques include common image adjustment processing such as brightness adjustment processing and white balance adjustment processing, and image enhancement processing such as contour enhancement processing and texture and color enhancement imaging (TXI).

[0052] The display control unit 12 is a structural or circuit unit that controls the display device 30 to display the display image data output from the image processing unit 11 in an appropriate display mode. In addition to controlling the display of the display image data (setting various parameters such as display position, display area, and display size), the display control unit 12 also controls the display of various other information (text information, icon information, etc.) besides image information. Among these other information, besides examination date and time information, patient information, and medical record information, there are also various alarm information (information information) and operation guidance information appropriately displayed during endoscopic examinations.

[0053] For example, the display control unit 12 performs display control to continuously display endoscopic images in the display area. Furthermore, when a specific object is detected in the endoscopic image, the display control unit 12 performs display control to add a prescribed display (e.g., a frame display) to the image area containing the specific object.

[0054] Furthermore, the display control unit 12 includes a guidance display unit (not shown). This guidance display unit is, for example, a structural unit or circuit unit that displays various alarm information (information information), guidance information, etc., on the display screen of the display device 30 at appropriate times for equipment operators such as doctors during endoscopic examinations.

[0055] The storage control unit 13 is a structural or circuit unit that controls the storage device 50 (including the temporary storage unit 51) to save (record or store) the image data for storage output from the camera unit 26 or the image processing unit 11 in an appropriate storage manner. The storage control unit 13 performs storage control to permanently store image data representing the endoscopic image obtained by the endoscope 20 in the storage area of ​​the storage device 50.

[0056] The image detection unit 14 is a structural unit or circuit unit that detects an image region containing an image of a specified object (such as a tumor or other specific object, a treatment device, or other objects) from each frame of continuously displayed endoscopic images based on image data acquired by the endoscope 20 and output from the image processing unit 11.

[0057] Therefore, the image detection unit 14 includes an object detection unit 14a, a camera distance detection unit 14b, a brightness distribution detection unit 14c, and a light quantity detection unit 14d.

[0058] The object detection unit 14a is a structural unit or circuit unit that mainly detects image areas containing specific objects such as tumors or treatment instruments based on image information from the output signal from the camera unit 26.

[0059] The camera distance detection unit 14b is a structural or circuit unit that primarily detects the camera distance of objects (specific objects, treatment instruments, other objects, etc.) within an endoscopic image based on phase difference information or image information from the output signal from the camera unit 26. Regarding camera distance detection, it is possible to detect distance information from the front surface of the camera optical system 26a to the object (object, treatment instrument, other objects, etc.), and it is also possible to perform graded region detection within approximately three regions defined at each specified distance range.

[0060] In this embodiment, the camera distance detection unit 14b is envisioned to detect, for example, three levels of regions: a first region where the camera distance exceeds a predetermined first distance (e.g., approximately 4 to 5 mm), a predetermined region within a predetermined distance (e.g., approximately 3 to 4 mm), and a second region where the camera distance is less than a predetermined second distance (e.g., approximately 1.5 to 2 mm). Furthermore, in the following description, the first distance will sometimes be referred to as the "far point" and the second distance as the "near point."

[0061] The brightness distribution detection unit 14c is a structural unit or circuit unit that detects the brightness distribution within an image based on brightness information, primarily obtained from image information received from the output signal of the imaging unit 26. Here, the brightness distribution is the brightness distribution within the image represented by the brightness information of each region when the image is divided into defined regions according to the brightness information output by each pixel of the imaging element.

[0062] The light quantity detection unit 14d is a structural unit or circuit unit that detects the amount of illumination light in a specified area of ​​the image based on brightness distribution information obtained mainly from image information from the output signal from the camera unit 26, or light quantity control information of the illumination light from the light source control unit 17.

[0063] Regarding the detection of specified objects by the image detection unit 14, for example, it identifies and detects specified image regions within the image based on factors such as brightness distribution, exposure distribution, and distance distribution within the image. Furthermore, regarding the detection of specified objects by the image detection unit 14, it can also perform similar image recognition based on multiple pre-prepared object images (similar images, etc.), or perform object detection by data analysis using deep learning or machine learning.

[0064] For example, in recent years, object detection technology using AI (Artificial Intelligence) has been developed. This AI obtains the desired inference results (such as the detection of specific objects such as tumors) by machine learning on endoscopic images obtained by endoscopes.

[0065] Machine learning learns features, time-series information, spatial information, etc., from known input information, and then infers based on the learning results to obtain inferences about unknown matters. That is, in machine learning, a fully learned model (or inference model) is first obtained, which can infer a decidable output result based on specific input information.

[0066] In this context, when generating a fully learned model, a large amount of information about the known relationship between input and output is used as training data to obtain highly reliable inference results. For example, in deep learning, a large amount of training data is used to design the network to obtain the expected output for known inputs. The fully learned model obtained through this process can be used independently of the learned network.

[0067] Therefore, by having such a reasoning model, the processor 10 can accurately and quickly perform various judgments or detections, such as the detection of objects specified in the image detection unit 14, for example, the judgment of the camera distance range that can be captured in the focused state during deep focus control, the judgment of the endoscope usage status during endoscopic examination, and the detection of the surface vascular pattern of a specific object (including the detection of information such as the position of the vascular pattern in the image).

[0068] Here, the inference model is constructed roughly as follows. For example, a large amount of image data corresponding to the inputs and outputs is provided as training data to the network (not shown) used to construct the inference model. Here, the inputs are image data suitable for detection or judgment, such as image data of endoscopic images acquired sequentially in time during endoscopic examinations, and image data of images of specific objects (case images, etc.) captured in the endoscopic images. In this case, annotations such as image regions surrounding specific objects by boxes can also be set.

[0069] By learning from a large amount of training data, a predetermined network (not shown) is designed to produce an output corresponding to the input. For example, Figure 3 It is a conceptual diagram illustrating the reasoning model acquired after learning in a prescribed learning device.

[0070] For example, such as Figure 3 As shown, when an input set of endoscopic images containing a specific object (such as a tumor) (first training data set 201) is provided, additional information such as the specific object captured in the image and the bounding box display of the image region surrounding the specific object is obtained along with the reliability information. Thus, a first inference model 200 for detecting a specific object from endoscopic images is constructed.

[0071] When endoscopic image data (time-series image data) 202 from an endoscopic examination is input into the first inference model 200 constructed in this way, the first inference model 200 detects image regions containing specific objects in frames captured in the endoscopic image data 202, and outputs image data (inference result image 203) with a bounding box attached to the image region of the specific object. In this way, specific objects can be detected based on endoscopic images. Furthermore, by temporarily storing the image data of image 203 or displaying it as needed in a designated display area of ​​the display device 30, it can serve as a reference for the endoscopic examination performed by the endoscopic user.

[0072] Furthermore, although illustrations are omitted, similarly, for example, when an input set of endoscopic images containing instruments such as surgical devices is used as a second training data set, information about the surgical devices captured in the images is obtained along with reliability information. Thus, an inference model for detecting surgical devices based on endoscopic images is constructed.

[0073] Even with such a constructed inference model, it is still possible to detect instruments and other devices based on endoscopic images.

[0074] Furthermore, deep learning is a multi-layered structure of the "machine learning" process that uses neural networks. A representative example is the "forward propagation neural network," which makes decisions by sending information from front to back. In its simplest case, this only requires three layers: an input layer of N1 neurons, an intermediate layer of N2 neurons provided by parameters, and an output layer of N3 neurons corresponding to the number of classes to be classified. Moreover, the neurons in the input layer and the intermediate layer, and the intermediate layer and the output layer, are connected by weighted combinations, and bias values ​​are applied to the intermediate and output layers, easily forming logic gates. For simple discrimination, three layers are sufficient, but by using multiple intermediate layers, it is possible to learn combinations of multiple features during machine learning. In recent years, networks with 9 to 152 layers have become practical due to their advantages in learning time, judgment accuracy, and energy consumption.

[0075] As for the networks used in machine learning, various well-known networks can be employed. For example, R-CNN (Regions with CNN features) and FCN (Fully Convolutional Networks) can be used. This involves a process called "convolution" that compresses the features of an image, performing actions with minimal processing, and is strong in pattern recognition. Furthermore, more complex information can be processed, corresponding to information analysis where meaning changes sequentially or in order, utilizing "recurrent neural networks" (fully connected recurrent neural networks) that allow information to flow bidirectionally.

[0076] To implement these technologies, conventional computing circuits such as CPUs and FPGAs can be used. However, since neural network processing mostly involves matrix multiplication, dedicated computing circuits called GPUs (Graphics Processing Units) or Tensor Processing Units (TPUs) are sometimes utilized. In recent years, such dedicated artificial intelligence (AI) hardware, "Neural Processing Units (NPUs)," has sometimes been designed to be integrated and embedded with other circuits such as CPUs, becoming part of the processing circuitry.

[0077] Furthermore, beyond deep learning, various well-known machine learning methods can also be used to obtain inference models. For example, there are methods called Support Vector Machines (SVMs) and Support Vector Regression (SVR). Here, learning involves calculating the weights, filter coefficients, and biases of the recognizer. In addition, there are methods using logistic regression. When instructing a machine to determine something, the human needs to inform the machine of the determination method. In this embodiment, a method deriving image determination through machine learning is used. However, rule-based methods can also be applied to make specific judgments. This rule base adapts to rules obtained by humans through experience / heuristics.

[0078] Figure 2 The distance distribution information acquisition unit 15 is a structural unit or circuit unit that acquires distance distribution information (hereinafter referred to as distance distribution information) within the camera frame based on the output signal (camera signal; image information or phase difference information) from the camera element 26b.

[0079] Here, the distance distribution information within the camera frame refers to information representing the distribution of distances in the front-to-back direction of objects within the captured image. Furthermore, the front-to-back direction of the object refers to the front-to-back direction (in other words, the depth direction) when observing the object in the image from the camera unit 26 (camera element 26b) located at the front end 21a of the endoscope 20, which is the direction along the direction of the endoscope 20's movement forward and backward.

[0080] In this embodiment, the imaging unit 26 of the endoscope 20 includes an imaging element 26b with a structure having multiple pixels that receive a light beam from the subject and output an image signal. Phase difference detection is performed on some or all of these pixels, which can, for example, aid in autofocus control (AF control) and predefined focus control. In this embodiment, by applying an imaging element 26b in this manner, image information and phase difference information can be acquired.

[0081] Additionally, the following is a brief explanation of the structure that enables phase difference detection of the imaging surface using an imaging element. Figure 4 This is a schematic diagram illustrating the principle of camera surface phase difference AF control, which detects the phase difference of the camera surface through the camera element.

[0082] Figure 4 The reference numeral 260 in the attached figure indicates one pixel of the imaging element 26b. Pixel 260 includes a photodiode 262 and a microlens 261. Here, the photodiode 262 is formed, for example, as being divided into two parts: a right photodiode 262R and a left photodiode 262L.

[0083] Microlens 261 is an optical lens that allows a light beam passing through the imaging optical system 26a to form an optical image on the light-receiving surface of photodiode 262. Photodiode 262 is a photoelectric conversion element that converts the optical image formed by microlens 261 into an electrical signal and outputs the electrical signal.

[0084] The right-side photodiode 262R receives the beam LL from the left-side region. Figure 4 The area indicated by the left-hand diagonal shaded line) is imaged onto the light-receiving surface. Additionally, the left-hand photodiode 262L receives the light beam LR from the right-hand area (in...). Figure 4 The area (represented by the right-hand shading line) is imaged on the illuminated surface.

[0085] The pixel 260 with this structure can calculate the amount of defocus on the object by calculating the two image intervals (parallax) based on the output signals (phase difference information) from the left and right photodiodes 262R and 262L. Furthermore, by matching the output signals of the left and right photodiodes 262R and 262L, they can be processed as an image signal for a single pixel.

[0086] The distance distribution information acquisition unit 15 receives the phase difference information in the output signal obtained by the imaging element 26b, and obtains the distance distribution information of a specified area (e.g., the treatment device area, the biological tissue area containing a specific object, etc.) within the imaging plane.

[0087] In addition, Figure 4 The structure shown divides a single photodiode into two parts, processing signals from the left and right photodiodes respectively, thereby performing pupil segmentation of the microlens. However, this structure is not limited to this one. For example, a similar pupil segmentation structure can be achieved by using a light-shielding component.

[0088] Furthermore, the distance distribution information acquisition unit 15 acquires the temporal change information of the distance information of the object in the front and rear directions based on multiple image data (hereinafter referred to as time series image data) continuously output from the camera unit 26 in a time sequence.

[0089] Furthermore, in the above structural example, the distance distribution information acquisition unit 15 is configured to acquire distance distribution information based on phase difference information from the imaging element 26b, wherein the imaging element 26b has multiple pixels for image signal acquisition and imaging surface phase difference detection, but is not limited to this structure. For example, in a typical general imaging element, all pixels provided on the imaging surface acquire the imaging signal.

[0090] In an imaging unit having an imaging element 26b in such a conventional manner, for example, at least a portion of the optical lenses constituting the imaging optical system 26a, namely the optical lens that aids in focus control (focusing lens 26ab), is moved slightly forward and backward along the optical axis to change the lens position. This results in a change in the contrast of objects in the image, thereby changing the focus state. At this time, distance information can be obtained based on the correlation between the lens position and the focus state corresponding to the lens position. Based on this obtained distance information, distance distribution information within the frame can be obtained. That is, the distance distribution information can also be obtained based on the image signal from the imaging element 26b.

[0091] In this way, the distance distribution information acquisition unit 15 acquires distance distribution information based on the output signal (phase difference information or image information) of the imaging element 26b. That is, the endoscopic image generated based on the output signal (image signal) of the imaging element 26b is correlated and corresponds with the distance distribution information generated based on the output signal (phase difference information or image information) of the imaging element 26b.

[0092] Figure 2 The focusing control unit 16 is a structural or circuit unit that drives and controls the imaging optical system 26a of the imaging unit 26, adjusts the relative position of the focusing lens 26ab and the imaging element 26b on the optical axis, and controls the maintenance of a predetermined focusing lens position. The focusing control unit 16 drives the focusing lens drive mechanism 26c based on the detection results and distance distribution information of the image detection unit 14 to control the focusing of the imaging optical system 26a. In this case, the focusing control unit 16 switches between autofocus control, panoramic focus control, and near-priority focus control depending on the state of the endoscope.

[0093] The light source control unit 17 is a structural unit or circuit unit that drives and controls the light source device 40 and the illumination unit 25. The light source control unit 17 controls the light source device 40 or the illumination unit 25 to switch the type of light source (white light, special light, etc.) based on the operation instruction signals from the various operation components 22c provided in the operation unit 22, or the instruction signals or acquired data based on the detection results of the image detection unit 14 and the distance distribution information acquisition unit 15.

[0094] Here, as for the types of light sources, in addition to the white light that is used to illuminate the subject during normal observation, there are also special lights that are used to illuminate the subject during image emphasis observation, such as narrow band imaging (NBI) or red dichromatic imaging (RDI).

[0095] The imager drive unit 18 is a structural unit or circuit unit that drives and controls the imager element 26b, which is an imager element of the imaging unit 26. The imager drive unit 18 receives predetermined instruction signals, etc., and drives and controls the imager element 26b at predetermined times. As a result, a predetermined imaging signal is output from the imager element 26b. Then, as described above, the image signal in this output signal is output to the image processing unit 11, and the phase difference information is output to the distance distribution information acquisition unit 15.

[0096] The above describes the structure of the endoscope system 1 including the focusing control device of this embodiment. Furthermore, the structure omitted from the description is the same as that of a conventional endoscope system.

[0097] Furthermore, all or part of the processor 10 includes hardware. Here, the processor 10 may be composed of, for example, a central processing unit (CPU), RAM (Random Access Memory), ROM (Read Only Memory), non-volatile memory, non-volatile storage, etc., as well as well-known structures and peripheral devices such as a non-transitory computer readable medium.

[0098] The ROM stores fixed data such as software programs and data tables to be executed by the CPU in advance in non-volatile memory or non-volatile storage devices. Furthermore, the CPU reads the software program stored in the ROM and expands it in RAM for execution. In addition, the software program appropriately references various data to realize the functions of the processor 10.

[0099] Alternatively, the processor 10 can also be constructed from semiconductor chips such as FPGAs (Field Programmable Gate Arrays). Furthermore, the processor 10 can also be constructed from electronic circuits.

[0100] Furthermore, software programs can also be configured as follows: as computer program products, they record or store in whole or in part on removable media such as floppy disks, CD-ROMs, DVD-ROMs, card-type storage devices, HDD (Hard Disk Drive) devices, SSD (Solid State Drive) devices, and other non-transitory computer readable mediums.

[0101] During endoscopic examinations using such an endoscope system, doctors and other equipment operators observe the endoscopic images displayed on the display device while performing a screening process called "screening," which involves searching for lesions such as polyps or tumors that have formed on the inner walls of organs as specific objects (hereinafter referred to as specific objects).

[0102] In such cases, for example, because specific objects such as tumors being observed are very small and their color tone is similar to that of the surrounding organ walls, it is necessary to be skilled in reliably identifying and detecting the image region of the specific object from the endoscopic image. Furthermore, in addition, skill is required in the endoscopic procedures themselves, such as manipulating the tip of the endoscope insertion section (the viewing window of the camera unit) to photograph the desired object during the endoscopic examination.

[0103] Generally, the use of the endoscope during endoscopic examinations can be roughly divided into several states. In addition to the screening state (referred to as the first observation state or screening state), there is also a state for detailed observation of the specific object being observed (referred to as the third observation state or detailed observation state) and a state in the process of transfer (referred to as the second observation state or approach state). This transfer process is the process of bringing the tip of the endoscope closer to the specific object being observed in order to conduct detailed observation from the screening state.

[0104] Here, use Figures 5-8 A brief explanation of how to use an endoscope. Figure 5 , Figure 6 This is a schematic diagram showing the state of the endoscope 20 in the first observation state (screening state). Wherein, Figure 5 The first scenario, where the filtering process is in progress and no specific object is detected (during a specific object search), is shown. Additionally, Figure 6 The second scenario, representing the filtering state and the state when a specific object is detected, is shown (when a specific object is detected).

[0105] exist Figure 5 In the accompanying drawing, reference numeral 101b schematically illustrates the lumen of organs, etc. Figure 5 The image shows an endoscope 20 with its insertion portion 21 (front end portion 21a) inserted into the lumen of the organ 101b. Furthermore, Figure 5 Reference numeral 101a in the attached figures indicates a specific object such as a tumor present on the inner wall of an organ 101b. Additionally, in Figure 5 In the illustrated situation, the specific object 101a is not detected by the endoscope 20. That is, the specific object 101a is outside the field of view of the endoscope 20. Furthermore, Figure 5The reference numeral V in the attached figure indicates the imaging range (field of view) of the imaging optical system 26a of the imaging unit 26 built into the front end 21a.

[0106] Furthermore, in Figure 5 In the figure, reference numeral O indicates the central axis of the insertion portion 21 of the endoscope 20 (or it can be considered as the optical axis of the imaging optical system 26a of the imaging unit 26, hereinafter referred to as the central axis). The insertion portion 21 of the endoscope 20 is operated so that its central axis O moves forward and backward in the direction along the lumen of the organ, etc. 101b.

[0107] When endoscope 20 is in screening mode, such as Figure 5 As shown, the insertion portion 21 of the endoscope 20 is arranged in a generally straight line along the lumen of the organ, etc. 101b.

[0108] Next, set the endoscope 20 along... Figure 5 The arrow X direction from Figure 5 The position of [A] (also refer to) Figure 6 (dashed line) towards Figure 6 [B] position movement (refer to) Figure 6 At this time, such as Figure 6 As shown, the endoscope 20 is in a state where a specific object 101a is included within its imaging range (field of view) V. Thus, when the endoscope 20 is in a state where the specific object 101a is included within its imaging range (field of view) V, the image detection unit 14 (imaging distance detection unit 14b) detects the specific object 101a.

[0109] Here, as for the case where a specific object 101a is detected, for example, as described above, in addition to the case detected by the image detection unit 14, it also includes the case where the user of the endoscope 20 (such as a doctor) recognizes the specific object in the image by visually observing the endoscope image.

[0110] When a specific object 101a is detected by the image detection unit 14, the detection result is communicated to the user of the endoscope 20 (such as a doctor) in a prescribed manner. This notification is displayed on the display screen of the display device 30 in a prescribed manner. Alternatively, notification may be provided via sound or other means. Thus, the user of the endoscope 20 (such as a doctor) is able to recognize that a specific object 101a has been detected.

[0111] Thus, when the user of the endoscope 20 (such as a doctor) identifies a specific object 101a, the user (such as a doctor) sometimes needs to observe that specific object 101a in detail. Therefore, the user of the endoscope 20 (such as a doctor) performs an operation to bring the tip 21a of the endoscope 20 close to the specific object 101a. By performing this operation, the usage state of the endoscope 20 changes from the screening state to the approach state.

[0112] Figure 7 This is a schematic diagram showing the state of the endoscope 20 when it is in the second observation state (approach state). For example... Figure 7 As shown, when the endoscope 20 is in the approach state, the bending portion 21b of the insertion portion 21 of the endoscope 20 is bent by a predetermined amount in a predetermined direction. At this time, the bending direction of the bending portion 21b is the direction in which the front end face of the front end portion 21a of the endoscope 20 is positioned opposite the detected specific object 101a.

[0113] That is, at this time, the insertion part 21 of the endoscope 20 is configured with its front end facing the object 101a and is opposite to the object 101a, and the central axis O (optical axis O of the camera optical system 26a) of the insertion part 21 extends from the front end facing the object 101a.

[0114] In this state, the user (doctor, etc.) performs an operation to gradually bring the tip 21a of the endoscope 20 closer to the specific object 101a. In this way, during an endoscopic examination, the state change of the endoscope 20 from the screening state to the approach state can be determined by detecting the time-varying changes in the output signals from the camera unit 26 acquired sequentially in a time sequence.

[0115] Next, consider the case where the endoscope 20 transitions from a proximity state to a detailed observation state. In the proximity state during an endoscopic examination, after the tip 21a of the endoscope 20 has sufficiently approached the specific object 101a, which is the target of observation, to a predetermined extremely close distance (e.g., the limit distance at which a focusing state cannot be achieved if it approaches further), a detailed observation of the specific object 101a is performed while maintaining the distance between the tip surface of the tip 21a and the specific object 101a. By performing this operation, the use state of the endoscope 20 transitions from the proximity state to the detailed observation state.

[0116] Figure 8 This is a schematic diagram showing the state of the endoscope 20 in the third observation state (detailed observation state). Figure 8As shown, when the endoscope 20 is in a detailed observation state, the curved portion 21b of the endoscope 20 is maintained in a state where it is bent in a predetermined direction by a predetermined amount, and the front end face of the front end portion 21a is positioned close to the specific object 101a to be observed. Then, the user (doctor, etc.) observes the specific object 101a from various angles while pointing the front end portion 21a of the endoscope 20 toward the specific object 101a.

[0117] Previously, when performing endoscopic examinations using medical or diagnostic equipment such as endoscopes, it was essential to maintain a reliable observation environment to consistently display good images. Therefore, appropriate focus control of the camera unit was required based on the state of the endoscope (the three states mentioned above).

[0118] Here, for example, the appropriate focus control corresponding to the usage state of the endoscope 20 is as follows.

[0119] Typically, when an endoscope is in screening mode, observation is performed while moving the endoscope inside the lumen of an organ or similar structure. In this case, the endoscope is actually moving, but it can be said that the object, such as the inner wall of the organ or similar structure, is moving relative to the endoscope. Thus, when focusing on an object moving relative to the endoscope (camera unit) each time, the ranging operation and focusing lens movement become cumbersome, potentially resulting in inaccurate and stable images. Therefore, during screening, it is preferable to use so-called deep focus control, which can fix the focusing lens in a predetermined position and ensure focus within a predetermined imaging distance.

[0120] That is, for example, observation in the screening state is performed when the distance between the camera unit and the object being observed is relatively large (e.g., more than about 4-5 millimeters (mm) and roughly a few centimeters apart). In this case, sufficient observation can be performed within the depth of field of the camera optical system. Therefore, the focus control in the camera unit at this time is what is called panoramic focus control.

[0121] Furthermore, panoramic focus control in this case is, for example, the following focus control: for an object within a preset distance range (e.g., approximately 1-3 cm from the front surface of the lens), the position of the camera optical system is set to achieve a roughly focused state, and this position is maintained. The position of the camera optical system (focusing lens) set during this panoramic focus control is referred to as the first position or first focus position. Here, a position slightly farther than the first position is defined as the first' position.

[0122] Furthermore, the panoramic focus control performed here can be a focus control that sets a specified focus lens position based on fine focus control according to distance, or a focus control that selects any one of several pre-determined position settings. Alternatively, it can be a two-level switching control that simply changes from panoramic focus state to other pre-determined focus lens positions (close-range corresponding focus positions).

[0123] Furthermore, when the endoscope is approaching, it is generally desirable to continuously adjust the focus on the object being photographed or observed, ensuring that the image of the object remains in focus and that the surrounding area, including the object, is clearly visible. Therefore, during the approach, it is preferable to use autofocus control (AF control) to keep the object and its surrounding area in focus at all times.

[0124] Typically, when approaching, the camera is taken from a slightly farther position (e.g., within a range of about 3 to 4 millimeters) than from an extremely close position (described later; for example, about 1.5 to 2 millimeters). In this case, a deeper depth of field can be ensured than in the case of an extremely close position. Therefore, if a deeper depth of field is used, it is easy to ensure focus within a specified range even when there is a distance difference in the front-to-back direction of the object. In this way, by taking into account the object and its surrounding parts to perform fine focus adjustment in autofocus control, a wide area including the object can be observed from above as if in focus.

[0125] Here, as an example of autofocus control that takes into account the object and its surrounding parts, it is conceived to set the focus at a predetermined position between the main part of the object being observed and the peripheral part of the object near its front side.

[0126] Furthermore, when approaching, continuous observation is necessary to avoid missing the target object. During this approach, the endoscope's tip (camera unit) is gradually moved closer to the target object for magnified observation. Therefore, if panoramic focus control is maintained during the approach, maintaining focus becomes increasingly difficult. For this reason, it is preferable to perform autofocus control during the approach.

[0127] On the other hand, in order to display the observed object as large as possible, detailed observation is often conducted with the front surface of the imaging unit (the front surface of the imaging optical system) close to the surface of the observed object (such as a tumor or other specific object) (for example, within a distance of about 1.5 to 2 millimeters). In this situation, the depth of field becomes extremely shallow. Consequently, in such a state, it is often difficult to maintain the imaging unit stably. Under these conditions, if imaging is performed using autofocus control, the acquired image of the object becomes unstable.

[0128] On the other hand, it is known that during such close-up shooting (magnified shooting), the shooting distance between the camera unit 26 and the observed object is approximately constant in the central and peripheral regions of the image. Therefore, the focus control at this time is preferably set to extremely close-priority focus control suitable for extremely close-up shooting (or extremely close observation).

[0129] Here, the extremely close priority focus control is, for example, the following focus control: when the camera unit 26 is brought close to an object located in the central area of ​​the endoscope image, the focusing lens is fixed and maintained at the closest position that the camera unit's camera optical system can ensure the focusing state of the object.

[0130] Specifically, the control is as follows: the focusing lens is moved along the optical axis and maintained in a predetermined position to achieve focus on a predetermined object located at an extremely close distance of approximately 1.5 to 2 millimeters (mm) from the front surface of the lens. In this case, the position of the imaging optical system (focusing lens) set during the extremely close-priority focus control is referred to as the second position or second focus position. Furthermore, here, a position slightly closer than the second position is defined as the second' position.

[0131] Under normal circumstances, the closer the camera distance, the shallower the depth of field. In such cases, for example, when the camera unit swings back and forth relative to the object, if focus control is performed every time, it is impossible to obtain an image with a stable focus. This focus control involves moving the focusing lens.

[0132] Therefore, as described above, in detailed observation mode, near-priority focus control is performed to fix the position of the focusing lens. This ensures a focused state within a specified imaging distance and allows for the acquisition of a stable image.

[0133] In addition, as another means of close-priority focus control, it can also be focus control that performs precise focus adjustment within a very close distance range (e.g., within a shooting distance of 1.5 to 3 millimeters (mm)). In this case, focus control can be used to ensure accurate focus by performing extremely fine and rapid movement control of the focusing lens.

[0134] In addition to these focus controls, in recent years, there have been various other types of autofocus control, such as those that track a specified object detected within the camera frame, those that prioritize a distant or nearby object within the camera frame, those that automatically focus on objects within a region of the camera frame arbitrarily specified by the user, and those that consider the depth of field of the camera optical system to ensure good focus within a specified distance in front of the camera unit. The focus control unit 16 can appropriately select or combine these various autofocus control methods.

[0135] The following describes the function of the endoscope system 1, which includes the focusing control device of this embodiment. Figure 9 This is a flowchart illustrating the function of an endoscope system incorporating a focusing control device according to an embodiment of the present invention. Figure 9 The flowchart shown illustrates the process sequence of actions when using endoscope system 1 for endoscopic examination.

[0136] First, the endoscope system 1 is activated and ready to perform endoscopic examinations. At this time, the endoscope 20 in the endoscope system 1 is inserted into the organ of the subject (patient, etc.) to be examined. Furthermore, the focus control setting of the endoscope 20 in its initial state is, for example, set to panoramic focus control.

[0137] When the endoscope system 1 of this embodiment is in such a state, Figure 9 In step S1, the image processing unit 11 of the processor 10 begins acquiring the output signal (image data; image information and phase difference information) from the camera unit 26. Simultaneously, based on the acquired output signal (image data), various prescribed image processing operations are performed on each frame. Specifically, for example, the various prescribed image processing operations may be performed every 2 frames or every 4 frames.

[0138] In this case, image processing performed on a per-frame basis does not necessarily require treating all frames acquired according to the frame rate of the imaging element 26b as objects. For example, for the acquired image data, frame data spaced several frames apart can also be processed as objects.

[0139] Next, in step S2, the processor 10 executes endoscope state determination processing to determine the current state of the endoscope. Here, Figure 10 This is a flowchart showing the detailed process of determining the state of the endoscope.

[0140] First of all, Figure 10 In step S11, the processor 10 confirms the following conditions based on the image data processed by the image processing unit 11 according to the output signal from the camera unit 26. Furthermore, in the following description, numbers such as first, second, etc., are used for explanation, but these numbers do not indicate the order of processing; they are simply beginning marks when listing the conditions.

[0141] First, confirm whether the light emission of the lighting source is sufficient. The light emission of the lighting source can be confirmed by detecting the light emission control signal of the light source control unit 17 to the light source 41.

[0142] Typically, when using a camera unit to capture images in dark places such as inside the lumens of organs, illumination light is used to light the subject. In this case, the greater the distance to the subject, the greater the amount of light emitted from the illumination source needs to be in order to capture the subject clearly.

[0143] Figure 11 This is a graph showing the relationship between the light emission L of the light source and the distance D in the endoscope system of this embodiment. For example, when using... Figure 11 In the case of the illustrated lighting source, when emitting light at a light emission intensity L that is approximately 30% of the light emission intensity at maximum emission, it can adequately illuminate an object located at a distance of D=5mm. Here, the distance setting of D=5mm corresponds to the first distance mentioned above.

[0144] Furthermore, in the aforementioned step S11, as a confirmation of whether the light emission level is high, for example, if the light emission level L is 30% or higher, it can be inferred that light source control is performed to search for a first region exceeding the first distance (approximately 4-5 mm). Therefore, it can be determined that the endoscope 20 is in a first observation state (screening state) at this time. And, if the light emission level L is less than 30%, it can be determined that the endoscope 20 is in a state other than the first observation state.

[0145] Furthermore, in endoscopic examinations, observation is sometimes performed after staining biological tissues within the lumens of organs or other structures with various pigment solutions. In such cases, for example, when staining with blue pigment solutions, it is necessary to increase the luminescence intensity compared to normal (unstained) conditions. Therefore, to confirm the need for high luminescence intensity illumination accompanying such procedures, the threshold is adjusted, for example, to 50% of the luminescence intensity L. Regarding this threshold adjustment, the overall hue of the image is determined based on the endoscopic image data, and the threshold for confirming the luminescence intensity L is adjusted accordingly.

[0146] Second, confirm the presence of a lumen region within the image. Furthermore, if the presence of a lumen region is confirmed, it is also determined whether this lumen region is located approximately in the central region. Here, the presence of a lumen region within the image can be confirmed by using image data obtained from the endoscopic image, such as brightness distribution information within the image and tonal image detection of black areas.

[0147] Typically, when the endoscope is in the first observation state (screening state), such as when using... Figure 5 As explained, the insertion part 21 is arranged in a generally straight line along the interior of the lumen of organs, etc. The endoscopic images obtained at this time are, for example, as shown in... Figure 12 As shown.

[0148] Figure 12 This shows that the endoscope in the endoscope system of this embodiment is in the first observation state (screening state); Figure 5 An example of an endoscopic image in a certain state (e.g., a specific endoscopic condition). Figure 12 In the accompanying drawing, reference numeral 101 is a display frame indicating the display area for the endoscopic image. The endoscopic image is displayed within the area of ​​this display frame 101. Figure 12 Reference numeral 110a in the accompanying drawings indicates a dark area in an endoscopic image. Additionally, this... Figure 12 The reference numeral 110b in the attached figure indicates the bright area 110b displayed around the dark area 110a.

[0149] Here, the dark region 110a in the endoscopic image is an area where the image is not clear because the illumination light does not reach it sufficiently. In other words, the dark region 110a can also be described as an underexposed area. On the other hand, the bright region 110b in the endoscopic image is an area that is fully illuminated by the illumination light and has a high brightness distribution. Therefore, it is conceivable that the aforementioned lumen region mainly corresponds to the dark region 110a.

[0150] In typical endoscopic images taken during an endoscopic examination when the endoscope is in the first observation state (screening state), such as... Figure 12As shown, it is observed that there is a tendency for many dark regions 110a to exist in the roughly central region.

[0151] In this case, whether it is a lumen region is determined by assessing the size of the area occupied by the dark region 110a within the overall endoscopic image. Specifically, for example, as... Figure 12 As shown, the display frame 101 for the endoscopic image is divided into multiple regions. Figure 12 The example shows a division into 8 vertical regions × 8 horizontal regions. Additionally, in... Figure 12 The numbers marked with reference numeral N outside the display box 101 indicate the area number. This area number is used for ease of explanation.

[0152] Here, for example, the area within the thick frame line (reference numeral 101x) surrounded by four areas from the vertical N3 area to the vertical N6 area and four areas from the horizontal N3 area to the horizontal N6 area is defined as the central area 101x. Additionally, for example, the area within the thick frame line located at the four corners of the display frame 101 and surrounded by three areas in each direction (reference numeral 101y) is defined as the corner area 101y.

[0153] The following verifications are made for each of the defined regions 101x and 101y: First, the minimum brightness (brightness of the darkest part within the region) of the central region 101x is less than 50% of the maximum average brightness of each corner region 101y. Second, the area of ​​the dark region 110a within the central region 101x is more than 20% of the area of ​​the entire endoscopic image. Third, the dark region 110a of the central region 101x exists continuously.

[0154] If all of the above-mentioned first to third conditions are met, the endoscope 20 can be determined to be in the first observation state (screening state). Furthermore, if the above-mentioned first to third conditions are not met, the endoscope 20 can be determined to be in a state other than the first observation state.

[0155] Thus, in the above step S11, it is confirmed whether a lumen region exists in the image and, if it exists, whether the lumen region is located in the approximate central region.

[0156] Third, confirm whether fog or other similar phenomena have occurred within the image. The presence of fog or similar phenomena within the image can be confirmed using information such as the brightness distribution within the image obtained from image data based on the endoscopic image.

[0157] During endoscopic examinations, water is sometimes injected to clean the anterior surface of the viewing window or to remove mucus, intestinal residue, etc., from the area being observed. When this injection is performed, the saline solution or other liquid is forcefully ejected from the tip of the endoscope. This can sometimes cause fine liquid particles to spread in a mist-like manner at the tip of the endoscope 20, creating what is known as fog. Furthermore, during endoscopic examinations, procedures such as cauterization are sometimes performed. In these cases, surgical fumes are generated.

[0158] Thus, fog and surgical fumes generated during endoscopic examinations become obstacles to the field of view of the endoscope 20. Furthermore, when autofocus control is performed, sometimes these fog and surgical fumes generated near the object to be observed are focused.

[0159] Furthermore, the camera distance is usually quite far when water delivery or treatment is being carried out. That is, water delivery or treatment is rarely carried out when detailed observation is being conducted (when the camera distance is extremely close) or when close proximity is required for video observation (when the camera distance is close).

[0160] Therefore, in this embodiment, if fog or the like is detected, the endoscope 20 is determined to be in a first observation state (screening state). Here, the detection of fog or the like is performed as follows. Figure 13 This is a conceptual diagram of an endoscopic image illustrating a fogging situation during an endoscopic examination using the endoscopic system of this embodiment. Figure 13 In the attached figures, reference numerals 101, 110a, 110b and... Figure 12 Same. Figure 13 In the attached figure, reference numeral 111 indicates fog.

[0161] Typically, in the event of fog formation, the illumination light emitted from the tip of the endoscope 20 is reflected by the liquid particles forming the fog, thereby discretely displaying multiple high-brightness fine pixels within the endoscopic image. Examples of discrete multiple high-brightness pixels caused by such fog include... Figure 13 As shown by reference numeral 111 in the attached figure, it is produced discontinuously.

[0162] Therefore, firstly, the presence of a predetermined number or more high-brightness pixels within the endoscopic image is detected based on brightness distribution information. In this case, a high-brightness pixel is determined, for example, if the pixel signal output value is 50% or more relative to the maximum value.

[0163] In addition to the above-described determination, in this case, for example, the determination can also be made by adding a condition such as the average brightness of the area other than the pixel of interest being below a specified level (e.g., below 50%) within a defined area containing the pixel of interest (e.g., a 5-pixel x 5-pixel area). This determination assumes that the background of the pixel of interest is, for example, a dark area.

[0164] Next, it is determined whether the multiple high-brightness pixels detected in this way are caused by fog. To do this, it is confirmed whether the high-brightness pixels are consecutively adjacent. Here, in the case where the detected high-brightness pixels are caused by fog, they appear discontinuously. On the other hand, for example, high-brightness pixels are sometimes detected consecutively in the outline of an object, the ridges of folds inside an organ, etc.

[0165] Here, "pixels are consecutively adjacent" means, for example, the situation shown below. Figure 14 This is a schematic diagram illustrating the concept of consecutive adjacent pixels in an endoscopic image. For example... Figure 14 As shown, consider a straight line extending in a total of eight directions, centered on the pixel of interest indicated by the reference numeral P0, in two horizontal directions H1 and H2, two vertical directions V1 and V2, two tilted directions DR1 and DL1 obtained by tilting the two horizontal directions H1 and H2 counterclockwise by an angle of 45 degrees, and two tilted directions DL2 and DR2 obtained by tilting the two vertical directions V1 and V2 counterclockwise by an angle of 45 degrees.

[0166] In addition, Figure 14 middle, The region consisting of multiple pixels arranged along the horizontal direction H1 is indicated by the reference numeral P1. The region consisting of multiple pixels arranged along the horizontal direction H2 is indicated by the reference numeral P2. The region consisting of multiple pixels arranged along the vertical direction V1 is indicated by the reference numeral P3 in the attached figure. The region consisting of multiple pixels arranged along the vertical direction V2 is indicated by the reference numeral P4 in the attached figure. The region consisting of multiple pixels arranged along the right diagonal line DR1 is indicated by the reference numeral P5 in the attached diagram. The region consisting of multiple pixels arranged along the right diagonal line DL1 is indicated by the reference numeral P6 in the attached diagram. The region consisting of multiple pixels arranged along the left diagonal line DL2 is indicated by the reference numeral P7 in the attached diagram. The region consisting of multiple pixels arranged along the left diagonal direction DR2 is indicated by the reference numeral P8.

[0167] In this case, the average brightness of each pixel region P1 to P8 on each straight line H1, H2, V1, V2, DR1, DR2, DL1, DL2 extending in eight directions centered on the pixel of interest P0 is calculated. Then, if there is at least one region among these regions with an average brightness higher than that of the pixel of interest P0, it is determined that there are consecutive adjacent high-brightness pixels in the endoscopic image. That is, in this case, it is determined that there is no fog. Furthermore, here, if it is determined that fog has occurred, it can be determined that the endoscope 20 is in a first observation state (screening state).

[0168] Fourth, confirm whether there are large (huge) bright areas (also known as halo areas) in the endoscopic image. The existence of large bright areas in the image can be confirmed by obtaining brightness distribution information within the image based on the image data of the endoscopic image.

[0169] Here, the determination of high-brightness pixels is similar to that in fog detection described above: a pixel is determined to be a high-brightness pixel if its output value is 50% or more above its maximum value. Furthermore, if the area formed by such high-brightness pixels occupies 20% or more of the area of ​​the entire endoscopic image and this high-brightness pixel area exists continuously, it is determined that a large area of ​​brightness exists.

[0170] In this case, the shape of the detected large area of ​​brightness is also confirmed. For example, it is confirmed that the detected large area of ​​brightness is not approximately circular or approximately elliptical.

[0171] Figure 15 This is a conceptual diagram illustrating a situation where a large area of ​​brightness exists within the endoscopic image during an endoscopic examination using the endoscopic system of this embodiment. Figure 15 In the figure, reference numeral 101 is a display frame indicating the display area of ​​the endoscopic image. The endoscopic image is displayed within the area of ​​this display frame 101. Reference numeral 101a indicates a specific object such as a tumor. Reference numeral 101b indicates the inner wall of an organ, etc.

[0172] exist Figure 15 In the attached figures, reference numeral 112 indicates surgical fumes, and reference numeral 113 indicates treatment instruments, etc. Figure 15 The treatment instruments 113 shown include, for example, a hemostatic clip.

[0173] Typically, the shapes of specific objects observed during endoscopic examinations, such as tumors, are mostly roughly circular or roughly elliptical. On the other hand, surgical instruments are mostly roughly rectangular. Surgical fumes are mostly cloud-shaped or irregular in shape. Therefore, when a large area of ​​brightness is detected, if it is confirmed that its shape is not roughly circular or roughly elliptical, it can be determined that the endoscope 20 is in the first observation state (screening state).

[0174] Furthermore, the first to fourth determination results mentioned above exhibit a time lag. For example, if the virus is detected continuously for 5 consecutive frames, the determination is true; if it is detected only for less than 5 consecutive frames, the determination is false.

[0175] Furthermore, the decision threshold also exhibits hysteresis. In this case, the case where the decision is true is recorded, but the threshold is set more strictly when the decision is false. For example, the case where a large area of ​​brightness (halo area) exists (true) is set as false, or the case where the area of ​​a high-brightness pixel region is less than 15% of the area of ​​the entire endoscopic image is set as false.

[0176] In the above Figure 10 In step S11, if any one of the first to fourth conditions is met (if any one of them is "yes"), the process proceeds to step S12.

[0177] In step S12, the processor 10 determines that the current state of the endoscope 20 is the first observation state (screening state). Afterwards, it ends the series of processes and returns to the original processing (entering...). Figure 9 (Processing step S3).

[0178] On the other hand, Figure 10 In step S11, if none of the first to fourth conditions are met (if all are "N"), proceed to step S13.

[0179] In step S13, the processor 10 confirms whether the endoscope state determination result for the previous frame is a first observation state. Here, if the previous frame determines it is a first observation state, the process proceeds to the next step S14. And, if the previous frame determines it is not a first observation state, the process proceeds to step S20.

[0180] In step S14, the processor 10 confirms whether the vector change of the current frame relative to the previous frame is significant. Here, vector change refers to the situation shown below. Figure 16 This is a conceptual diagram illustrating the situation where vector changes occur within the endoscopic image during endoscopic examination using the endoscopic system of this embodiment.

[0181] exist Figure 16 In the figure, reference numeral 101 is the display frame of the endoscopic image. Reference numerals 101a (1) and 101a (2) denote specific objects. Among them, reference numeral 101a (1) shown by solid lines indicates, for example, the state of the specific object at any moment when the endoscope 20 is in the first observation state or the second observation state. Reference numeral 101a (2) shown by dashed lines indicates, for example, the state in which the specific object 101a (1) is displayed more prominently after a predetermined time has elapsed after the endoscope 20 has been transferred to the second observation state. In addition, reference numeral 101b denotes the inner wall of organs, etc.

[0182] Here, for example, in the second observation state, consider the case where the endoscope 20 approaches or moves away from the detected specific object 101a. For example, when the endoscope approaches the object, it begins as follows: Figure 16 The attached reference numeral 101a (1) shows a specific object that appears small as the endoscope 20 approaches, such as Figure 16 The figure shown by reference numeral 101a (2) appears to be large. In this case, if viewed in terms of image data obtained in a time series for each frame, the specific object 101a (1) gradually increases in size in the direction indicated by the outward arrow in the reference numeral Ex.

[0183] On the other hand, when the endoscope 20 is far away from the specific object 101a, it begins as follows Figure 16 The attached reference numeral 101a (2) shows a specific object that appears larger as the endoscope 20 moves away, such as Figure 16 The reference numeral 101a (1) appears smaller. In this case, if viewed in terms of image data obtained in a time series for each frame, the specific object 101a (2) gradually decreases in size in the direction indicated by the inward arrow in the reference numeral Ex. The direction and length of the arrow Ex in these cases are referred to as vector changes. Moreover, the change in distance information in this case changes from large (small) to small (large) (from far (near) to near (far)). That is, the distance in the depth direction (front and back direction) gradually changes towards a smaller or larger direction.

[0184] Typically, during endoscopic examinations, the vector change in the depth direction of a specific object relative to the endoscope is observed to be significant in the second observation state. On the other hand, it is known that in the first or third observation states, sometimes the vector change in the depth direction is small, but the vector changes in the vertical and horizontal tilt directions (such as horizontal movement within the XY plane) are large. In this case, the horizontal movement of the object actually assumes movement of the endoscope relative to the object, rather than movement of the object itself. Therefore, the description of object movement refers to the inclusion of relative movement in the relationship between the endoscope and the object.

[0185] Therefore, in Figure 10 In step S14, the vector change of a specific object is confirmed. If the vector change in the depth direction is large, the process proceeds to step S15. Otherwise, if the vector change in the depth direction is small, the process proceeds to step S12.

[0186] In this case, for example, if the movement is relative to the XY plane of the image detected by the camera unit 26, i.e., a movement in the horizontal, vertical, or tilting direction (up, down, left, or right tilting directions) rather than the depth direction, consider the case where the observation of the object being observed is canceled and the endoscopic field of view swings laterally. Therefore, in such cases, even if the vector change is large, the process proceeds to step S12.

[0187] Alternatively, it can be handled separately from such control, for example, in the processing of step S14, detecting a vector change in the horizontal direction for the camera unit 26, and skipping the process if the vector change is large. Figure 10 The subsequent processing steps are set as the second observation state determination for AF control.

[0188] exist Figure 10 In step S15, the processor 10 confirms whether there are multiple minute bright spots in the endoscopic image. In this case, the multiple minute bright spots in the endoscopic image are envisioned as follows. Figure 17 This is a conceptual diagram illustrating a situation where multiple tiny bright spots are generated in the endoscopic image during an endoscopic examination using the endoscopic system of this embodiment.

[0189] exist Figure 17 In the attached figures, reference numerals 101, 110a, 110b and... Figure 12 , Figure 13 Same. Additionally, reference numeral 114 indicates multiple fine bright spots generated within the endoscopic image. Here, a fine bright spot is, for example, an area of ​​high-brightness pixels that has a brightness greater than that described above. Figure 13 The area shown is slightly larger than that of tiny bright spots (high-brightness pixels) at the pixel unit level caused by fog, etc. Specifically, it is defined as a case where there is one or less high-brightness pixel area with an area of ​​5 or more adjacent pixels, and there are 5 or more high-brightness pixel areas with an area smaller than 5 adjacent pixels.

[0190] Typically, the appearance of multiple tiny bright spots, such as in the first observation state, is caused by the illumination light from the endoscope 20 being reflected by the inner wall of an organ or similar object located at a certain distance.

[0191] Therefore, in Figure 10If multiple minor bright spots are identified during step S15, proceed to step S12. Otherwise, if no multiple minor bright spots are identified, proceed to step S16.

[0192] In step S16, the processor 10 performs a focus position detection process. Here, the focus position is the distance from the object in the endoscopic image to the position where it is in focus (camera position), i.e., the camera distance. This camera distance information can be obtained through the phase difference information contained in the image data.

[0193] In step S17, the processor 10 confirms whether the detection result (camera distance) of the processing in step S16 is located on the far side. Here, the case where the camera distance is located on the far side refers to the case where it is located on the side close to the first distance (about 4~5mm) as described above.

[0194] In step S17, if the detected camera distance is on the far side, the process proceeds to step S12. Otherwise, if the detected camera distance is not on the far side, the process proceeds to step S18.

[0195] Next, in step S18, the processor 10 confirms whether the detection result (camera distance) of the processing in step S16 is located on the near point side. Here, the case where the camera distance is located on the near point side refers to the case where it is located on the side close to the second distance (approximately 1.5~2mm) as described above.

[0196] In step S18, if the detected camera distance is on the near side, the process proceeds to step S23. Otherwise, if the detected camera distance is not on the near side, the process proceeds to step S19.

[0197] In step S19, the processor 10 determines that the current state of the endoscope 20 is the second observation state (approach state). Afterwards, the series of processes ends, and the processor returns to the original processing (entering...). Figure 9 (Processing step S3).

[0198] Furthermore, in step S23, the processor 10 determines that the current state of the endoscope 20 is the third observation state (detailed observation state). Afterwards, the series of processes ends, and the processor returns to the original processing (entering...). Figure 9 (Processing step S3).

[0199] On the other hand, if the previous frame is determined not to be in the first observation state during the processing of step S13, when proceeding to step S20, the processor 10 checks whether the previous frame is determined to be in the third observation state. Here, if the previous frame is determined to be in the third observation state, the process proceeds to the next step S21. Furthermore, if the previous frame is determined not to be in the third observation state, the process proceeds to step S15.

[0200] In step S21, the processor 10 determines whether the vector change of the current frame relative to the previous frame is large. If the vector change is large, the process proceeds to step S22. If the vector change is small, the process proceeds to step S23.

[0201] In step S22, the processor 10 confirms whether there are "three relatively large bright spots" in the endoscopic image in a prescribed manner.

[0202] In the endoscope 20 included in the endoscope system 1 of this embodiment, such as Figure 1 As shown and as described above, three illumination lenses 25b are provided on the front end face of the front end 21a. During endoscopic examination, the illumination light emitted from these multiple (3) illumination lenses 25b toward the front (observation object) is positively reflected on the object, thereby appearing as "three relatively large bright spots" in the endoscopic image.

[0203] Here, the detection of the "three relatively large bright spots" in the endoscopic image can be confirmed by using information such as the brightness distribution within the image obtained from the image data.

[0204] Figure 18 This is a conceptual diagram illustrating a situation where three relatively large bright spots are generated in the endoscopic image during an endoscopic examination using the endoscopic system of this embodiment. Figure 19 yes Figure 18 The rectangle F shows an enlarged view of the area containing at least one bright spot.

[0205] exist Figure 18 In the attached figures, reference numerals 101, 101a, 101b and... Figure 15 Same. Additionally, in Figure 18 In the figure, reference numerals 115a (1~3) and 115b represent bright spots. Reference numeral F represents a box that indicates a region containing at least one of the multiple bright spots 115a (1). Reference numeral Tr represents a triangle formed by connecting three bright spots 115a (1~3).

[0206] Here, the confirmation of whether the size of the bright spot appearing in the endoscopic image is "relatively large" is performed as follows. When an area containing high-brightness pixels is detected in the endoscopic image, focus on one of these areas and divide that area into multiple regions. Specifically, for example... Figure 18 Region F such as Figure 19 As shown, it is divided into 20 horizontal regions × 20 vertical regions, etc. In this case, Figure 19 The numbers shown outside the column of region F represent the region coordinates. For example, the segmented region represented by reference numeral Q1 can be represented as coordinates (7,6) with x-axis 7 and y-axis 6. Reference numerals Q2, Q3, Q4, and Q5 can be represented in the same way.

[0207] like Figure 19 As shown, for multiple regions that have been divided into segments, the ratio of the number of high-brightness pixels to the total number of pixels in each region is calculated. Then, if the number of high-brightness pixels in any segmented region is greater than 50%, that segmented region is determined to contain high-brightness pixels. For example, if... Figure 19 If the high-brightness pixels in segmented region Q1 (7,6) are 40%, the high-brightness pixels in segmented region Q2 (11,6) are 15%, the high-brightness pixels in segmented region Q3 (6,8) are 100%, the high-brightness pixels in segmented region Q4 (4,9) are 40%, and the high-brightness pixels in segmented region Q5 (11,14) are 75%, then it can be determined that segmented regions Q3 and Q5 are regions containing high-brightness pixels.

[0208] Then, for example, if a region is defined as a "relatively large" bright spot, it is a region that is five or more adjacent to a segmented region containing high-brightness pixels. Then, the center coordinates of these five or more adjacent regions are stored. As a specific example, for instance... Figure 19 The regions indicated by reference numerals R1 and R2 in the attached figures are regions that "contain more than 5 adjacent segmented regions of high-brightness pixels" and can be determined as "relatively large". In this case, the center coordinates of region R1 are (7,11) and the center coordinates of region R2 are (11,10). The same confirmation is made for other bright spots 115a (2), (3) and 115b.

[0209] so, Figure 18 , Figure 19 Highlight 115a (1~3) was judged to be "relatively large". On the other hand, this Figure 18 , Figure 19 Highlight 115b was judged not to be “relatively large” (small).

[0210] This confirms whether three "relatively large" bright spots exist in the endoscopic image. If so, it confirms whether these three bright spots are configured in a prescribed manner. Here, as a prescribed manner, consider, for example, the following configuration: a triangle formed by straight lines connecting the approximate center points of the three bright spots. Figure 18 The reference numeral Tr in the attached diagram is arranged to roughly surround the central region of the endoscopic image, and the three bright spots are separated from each other by a certain distance on the plane of the endoscopic image. Specifically, for example, imagine that the length of each side of the triangle (the distance between the centers of the three bright spots) is more than 20% of the length of the short side of the endoscopic image.

[0211] Thus, in the endoscopic images during endoscopic examination, if the three largest bright spots are detected in accordance with the prescribed method, it can be determined that the endoscope 20 is in the third observation state.

[0212] exist Figure 10 In step S22, if three relatively large bright spots are confirmed, the process proceeds to step S23. Otherwise, if no bright spots are confirmed, the process proceeds to step S16.

[0213] Thus, in Figure 10 In the endoscope status determination process, the first observation state is determined in step S12, the second observation state is determined in step S19, and the third observation state is determined in step S23.

[0214] Then, in Figure 9 In step S3, the processor 10 confirms the processing in step S2 above. Figure 10 The endoscope status determination process checks whether the result is the first observation state. If it is confirmed to be the first observation state, the process proceeds to step S7.

[0215] Then, in step S7, the processor 10 sets the position of the focusing lens 26ab to a first position (first focusing position) via the focusing control unit 16, and performs control to maintain this position (panoramic focusing control setting). Furthermore, since the position of the focusing lens 26ab has already been set to the first position, control to maintain this position is performed. Afterwards, the process proceeds to step S6.

[0216] On the other hand, in the process of step S3 described above, if it is not the first observation state, the process proceeds to step S4. Then, in step S4, the processor 10 confirms the process of step S2 described above ( Figure 10 The endoscope status determination process checks whether the result is a third observation state. If it is confirmed to be a third observation state, the process proceeds to step S8.

[0217] Next, in step S8, the processor 10 sets the position of the focusing lens 26ab to a second position (second focusing position) via the focusing control unit 16, and maintains this position (extremely close priority focusing control setting). Furthermore, since the position of the focusing lens 26ab has already been set to the second position, this position is maintained. Afterwards, the process proceeds to step S6.

[0218] On the other hand, in the process of step S4 described above, if it is not the third observation state, the process proceeds to step S5. In this case, consider the process of step S2 described above ( Figure 10 The determination result of the endoscope status determination process is the second observation state.

[0219] Therefore, in step S5, the processor 10 performs autofocus control (AF control) targeting a specific object through the focus control unit 16. Then, the process proceeds to step S6.

[0220] In step S6, the processor 10 confirms an instruction to end the ongoing endoscopic examination. Here, the instruction to end the endoscopic examination may be, for example, a predetermined examination end instruction signal generated by a predetermined operating component in the operating component 22c provided in the operating unit 22 of the endoscope 20, such as the processor 10 operating the endoscope system 1. Furthermore, the intention to end the endoscopic examination can also be determined, for example, by confirming the image data of the endoscopic images acquired by the endoscope 20.

[0221] If the inspection end indication is confirmed in step S6, the series of processes ends (end). Otherwise, if the inspection end indication is not confirmed, the process returns to step S1 above, and the subsequent processes are repeated.

[0222] As explained above, the focus control device of this embodiment adjusts the focus by moving the focusing lens 26ab included in the imaging optical system 26a back and forth along the optical axis O according to the output signal from the imaging unit 26. The focus control device includes: an imaging distance detection unit 14b that detects distance information from the front end of the imaging optical system 26a to the object being imaged; and a focus control unit 16 that adjusts the position of the focusing lens 26ab on the optical axis O according to the detection result of the imaging distance detection unit 14b. When the distance information detected by the imaging distance detection unit 14b is within a specified distance (e.g., 3~4mm), the focus control unit 16 performs automatic focus control by adjusting the position of the focusing lens 26ab on the optical axis O according to the change in imaging distance.

[0223] Furthermore, if the distance information detected by the camera distance detection unit 14b deviates from the specified distance (e.g., 3~4mm), the focus control unit 16 performs focus control to keep the focusing lens 26ab in the specified position.

[0224] In this case, for example, if the detected distance information is greater than a first distance (e.g., about 4-5 mm), the focus control unit 16 performs focus control (panoramic focus control) to keep the focusing lens 26ab at a first position (first focus position) or a first' position farther than the first position. Alternatively, if the detected distance information is equal to or smaller than a second distance (e.g., about 1.5-2 mm), focus control is performed to keep the focusing lens 26ab at a second position (second focus position) or a second' position closer to the second position (extremely close priority focus control).

[0225] With this structure, in the endoscope system 1 that includes the focusing control device of this embodiment, when in the first observation state (screening state), panoramic focusing control can be used to observe the entire field of view from above, and the search for specific objects such as tumors that are desired to be observed can be performed easily and quickly.

[0226] Furthermore, when a specific object such as a tumor is detected and the observation state is transitioned from the first observation state to the second observation state (approach state), automatic focus control is executed to maintain a focused state within a specified area containing the detected specific object, ensuring a consistently clear image. Therefore, it is possible to continue observing the desired specific object while simultaneously approaching it easily and smoothly with a good image.

[0227] Furthermore, when transitioning from the second observation state to the third observation state (detailed observation state), the system can approach the closest distance to achieve a focused state for the specific object to be observed, and maintain a stable state while performing magnified observation.

[0228] Furthermore, during endoscopic examinations, in the magnified observation of the third observation state, even if the imaging distance of the endoscope 20 deviates from the second distance due to the shaking of the endoscope user's (doctor, etc.) hand holding the endoscope, the focus control can be fixed in advance.

[0229] Furthermore, in one embodiment described above, the first distance (distance during screening) of the camera distance is specified as, for example, approximately 4 to 5 mm, but is not limited to this. The first distance can be appropriately changed, for example, set to a distance slightly further than approximately 4 to 5 mm. Similarly, the specified distance (distance when approaching) and the second distance (distance for detailed observation) can also be appropriately changed.

[0230] This invention is not limited to the embodiments described above, and various modifications and applications can be implemented without departing from the spirit of the invention. For example, an endoscope was used as an example in this description, but any device that sequentially obtains and judges images (such as an image inspection device) can be used. Furthermore, the invention includes various stages in the above embodiments, and various inventions can be extracted by appropriately combining the multiple constituent elements disclosed. For example, if the problem to be solved by the invention can be solved and the effect of the invention can be obtained even if several constituent elements are deleted from all the constituent elements shown in one of the above embodiments, the structure with the constituent elements deleted can be extracted as an invention. Furthermore, the constituent elements of different embodiments can also be appropriately combined. This invention is not limited to its specific embodiments except as defined by the appended claims.

Claims

1. A focus control device, which adjusts the focus by moving a focusing lens included in a camera optical system back and forth along the optical axis according to an output signal from a camera unit, characterized in that, The focusing control device is equipped with a processor. The processor acquires distance information from the front end of the camera optical system to the object being photographed. If the distance information is within a specified distance range, it executes autofocus control, which adjusts the position of the focusing lens on the optical axis according to the change in the camera distance.

2. The focusing control device according to claim 1, characterized in that, If the distance information deviates from the specified distance range, the processor performs focusing control to keep the focusing lens in the specified position.

3. The focusing control device according to claim 2, characterized in that, If the distance information is greater than a first distance, the processor performs focusing control to maintain the focusing lens at a first position or a first' position that is farther than the first position. If the distance information is equal to or less than the second distance, the processor performs focusing control to keep the focusing lens in a second position or a second' position closer to the second position.

4. The focusing control device according to claim 3, characterized in that, The first position is the position where the focusing lens can achieve a focused state on the object being photographed within a predetermined distance range. The second position is the position of the focusing lens corresponding to the shortest distance at which a focused state can be obtained for the object being photographed.

5. The focusing control device according to claim 4, characterized in that, The first distance is 4 mm to 5 mm. The second distance is 1.5 mm to 2 mm.

6. The focusing control device according to claim 1, characterized in that, The camera unit is configured to include: the camera optical system, which includes the focusing lens; and an imaging element that receives the optical image of the object being imaged formed by the camera optical system and performs photoelectric conversion. The processor also obtains distance distribution information within the image based on the phase difference information contained in the output signal from the camera unit, and detects the distance information based on the distance distribution information.

7. The focusing control device according to claim 3, characterized in that, The processor also performs the following processing: Detecting specific objects, The vector change information of the object being photographed is obtained from multiple images continuously acquired from the camera unit in a time sequence. Based on the vector change information, it is determined whether the distance information changes from a state greater than the first distance to the specified distance range, or whether it changes from a state equal to or smaller than the second distance to the specified distance range.

8. The focusing control device according to claim 7, characterized in that, When the processor determines, based on the vector change information, that the change in the distance information is small and the object being photographed is moving relatively in the XY direction, it performs autofocus control.

9. The focusing control device according to claim 1, characterized in that, The processor also detects the amount of illumination light emanating from the illumination unit toward the object being photographed, based on emitted light quantity data, which is emitted light quantity data of a light source device controlled according to an output signal from the camera unit. The processor detects the distance information based on the amount of illumination.

10. The focusing control device according to claim 1, characterized in that, The processor also obtains brightness distribution information within the image based on the output signal from the camera unit, and detects the distance information based on the brightness distribution information.

11. The focusing control device according to claim 10, characterized in that, The processor detects three larger bright spots based on the brightness distribution information. When the three larger bright spots are detected, the processor performs the following focusing control: when the three larger bright spots are formed to be separated from each other by a predetermined distance and to form a triangle with the central coordinates of each of the three larger bright spots as vertices surrounding the central region of the endoscope image, the focusing lens is maintained at the second position or a second' position that is closer to the second position.

12. The focusing control device according to claim 10, characterized in that, The processor detects multiple discrete bright spots based on the brightness distribution information. When multiple discrete bright spots are detected, the processor performs focusing control to keep the focusing lens at the first position or a first lens position that is farther than the first position.

13. A focus control method, wherein, based on an output signal from a camera unit, a focusing lens included in a camera optical system is moved back and forth along the optical axis to adjust the focus, characterized in that, The system detects distance information from the front end of the imaging optical system to the object being imaged. If the distance information is within a specified range, it performs autofocus control, which adjusts the position of the focusing lens on the optical axis according to changes in the imaging distance. If the distance information is greater than a first distance, focus control is performed to maintain the focusing lens at a first position or a first' position that is farther than the first position. If the distance information is equal to or less than the second distance, focus control is performed to keep the focusing lens in a second position or a second' position that is closer to the second position.

14. An endoscope system comprising an endoscope and a processor, the endoscope including a camera unit and a focus adjustment mechanism, the camera unit including a camera optical system for forming an optical image of an object and an image sensor for photoelectric conversion of the optical image formed by the camera optical system to obtain an image signal, the focus adjustment mechanism causing a portion of the optical lens included in the camera optical system to move back and forth in a direction along the optical axis; The processor performs the following processing: Detect the distance information from the front end of the camera optical system to the object being photographed. If the distance information is within a specified distance range, autofocus control is executed, which adjusts the position of the focusing lens on the optical axis according to the change in the shooting distance. If the distance information deviates from the specified distance range. When the distance information is greater than a first distance, focus control is performed to maintain the focusing lens at a first position or a first' position that is farther than the first position. When the distance information is equal to or less than the second distance, focus control is performed to keep the focusing lens in a second position or a second' position that is closer to the second position.

15. The endoscope system according to claim 14, characterized in that, The second distance is the distance at which diagnostic findings can be obtained by magnifying and observing the photographed object.

Images