Endoscope system and method of operating the same

Abstract

To provide an endoscope system and a method of operating the same that enable imaging with optimal focus and brightness even though a region of interest is in a range outside a user's viewpoint.SOLUTION: An endoscope system according to the invention consecutively images, with an endoscope, a subject by switching between a reference focus position where a focus position is a preset position and a moved focus position where a position of a focus lens is moved from the reference focus position, performs image analysis on a focus position movement image captured at the moved focus position to recognize a region of interest from the subject, and displays an endoscope image captured at the reference focus position on a screen and provides notification of the result of image analysis.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an endoscopic system and a method of operation thereof. [Background technology]

[0002] In recent years, diagnoses and other procedures using endoscope systems that acquire images and perform observations have become widespread in the medical field, and doctors generally observe by manually changing the position of the camera to identify areas of interest such as lesions. Doctors strive to constantly detect all areas of interest within organs that should be observed.

[0003] The accuracy of recognizing the area of ​​interest in endoscopic observation is affected by the doctor's experience and skill, as well as the doctor's degree of fatigue. In order to reduce the variation in diagnostic accuracy among doctors, technology is being developed that uses computers to analyze the huge amount of endoscopic data acquired in daily medical practice and extract information useful for diagnosis. For example, a recognition support function in which a computer automatically detects the area of ​​interest during endoscopic observation is expected to prevent doctors from overlooking detection targets.

[0004] Specifically, the endoscope system of Patent Document 1 has a technique for supporting the recognition of characteristic regions based on a predetermined feature amount of an observed image. Also, when a characteristic region is continuously recognized in successively input observed images, the position corresponding to the characteristic region is highlighted. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2017-81976 Summary of the Invention [Problem to be solved by the invention]

[0006] The feature region recognition in Patent Document 1 is premised on the fact that the region of interest is clearly reflected in the image. However, depending on the shape of the subject, the camera may not focus on the image in the range outside the user's viewpoint, such as the edge of the image, resulting in an out-of-focus image or an image with excessive or insufficient brightness. Image analysis of such images tends to reduce the accuracy of analysis. In order to always observe with optimal focus and brightness, it is necessary to rely on the skill of the user, and it is difficult to maintain stable analysis accuracy.

[0007] An object of the present invention is to provide an endoscope system and an operating method thereof that can capture images with optimal focus and brightness even when the region of interest is in a range outside the user's viewpoint. [Means for solving the problem]

[0008] The endoscopic system of the present invention includes an endoscope that images a subject, and a processor. The endoscope continuously images the subject by switching between a reference focus position, which is a preset focus position, and a moving focus position, which is a position to which the focus position is moved from the reference focus position. The processor acquires an endoscopic image captured by the endoscope at the reference focus position and a focus position movement image captured by the endoscope at the moving focus position, performs image analysis on the focus position movement image to recognize an area of ​​interest from the subject, displays the endoscopic image on a screen, and notifies the user of the results of the image analysis.

[0009] It is preferable that the endoscope captures at least one of a near point image whose focus position is closer to the endoscope than the endoscopic image and a far point image whose focus position is farther from the endoscope than the endoscopic image as the focus position movement image.

[0010] It is preferable that the processor measures the amount of deviation of the focus position from the in-focus position of the subject using a phase difference detection imaging element provided in the endoscope, and acquires at least one of near point position information and far point position information depending on the amount of deviation.

[0011] It is preferable that the processor measures the amount of movement of the tip of the endoscope, compares the amount of movement with a predetermined reference range for the amount of movement, and if the amount of movement is greater than the reference range, reduces the frequency of capturing images of focus position movement for the endoscopic image, and if the amount of movement is smaller than the reference range, increases the frequency of capturing images of focus position movement for the endoscopic image.

[0012] When the amount of movement is a value included in the reference range, it is preferable that the endoscope captures the focus position movement image with a fixed frequency for the endoscope image.

[0013] It is preferable that the endoscope captures a recognition focus position image based on the focus position at which the region of interest is recognized at a higher capture frequency than the endoscope image, and the processor performs image analysis on the recognition focus position image.

[0014] The processor preferably displays the captured image of the recognized focus position on a screen.

[0015] It is preferable that the processor performs image analysis on the endoscopic image and determines the recognition focus position image according to the analysis result of the endoscopic image.

[0016] It is preferable that the endoscope repeats an operation of capturing an endoscopic image in a first period spanning a plurality of frames and capturing a focus position movement image in a second period spanning at least one frame.

[0017] It is preferable that the processor controls the exposure amount of illumination light that is emitted from a light source device connected to the endoscope and illuminates the subject, in accordance with the focus position.

[0018] The operation method of the endoscopic system of the present invention includes the steps of continuously capturing images of a subject by switching between a reference focus position, which is a focus position previously set by the endoscope, and a moving focus position, which is a focus position moved from the reference focus position; a step in which a processor device acquires an endoscopic image captured by the endoscope at the reference focus position and a focus position movement image captured by the endoscope at the moving focus position; a step in which the processor device performs image analysis on the focus position movement image to recognize an area of ​​interest from the subject; and a step in which the processor device displays the endoscopic image on a screen and notifies the user of the results of the image analysis. Effect of the Invention

[0019] According to the present invention, even if the area of ​​interest is in a range outside the user's viewpoint, imaging with optimal focus and brightness can be achieved. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of an endoscope system. [Diagram 2] 2 is a block diagram showing the functions of the endoscope system. FIG. [Diagram 3] FIG. 2 is an explanatory diagram of an imaging element having a phase difference detection pixel. [Figure 4] FIG. 13 is an explanatory diagram for capturing an image of a subject having a protruding portion. [Diagram 5] FIG. 13 is an explanatory diagram illustrating imaging of a subject having a recessed portion. [Figure 6] FIG. 2 is an explanatory diagram illustrating imaging of a subject that is a tubular digestive tract. [Figure 7] 1A to 1C are explanatory diagrams of an endoscopic image, a near point image, and a far point image captured in a special observation mode. [Figure 8] FIG. 11 is an explanatory diagram for capturing an image of a subject using a pre-recognition imaging pattern. [Figure 9] 11 is an explanatory diagram illustrating a pre-recognition imaging pattern in which a subject is imaged with an amount of movement larger than a reference range. FIG. [Figure 10] 11 is an explanatory diagram illustrating a pre-recognition imaging pattern in which an object is imaged with an amount of movement smaller than a reference range. FIG. [Figure 11] FIG. 13 is an explanatory diagram for switching to a first post-recognition imaging pattern to capture an image of a subject. [Figure 12] FIG. 13 is an explanatory diagram for switching to a second post-recognition imaging pattern to capture an image of a subject. [Figure 13] 13 is an explanatory diagram showing how to capture an image of a subject by switching to a third post-recognition imaging pattern. FIG. [Figure 14] 4 is a flowchart showing a series of steps for controlling capturing an image with a moving focus position according to the present invention. [Figure 15] FIG. 11 is an explanatory diagram showing how image analysis is performed on an image captured at a focus position set by a user in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] [First embodiment] 1, an endoscope system 10 according to a first embodiment of the present invention includes an endoscope 11, a light source device 12, a processor device 13, a display 14, and a user interface (UI) 15. The endoscope 11 is optically connected to the light source device 12 and electrically connected to the processor device 13. The light source device 12 supplies illumination light to the endoscope 11.

[0022] The endoscope 11 illuminates the subject with illumination light and captures an image of the subject to obtain an endoscopic image. The endoscope 11 has an insertion section 11a that is inserted into a living body (inside a subject) having the subject, and an operation section 11b that is provided at the base end portion of the insertion section 11a. A bending section 11c and a tip section 11d are provided at the tip side of the insertion section 11a. The bending section 11c is bent in a desired direction by operating the operation section 11b. The tip section 11d irradiates the subject with illumination light and receives reflected light from the subject to capture an image of the subject. The operation section 11b is provided with a mode changeover switch 11e that is used for mode changeover operation, and a zoom operation section 11f that is used for zoom operation.

[0023] The processor device 13 is electrically connected to the display 14 and the user interface 15. The processor device 13 receives an image signal from the endoscope 11 and performs various processes based on the image signal. An external recording unit (not shown) for recording images and image information, etc. may be connected to the processor device 13. The display 14 outputs and displays images and image information of a subject that have been processed by the processor device 13. The user interface 15 has a keyboard, a mouse, a touchpad, a microphone, a foot pedal, etc., and has a function of accepting input operations such as function settings.

[0024] As shown in FIG. 2, in the endoscopic system 10, the light source device 12 transmits emitted illumination light to the endoscope 11 via a light guide 29, and the endoscope 11 transmits an image signal captured using the illumination light to the processor device 13, and the processor device 13 generates an image to be displayed on the display 14 and performs image analysis.

[0025] The light source device 12 includes a light source unit 20 that emits one or more illumination lights, and a light emission control unit 22 that generates a drive current (drive signal) that controls the light emission timing and light emission amount of the light source unit 20, and supplies it to the light source unit 20 to emit light.

[0026] The function of the light emission control unit 22 is realized by a light source control processor (not shown) included in the light source device 12, and controls the emitted illumination light by supplying a drive current (drive signal) that controls the light emission timing and the amount of illumination light, etc., to the light source unit 20 in response to receiving a light emission control signal. Note that, when the light source device 12 and the processor unit 13 are electrically connected, the function of the light source control processor may be realized by a central control unit (not shown) instead of the light source control processor.

[0027] Light emitted by the light source unit 20 is incident on a light guide 29. The light guide 29 is built into the endoscope 11 and the universal cord. The universal cord is a cord that connects the endoscope 11 to the light source device 12 and the processor device 13. The light guide 29 propagates light from the light source unit 20 to the tip 11d of the endoscope 11.

[0028] The endoscope 11 is provided with an illumination optical system 30 and an imaging optical system 40. The illumination optical system 30 has an illumination lens 32, and illumination light propagated by a light guide 29 is irradiated onto a subject via the illumination lens 32. The imaging optical system 40 has an objective lens 42, a zoom lens 43, and an imaging sensor 44. Reflected light of the illumination light returning from a subject irradiated with the illumination light is incident on the imaging sensor 44 via the objective lens 42 and the zoom lens 43. As a result, an image of the subject is formed on the imaging sensor 44, which is a color imaging sensor. The image formed on the imaging sensor 44 is transmitted as an image signal to the processor device 13 via a CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 and an A / D (Analog / Digital) converter 48.

[0029] In the processor device 13, programs for each process are incorporated in a program memory (not shown). A central control unit constituted by a processor executes the programs in the program memory to realize the functions of an image signal acquisition unit 50, a DSP (Digital Signal Processor) 51, a noise reduction unit 52, an image processing unit 53, an output control unit 54, and a special observation control unit 60. The special observation control unit 60 includes an analysis image acquisition unit 62, an image analysis unit 64 having a recognizer 65, a recognition result discrimination unit 66, and an imaging pattern change unit 68.

[0030] The zoom lens 43 is a lens group consisting of multiple lenses for enlarging or reducing the size of a subject, and has a focus lens that can move in the direction of the optical axis. In close-up photography, where the subject is observed in close-up and enlarged view, such as when observing the structure of thin blood vessels or performing surgical procedures, the focus lens is moved so that the focus is on the close-up. On the other hand, in long-distance photography, where the subject is observed over a wide range, such as when checking a relatively large lesion, the focus lens is moved so that the focus is on the long-distance. The depth of field, which is the range in which the subject appears in focus, changes when the zoom lens 43, which moves the focus lens, is controlled.

[0031] The focus lens can be moved either manually by the user operating the zoom operation unit 11f or automatically based on a preset setting, and in either case, zoom control is performed on the zoom lens 43 by the imaging control unit 45. In the zoom control, the focus lens is freely moved in the optical axis direction to enlarge or reduce the subject imaged on the imaging sensor 44.

[0032] Since the image sensor 44 has a fixed length of image frame, it is controlled to alternate between an accumulation period and a readout period at regular intervals, for example, every 1 / 60 seconds. That is, image capture is performed at a frame rate of 60 fps (frames per second). The length of the image frame may be adjusted by changing the shutter speed of the electronic shutter.

[0033] The imaging sensor 44 may be a photoelectric conversion element (imaging element) such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The imaging sensor 44 performs, for example, a storage operation of photoelectrically converting received light and storing a signal charge according to the amount of received light for each pixel within an acquisition period of one frame, and a readout operation of reading out the stored signal charge. The signal charge for each pixel read out from the imaging sensor 44 is converted into a voltage signal and input to a CDS / AGC circuit 46. The light source device 12 generates illumination light in accordance with the timing of the storage operation of the imaging sensor 44, and causes the illumination light to enter the light guide 29.

[0034] As shown in FIG. 3, the imaging sensor 44 includes a phase difference detection imaging element 44a, which is an imaging element in which phase difference detection pixels 44b are arranged. The phase difference detection pixels 44b are evenly arranged on the phase difference detection imaging element 44a. By using the phase difference detection pixels 44b, it is possible to identify a focal position (focus position) where the image is focused by phase difference autofocus, which is one of the autofocusing techniques. As an example, in phase difference autofocus, it is preferable to use the phase difference detection pixels 44b arranged mainly on the periphery of the phase difference detection imaging element 44a. It is even more preferable to use the phase difference detection pixels 44b arranged only on the periphery of the phase difference detection imaging element 44a.

[0035] In endoscopic observation, the focus lens is moved to a focal position determined using a focus target to capture an image. The focal position is determined by splitting the light incident on the objective lens 42 into two by the zoom lens 43, which is a lens group, and expressing the difference between the focus target in the two formed images as a shift amount, and determining the position where the image is in focus based on the shift amount. In other words, the position of the focus lens where no difference in the subject occurs or where the shift amount is negligible is the focal position.

[0036] Each pixel of the image sensor 44 is provided with either a B pixel (blue pixel) having a B (blue) color filter, a G pixel (green pixel) having a G (green) color filter, or an R pixel (red pixel) having an R (red) color filter. For example, the image sensor 44 is preferably a color image sensor with a Bayer array in which the ratio of the number of B pixels, G pixels, and R pixels is 1:2:1.

[0037] The B color filter transmits mainly light in the blue band, specifically light with a wavelength band of 380 to 560 nm (blue transmission band). The peak wavelength at which the transmittance is maximum is around 460 to 470 nm. The G color filter transmits mainly light in the green band, specifically light with a wavelength band of 450 to 630 nm (green transmission band). The R color filter transmits mainly light in the red band, specifically light with a wavelength band of 580 to 760 nm (red transmission band).

[0038] Also, a complementary color image sensor equipped with complementary color filters of C (cyan), M (magenta), Y (yellow) and G (green) may be used instead of the primary color image sensor 44. When a complementary color image sensor is used, image signals of four colors, CMYG, are output, and by converting the four color image signals of CMYG into three color image signals of RGB by complementary color-primary color conversion, image signals of each color of RGB similar to those of the image sensor 44 can be obtained.

[0039] The imaging control unit 45 drives and controls the imaging sensor 44 in response to instructions from the mode change switch 11e, the user interface 15 via the processor device 13, and an imaging pattern change unit 68 (described later), and controls imaging in accordance with each observation mode. In controlling imaging, the exposure period is adjusted by setting the shutter speed of an electronic shutter (not shown) of the imaging sensor 44. When switching the imaging pattern, the illumination light emitted is switched via the light emission control unit 22. The imaging control unit 45 also generates a control signal that controls the light emission timing and light emission amount of the light source unit 20 based on the focus position (described later), and transmits a drive current (drive signal) corresponding to the control signal to each light source in the light emission control unit 22 to emit light.

[0040] The imaging control unit 45 has a function of adjusting the focus position, and in addition to a function of adjusting the focus position by operating the zoom operation unit 11f (manual focus) and a function of automatically adjusting the focus position for the subject S (autofocus) by the processor device 13, it has a function of changing the focus position by an arbitrary amount of movement from the focus position determined at the time of imaging. The determined focus position also includes a focus position that has not been changed since the start of imaging.

[0041] The CDS / AGC circuit 46 performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the image sensor 44. The image signal passed through the CDS / AGC circuit 46 is converted into a digital image signal by an A / D converter 48. The digital image signal after A / D conversion is input to the processor device 13.

[0042] The image signal acquisition unit 50 receives an image signal input from the endoscope 11, and transmits the received image signal to the DSP 51. The output control unit 54 transmits to the display 14 an image signal of an image to be displayed, which has been processed by the DSP 51, the noise reduction unit 52, and the image processing unit 53.

[0043] The DSP 51 performs various signal processing such as defect correction, offset processing, gain correction, linear matrix processing, gamma conversion, demosaic processing, and YC conversion processing on the image signal received by the image signal acquisition unit 50. In the defect correction processing, signals from defective pixels of the imaging sensor 44 are corrected. In the offset processing, dark current components are removed from the image signal that has been subjected to the defect correction processing, and an accurate zero level is set. In the gain correction processing, the signal level of each image signal is adjusted by multiplying the image signal of each color after the offset processing by a specific gain. The image signal of each color after the gain correction processing is subjected to linear matrix processing to improve color reproducibility.

[0044] Then, the brightness and saturation of each image signal are adjusted by gamma conversion processing. The image signals after the linear matrix processing are subjected to demosaic processing (also called isotropic processing or synchronization processing), and signals of the missing colors of each pixel are generated by interpolation. Through the demosaic processing, all pixels have signals of each of the RGB colors. The DSP 51 performs YC conversion processing on each image signal after the demosaic processing, and outputs a luminance signal Y and color difference signals Cb and Cr to the noise reduction unit 52.

[0045] The noise reduction unit 52 performs noise reduction processing, for example, by a moving average method, a median filter method, etc., on the image signal that has been subjected to demosaic processing, etc. by the DSP 51. The image signal with the noise reduced is input to the image processing unit 53.

[0046] The image processing unit 53 further performs color conversion processing such as 3×3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing on the input image signal for one frame. Then, various color enhancement processes are performed on the RGB image data that has been color converted. Then, structure enhancement processes such as spatial frequency enhancement are performed on the RGB image data that has been color enhanced. In the image processing, the RGB image data that has been subjected to structure enhancement processing is input to the output control unit 54 as an endoscopic image.

[0047] The output control unit 54 sequentially acquires the endoscopic images from the image processing unit 53 and converts them into video signals that enable full-color display on the display 14. The converted video signals are output and displayed on the display 14. This allows the doctor or other person to observe the subject using still or moving endoscopic images.

[0048] The analysis image acquisition section 62 acquires an image for performing attention area recognition from among the images captured in the special observation mode. The image to be acquired is acquired from the image processing section 53 according to the imaging pattern in the special observation mode.

[0049] The image analysis unit 64 performs image analysis on the image for analysis received from the image for analysis acquisition unit 62, and recognizes the area of ​​interest. To recognize the area of ​​interest, a recognizer 65, which is a trained model optimized for recognizing the area of ​​interest of the imaged part, is used. In addition, in order to perform recognition with higher accuracy, a plurality of recognizers 65 corresponding to each part of the subject may be provided. In this case, the image analysis unit 64 discriminates the imaged part, and inputs the image for which the area of ​​interest is to be recognized to the recognizer 65 corresponding to the part.

[0050] The recognizer 65 is equipped with a CNN (Convolutional neural network), which is a computer algorithm consisting of a neural network that performs machine learning, and recognizes the region of interest captured by the input endoscopic image according to the learning content of the region of interest in the subject that has been previously performed. Recognition of the region of interest includes detection of lesions, differentiation of lesions, identification of organs, measurement of lesions, etc. It is also preferable to calculate the reliability of the recognition result. The recognizer 65 has the function of a trained deep learning model trained with a large amount of data set. The large amount of data set includes images including the region of interest in each observation site and images not including the region of interest, and the region of interest is a lesion or a characteristic structure in each organ.

[0051] The recognizer 65 is trained using a data set that includes many images captured with the imaging site corresponding to endoscopic observation as the subject. The recognizer 65 is used in a state where it has been fully trained on images of the imaging site corresponding to the type of endoscope 11 used. When the endoscope 11 is an upper gastrointestinal endoscope, the imaging sites such as the "esophagus", "stomach", and "duodenum" are the subjects. When the endoscope 11 is a lower gastrointestinal endoscope, the imaging sites such as the "rectum", "large intestine", and "small intestine" are the subjects. Note that the image analysis unit 64 may classify the image sites for which attention area recognition is performed, and multiple recognizers 65 trained for each site may be used according to the classification results.

[0052] The recognition result discrimination unit 66 acquires at least information on the presence or absence of an attention area as the recognition result of the recognizer 65. It also has a function of discriminating whether the attention areas in images subjected to successive recognition processing or images captured within a certain period of time are the same. The discriminated recognition result is transmitted to the imaging pattern change unit 68 together with focus position information. Furthermore, when attention areas are recognized at multiple focus positions, the reliability calculated by the recognizer 65 is used to discriminate the recognition result of the focus position where the image is in focus.

[0053] The imaging pattern change unit 68 issues an instruction to change the imaging pattern in the special observation mode according to the determination result of the recognition result determination unit 66. When it is determined that an attention area is present in the continuous shooting, the imaging pattern is changed based on the recognition focus position image, which is an image in which the attention area is recognized. That is, the imaging patterns in the special observation mode include a pre-recognition imaging pattern before the attention area is recognized, and a post-recognition imaging pattern after the attention area is recognized. The imaging pattern change instruction is transmitted to the imaging control unit 45 to realize the change of the imaging pattern. The imaging pattern may be changed at the timing when it is determined that the attention area is the same consecutively, instead of at one attention area recognition.

[0054] The endoscope system 10 of this embodiment has a function of switching between a normal observation mode in which image analysis is not performed and a special observation mode in which image analysis is performed as desired. In the normal observation mode, an object is imaged and observed based on a focus position set manually or automatically by the endoscope 11.

[0055] In the special observation mode, images are captured by switching between frames of endoscopic images captured at the reference focus position, which is the focus position set in the same way as in the normal observation mode, and frames of images with the focus position moved from the reference focus position. The images with the focus position moved are used for image analysis. Observation in the special observation mode involves capturing images in which the focus position is controlled to be switched on a frame-by-frame basis. There are multiple patterns for capturing normal images and images with the focus position moved in the special observation mode, and each pattern is switched according to user operation or recognition results.

[0056] 4 and 5, in the normal observation mode, even if the user focuses on the focus target T being observed, the region of interest may not be in focus or the exposure may be inappropriate. For example, when the focus position is adjusted to the focus target T, which is a protruding or recessed part of the subject S, it is difficult to focus on a part that is relatively flat.

[0057] 4 shows an example in which the focus is adjusted to focus target T, which is a protruding portion of the subject S. The focus position and exposure amount are adjusted for the protruding portion, which is the focus target T, and the focus is adjusted to focus on the focus target T, and an image is captured with a brightness suitable for observation. On the other hand, a portion of the subject S that is not a protruding portion is farther away from the objective lens 42 than the protruding portion, and therefore may be out of focus and may have insufficient exposure amount.

[0058] 5 shows an example in which the focus is adjusted to focus target T, which is a recessed portion of the subject S. The focus position and exposure amount are adjusted for the recessed portion, which is the focus target T, and the focus is adjusted to focus on the focus target T, and an image is captured with a brightness suitable for observation. On the other hand, a portion of the subject S that is not a recessed portion may be out of focus and may be overexposed because the observation distance to the objective lens 42 is closer than the recessed portion.

[0059] FIG. 6 is a cross-sectional view of a case where an endoscopic observation is performed on a tubular digestive tract, and the distal end 11d of the endoscope 11 is advanced in the insertion direction to observe the tubular digestive tract. The tubular digestive tract is, for example, the esophagus. As shown in FIG. 6, in addition to clear protruding and recessed areas, there may be multiple options for the shape of the subject S and the focus position for photographing. In particular, when a user makes an observation with small movement of the distal end 11d, it is required to appropriately set the focus position and exposure amount, and perform highly accurate attention area recognition processing.

[0060] In the special observation mode, when the focal position is identified by using the amount of deviation as in the normal observation mode, the phase difference detection imaging element 44a is used to measure the amount of deviation of the focus position from the focal position of the subject S, and at least one of near point position information and far point position information is acquired according to the amount of deviation. A near point image is captured using the near point position information, and a far point image is captured using the far point position information. It is determined whether the focal position is a near point or a far point with respect to the tip 11d of the endoscope 11, based on the focus position before the phase difference autofocus was performed.

[0061] The amount of exposure is also controlled according to each mode and pattern, and the turning on or off of each light source and the amount of light emitted when turned on are independently controlled by the light emission control unit 22. In controlling the amount of exposure, it is preferable that the opening and closing of the shutter is controlled in synchronization with the imaging frame.

[0062] In the normal observation mode, the focus position, which is the position of the focus lens, is determined by manual focus or autofocus, and an image is captured to generate a normal image as an image to be displayed. The illumination light is continuously turned on during observation, and the exposure period is adjusted by opening and closing the shutter, etc.

[0063] In the pre-recognition imaging mode of the special observation mode, the subject S is continuously imaged by the endoscope by switching between a reference focus position, which is a preset focus position, and a focus position movement image, which is a position to which the focus position has been moved from the reference focus position, in the same manner as in the normal observation mode. An endoscopic image 70 imaged at the reference focus position and a focus position movement image imaged at the movement focus position are acquired by imaging, and image analysis is performed on the focus position movement image to recognize an area of ​​interest from the subject S, and the endoscopic image 70 is displayed on the screen, and the result of the image analysis is notified. Continuous imaging is performed using an imaging pattern that switches between an endoscopic image and a focus position movement image on a frame-by-frame basis.

[0064] 7, the focus position moving images captured by switching with the endoscopic image 70 on a frame-by-frame basis include a near point image 72 captured at a near point closer to the distal end 11d than the focus position of the endoscopic image 70, and a far point image 74 captured at a far point farther from the distal end 11d than the focus position of the endoscopic image 70. In the special observation mode, at least one of the near point image 72 and the far point image 74 is captured as a focus position moving image by switching with the imaging of the endoscopic image 70.

[0065] In the pre-recognition imaging pattern, the endoscope 11 is moved to a user-desired imaging site, and images with moving focus position are captured at regular intervals while the endoscopic images 70 are continuously captured. For example, after capturing the endoscopic images 70 in a first period spanning multiple frames, such as three frames, the operation of capturing images with moving focus position is repeated in a second period spanning at least one frame. The second period may be divided into a 2A period in which imaging is performed at a near point, and a 2B period in which imaging is performed at a far point.

[0066] 8, in a pre-recognition imaging pattern in which both a near point image 72 and a far point image 74 are captured as focus position movement images, for example, three frames of endoscopic images 70, one frame of near point image 72, three frames of endoscopic images 70, and one frame of far point image 74 are captured repeatedly in sequence. The endoscopic image 70 is displayed on the screen in the same manner as in the normal observation mode, and the near point image 72 and the far point image 74 are subjected to image analysis.

[0067] 9 and 10, in the pre-recognition imaging pattern, the ratio of capturing an endoscopic image 70 and a focus position movement image, i.e., the ratio between the first period and the second period, may be automatically varied based on the amount of movement of the endoscope 11. When the amount of movement is measured, the processor device 13 measures the amount of movement of the tip 11d of the endoscope 11 and compares the amount of movement with a predetermined reference range. The reference range is preferably determined based on the average movement speed of the endoscope 11 during endoscopic observation, etc.

[0068] As shown in FIG. 9, when the amount of movement is greater than the reference range as a result of the comparison, the imaging frequency of the focus position moving image with respect to the endoscopic image 70 is reduced. For example, the imaging ratio of the endoscopic image 70 to the focus position moving image is set to 6:1, and imaging is repeatedly performed by switching between imaging of 6 frames of the endoscopic image 70, 1 frame of the near point image 72, 6 frames of the endoscopic image 70, and 1 frame of the far point image 74 in sequence. When the amount of movement is large, that is, when the user moves the insertion portion 11a quickly, the imaging part during the change of the imaging area can be grasped with emphasis. In addition, by continuing the image analysis, it is possible to prevent overlooking during the movement. In addition, a threshold value may be further set for the amount of movement, and the imaging frequency of the focus position moving image may be gradually reduced as the amount of movement increases.

[0069] 10, if the comparison result indicates that the amount of movement is smaller than the reference range, the imaging frequency of the focus position movement images with respect to the endoscopic image 70 is increased. For example, the imaging ratio of the endoscopic image 70 to the focus position movement images is set to 1:1, and imaging is repeatedly performed by switching between imaging of two frames of the endoscopic image 70, two frames of the near point image 72, two frames of the endoscopic image 70, and two frames of the far point image 74 in order. When the amount of movement is small, that is, when the user moves the insertion portion 11a slowly, accurate analysis results can be obtained by performing image analysis on a large number of images.

[0070] It is preferable that the imaging frequency of the focus position moving image is set to a level that does not affect observation by the screen display of the endoscopic image 70, for example, the screen display of the endoscopic image 70 is maintained at 30 fps or more. In this case, if the screen display maintains a frame rate of 30 fps, the imaging frequency of the focus position moving image may be set higher than that of the endoscopic image 70. When imaging at the moving focus position is performed in multiple consecutive frames, the focus position may be further moved and imaging is performed. For example, when the near point image 72 is imaged in three consecutive frames, a first near point image, a second near point image, and a third near point image having different focus positions are imaged.

[0071] When the amount of motion is a value included in the reference range, the imaging frequency of the focus position moving image for the endoscopic image 70 is fixed and imaged. The fixed imaging frequency of the focus position moving image is set to be higher than when the amount of motion is greater than the reference range and lower than when the amount of motion is smaller than the reference range. For example, the imaging ratio of the endoscopic image 70 and the focus position moving image is set to 3:1, and switching between imaging one frame of the focus position moving image for every six frames of the endoscopic image 70 (see FIG. 8) is fixed.

[0072] The amount of movement can be obtained by measuring the bending operation or zoom operation using the operation unit 11b or the insertion length of the insertion unit 11a and calculating the amount of movement of the insertion unit 11a of the endoscope 11, or by comparing the analysis results of images with the same focus position.

[0073] When image analysis is used, for example, feature points are identified in image analysis of the near point image 72, and compared with feature points of the near point image 72 acquired before and after. The amount of movement may be measured by calculating an accurate amount of movement using the imaging magnification and observation distance, or a relative amount of movement based on a change in the position of the image may be measured. A similar determination is also made for the far point image 74. The amount of movement may be measured by inputting the endoscopic image 70 to the recognizer 65. In this case, the recognizer 65 performs the measurement in a manner that does not interfere with the image display.

[0074] When an attention area is recognized in a plurality of focus position movement images with different focus positions in the image analysis of the pre-recognition captured pattern, the focus position movement image that is an accurate recognition result is determined using the reliability etc. calculated by the recognizer 65. There is a method for recognizing an attention area by detecting the feature amount in an image, but even if the feature amount is sufficient, in a state where it can be determined that the focus position is inappropriate, such as when the outline is unclear and cannot be specified, the system is trained to calculate a low reliability.

[0075] In the post-recognition imaging pattern, images at the focus position at the time of recognition based on the focus position at which the region of interest was recognized in the image analysis of the pre-recognition imaging pattern are captured continuously or with a higher imaging frequency than the endoscopic image 70. The post-recognition imaging pattern is in a state where images at the focus position at the time of recognition are captured more frequently than images at the focus position moved in the pre-recognition imaging pattern in which the amount of movement is the same. By concentrating on capturing images for analysis, more accurate image analysis can be achieved.

[0076] The post-recognition imaging pattern may be used according to the purpose of the focus position movement image. For example, a first post-recognition imaging pattern for performing image analysis, a second post-recognition imaging pattern for performing screen display, and a third post-recognition imaging pattern for performing both screen display and image analysis are used. The post-recognition imaging pattern setting is changed during endoscopic observation by repeatedly pressing the mode change switch 11e, for example.

[0077] In the post-recognition imaging pattern, the frame rate for the screen display of the endoscopic image 70 is lower than the frame rate for imaging. For example, in a post-recognition imaging pattern in which imaging is performed at 60 fps, if the imaging ratio of the endoscopic image 70 to the image of the focus position movement is 1:1, the screen display of the endoscopic image 70 is adjusted to be an evenly spaced 30 fps.

[0078] When the focus position moving image is displayed on the screen, the display 14 may be split into two screens. For example, the endoscopic image 70 is displayed on the main screen, and the focus position moving image is displayed on the sub-screen. The focus position moving image displayed on the sub-screen may be a still image, or may be displayed as a moving image if the focus position moving image is captured at a sufficient frame rate, such as 30 fps or more. Also, instead of splitting the screen, a display (not shown) different from the display 14 may be electrically connected to the processor device 13, and the endoscopic image 70 and the frame position moving image may be displayed on separate screens.

[0079] 11, in the first post-recognition imaging pattern, image analysis is performed by capturing focus position images at the time of recognition based on the focus position at which the region of interest is recognized, with a higher imaging frequency and in succession than the endoscopic images 70. When the focus position image at the time of recognition is a near point image 72, the near point images are captured in succession and image analysis is performed.

[0080] When switching from image analysis to the post-recognition imaging pattern, image capture of the pre-recognition imaging pattern continues depending on the image size of the focus position movement image in the pre-recognition detection pattern, the performance of the recognizer 65, the frame rate setting, etc., but the imaging control unit 45 switches to the post-recognition imaging pattern in response to an imaging pattern change instruction from the imaging pattern change unit 68.

[0081] 12, in the second post-recognition imaging pattern, the captured focus position image at time of recognition is displayed on the screen. If the focus position movement image in which the attention area is recognized is a near point image 72, the near point image 72 is continuously displayed. Since a state in which imaging switching with the endoscopic image 70 is performed at short intervals is inappropriate for observation, the focus position image at time of recognition is captured and displayed on the screen continuously for a sufficient period, for example, 0.5 seconds or more.

[0082] As shown in Fig. 13, in the third post-recognition imaging pattern, both screen display and image analysis are performed on the captured focus position image at the time of recognition. The screen display may be a sub-screen display similar to the second post-recognition imaging pattern, or the image analysis may be performed and then a sub-screen display of the focus position movement image may be performed as a moving or still image together with the result of the image analysis.

[0083] In the first and third post-recognition imaging patterns for performing image analysis, when a region of interest is detected, a notification is made. The notification is made by a voice or a warning sound, or by highlighting on a display screen. The output control unit 54 controls the notification content based on the recognition result determined by the recognition result determination unit 66.

[0084] The post-recognition imaging pattern captures an image for analysis of the region of interest, so if the imaging range moves away from the region of interest or a predetermined period of time has passed, the system automatically switches to the pre-recognition imaging pattern. By automatically switching to the pre-recognition imaging pattern, observation and image analysis can be performed by applying a focus position suitable for a different imaging range.

[0085] A series of operations for controlling the capturing of focus position moving images in the present invention will be described with reference to the flowchart shown in Fig. 14. The endoscope system 10 sets the observation mode to a special observation mode by a user operation on the endoscope 11 or the processor device 13 (step ST110). By switching to the special observation mode, the endoscope 11 continuously captures images of the subject S in a pre-recognition imaging pattern in which the endoscope 11 switches between a reference focus position and a moving focus position, which is a position to which the focus position is moved from the reference focus position (step ST120).

[0086] In the pre-recognition imaging pattern, the processor device 13 acquires an endoscopic image 70 captured by the endoscope 11 at a reference focus position and a focus position moving image captured by the endoscope 11 at a moving focus position (step ST130). The endoscopic image 70 acquired by the processor device 13 is displayed on a screen, and the focus position moving image is subjected to image analysis (step ST140). If the region of interest is not recognized in the image analysis (N in step ST150), the pre-detection imaging pattern of acquiring the endoscopic image 70 and the focus position moving image is continued (step ST130).

[0087] In the image analysis, when the processor device 13 recognizes the region of interest (Y in step ST150), it acquires focus position information of the focus position movement image in which the region of interest is recognized (step ST160).The imaging pattern is switched to the post-recognition imaging pattern, and the endoscope 11 captures images at the recognition focus position and the reference focus position based on the acquired focus position (step ST170).

[0088] In the post-recognition imaging pattern, an image at the focus position at recognition and an endoscopic image 70 are acquired, and the acquired image at the focus position at recognition is displayed on the screen and image analysis is performed (step ST180). After the image at the focus position at recognition is acquired, if observation in the special observation mode is to be continued (Y in step ST190), observation is again performed in the pre-recognition imaging pattern according to the passage of time and movement of the imaging range (step ST120). If observation in the special observation mode is not to be continued (N in step ST190), the observation mode is switched to the normal observation mode, and the series of steps ends. If observation in the special observation mode is not to be continued, endoscopic observation may be ended as it is, rather than switching to the normal observation mode.

[0089] [Second embodiment] In the special observation mode of the first embodiment, image analysis is performed on the focus position movement image in the pre-recognition captured image pattern, but in the second embodiment, image analysis is performed on the endoscopic image 70. The focus position of the image used in the image analysis of the post-recognition captured image pattern is determined from the analysis result of the endoscopic image 70. Note that a description of other contents that are the same as those in the first embodiment will be omitted.

[0090] In the pre-recognition imaging pattern in the second embodiment, an endoscopic image 70 is captured in the same manner as in the normal observation mode. Image analysis is performed at any frequency on the captured endoscopic image 70. If the region of interest is not recognized by the image analysis, the pre-recognition imaging pattern is continued, and if the region of interest is recognized, focus position information to be applied to the post-recognition imaging pattern is estimated.

[0091] The image analysis unit 64 in the second embodiment has a function of estimating focus position information appropriate for imaging the region of interest by image analysis. The focus position information may be estimated by a recognizer 65 that has learned from a data set including the focus position information, or the image analysis unit 64 may have an estimator (not shown) that is a trained model different from the recognizer 65.

[0092] As shown in FIG. 15, in the pre-recognition imaging pattern, endoscopic images 70 are continuously acquired, and the screen display and image analysis of the endoscopic images 70 are switched in units of one frame in a pre-set pattern. For example, image analysis of one frame is performed for every two frames of screen display. When an attention area is recognized by image analysis, the acquired focus position information is estimated, and the imaging pattern is switched to the post-recognition imaging pattern. In the post-recognition imaging pattern, an estimated focus position image 76 based on the estimated focus position is captured more frequently or continuously than the endoscopic images 70. It is preferable that the estimated focus position image 76 is the same imaging pattern as the recognition focus position image in the first to third post-recognition imaging patterns of the first embodiment.

[0093] The estimated focus position image 76 is a focus position suitable for imaging the estimated region of interest, and therefore applies to any image including the near point image 72, the far point image 74, and the endoscopic image 70. The special observation mode of the second embodiment estimates an appropriate focus position including the focus position of the endoscopic image 70, enabling clear endoscopic observation and more accurate image analysis.

[0094] In the above embodiment, the hardware structure of the processing units that execute various processes, such as the central control unit, the light emission control unit 22, the imaging control unit 45, the image signal acquisition unit 50, the DSP 51, the noise reduction unit 52, the image processing unit 53, the output control unit 54, and the special observation control unit 60, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing various processes.

[0095] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU, etc.). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0096] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (Circuitry) in the form of a combination of circuit elements such as semiconductor elements. The hardware structure of the storage unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). From the above description, the endoscope systems described in the following appendices 1 to 10 can be understood.

[0097] [Appendix 1] An endoscope for imaging a subject; a processor; the endoscope continuously captures images of the subject by switching a focus position between a reference focus position, which is a preset position, and a moving focus position, which is a position to which the focus position is moved from the reference focus position; The processor, acquiring an endoscopic image captured by the endoscope at the reference focus position and a focus position movement image captured by the endoscope at the moving focus position; performing image analysis on the focus position movement image to recognize an attention area from the subject; An endoscope system that displays the endoscopic image on a screen and notifies the results of the image analysis. [Appendix 2] The endoscope includes: An endoscopic system as described in Appendix 1, which captures as the focus position movement image at least one of a near point image in which the focus position is closer to the endoscope than the endoscopic image, and a far point image in which the focus position is farther from the endoscope than the endoscopic image. [Appendix 3] The processor, measuring an amount of deviation of the focus position from a focus position of the subject using a phase difference detection imaging element provided in the endoscope; An endoscopic system as described in Appendix 2, which acquires at least one of position information of the near point and position information of the far point depending on the amount of deviation. [Appendix 4] The processor, Measure the amount of movement of the tip of the endoscope; comparing the amount of movement with a predetermined reference range for the amount of movement; When the amount of movement is greater than the reference range, the frequency of capturing the focus position movement image with respect to the endoscopic image is reduced. 4. The endoscope system according to claim 1, wherein, when the amount of movement is a value smaller than the reference range, a frequency of capturing the focus position movement image with respect to the endoscopic image is increased. [Appendix 5] The endoscope includes: The endoscope system according to claim 4, wherein, when the amount of movement is a value included in the reference range, the imaging frequency of the focus position movement image for the endoscopic image is fixed and the image is captured. [Appendix 6] The endoscope includes: capturing a recognition focus position image based on the focus position at which the attention area is recognized at a higher imaging frequency than the endoscopic image; The processor, 6. The endoscope system according to claim 1, wherein the image analysis is performed on the recognition focus position image. [Appendix 7] The processor, 8. The endoscope system according to claim 6 or 7, wherein the captured image of the focus position at recognition is displayed on a screen. [Appendix 8] The processor, performing the image analysis on the endoscopic image; 7. An endoscope system according to claim 6, wherein the recognition focus position image is determined according to the analysis result of the endoscope image. [Appendix 9] The endoscope includes: An endoscope system according to any one of claims 1 to 8, which repeats the operation of capturing the endoscopic image during a first period spanning multiple frames and capturing the focus position moving image during a second period spanning at least one frame. [Appendix 10] The processor, An endoscope system according to any one of claims 1 to 9, wherein the exposure amount of illumination light that is irradiated from a light source device connected to the endoscope and illuminates the subject is controlled according to the position of the focus lens. [Explanation of symbols]

[0098] 10 Endoscope System 11 Endoscopy 11a Insertion part 11b Operation section 11c Curved section 11d Tip 11e Mode switch 11f Zoom operation section 12 Light source device 13 Processor unit 14 Display 15 User Interface 20 Light source section 22 Light Emission Control Unit 29 Light Guide 30 Illumination optical system 32 Lighting Lens 40 Imaging Optical System 42 Objective Lens 43 Zoom Lens 44 Image sensor 44a Phase difference detection image sensor 44b Phase difference detection pixel 45 Imaging control section 46 CDS / AGC circuit 48 A / D Converter 50 Image signal acquisition unit 51 DSP 52 Noise reduction section 53 Image processing section 54 Output control section 60 Special Observation Control Unit 62 Image acquisition unit for analysis 64 Image Analysis Unit 65 Recognizer 66 Recognition result discriminator 68 Imaging pattern change unit 70 Endoscopic Images 72 Near point image 74 Far point image 76 Estimated focus position image R Image Range S Subject T Focus Target ST110~ST190 Step

Claims

1. an endoscope for imaging a subject; a processor; the endoscope continuously captures images of the subject by switching a focus position between a reference focus position, which is a preset position, and a moving focus position, which is a position to which the focus position is moved from the reference focus position; The processor, acquiring an endoscopic image captured by the endoscope at the reference focus position and a focus position movement image captured by the endoscope at the moving focus position; performing image analysis on the focus position movement image to recognize an attention area from the subject; An endoscope system that displays the endoscopic image on a screen and notifies the results of the image analysis.

2. The endoscope includes: The endoscopic system of claim 1, wherein at least one of a near point image in which the focus position is closer to the endoscope than the endoscopic image and a far point image in which the focus position is farther from the endoscope than the endoscopic image is captured as the focus position movement image.

3. The processor, measuring an amount of deviation of the focus position from a focus position of the subject using a phase difference detection imaging element provided in the endoscope; The endoscope system according to claim 2 , wherein at least one of position information of the near point and position information of the far point is acquired in accordance with the amount of deviation.

4. The processor, Measure the amount of movement of the tip of the endoscope; comparing the amount of movement with a predetermined reference range for the amount of movement; When the amount of movement is greater than the reference range, the frequency of capturing the focus position movement image with respect to the endoscopic image is reduced. The endoscope system according to claim 2 , wherein when the amount of movement is a value smaller than the reference range, the frequency of capturing the focus position movement image for the endoscope image is increased.

5. The endoscope includes: The endoscope system according to claim 4 , wherein, when the amount of movement is a value included in the reference range, the imaging frequency of the focus position movement image with respect to the endoscopic image is fixed.

6. The endoscope includes: capturing a recognition focus position image based on the focus position at which the attention area is recognized at a higher imaging frequency than the endoscopic image; The processor, The endoscope system according to claim 2 , wherein the image analysis is performed on the recognition focus position image.

7. The processor, The endoscope system according to claim 6, wherein the captured image of the focus position at recognition is displayed on a screen.

8. The processor, performing the image analysis on the endoscopic image; The endoscope system according to claim 6 , wherein the recognition focus position image is determined in accordance with an analysis result of the endoscope image.

9. The endoscope includes: The endoscope system according to claim 1 , further comprising: repeating an operation of capturing the endoscopic image in a first period spanning a plurality of frames and capturing the focus position movement image in a second period spanning at least one frame.

10. The processor, 10. The endoscope system according to claim 1, wherein an exposure amount of illumination light that is emitted from a light source device connected to the endoscope and illuminates the subject is controlled in accordance with the focus position.

11. a step of continuously capturing images of a subject by switching between a reference focus position, which is a focus position preset by an endoscope, and a moving focus position, which is the focus position moved from the reference focus position; a step in which the processor device acquires an endoscopic image captured by the endoscope at the reference focus position and a focus position movement image captured by the endoscope at the moving focus position; a step of performing image analysis by the processor device to recognize an attention area from the object on the focus position movement image; A method for operating an endoscopic system comprising a step in which the processor device displays the endoscopic image on a screen and notifies the results of the image analysis.

Images