Endoscope system, non-transitory computer readable medium, and method

Abstract

An endoscope system includes circuitry configured to monitor a feature of an endoscope, the endoscope including an imaging device and a scope device coupled to the imaging device, wherein the feature is indicative of whether the scope device is inserted into a living body; control a focal point of the endoscope system in a first manner if the feature satisfies a predetermined condition; and control the focal point of the endoscope system in a second manner different from the first manner if the feature does not satisfy the predetermined condition.

Description

Technical Field

[0001] This technology relates to endoscope systems, endoscope control methods, and imaging control devices, and particularly to endoscope systems, endoscope control methods, and imaging control devices for performing autofocus (AF) during in vivo imaging.

[0002] <Cross-reference to related applications>

[0003] This application claims the benefit of Japanese priority patent application JP2019-060221, filed on March 27, 2019, the entire contents of which are incorporated herein by reference. Background Technology

[0004] When using medical observation devices such as endoscopes or microscopes, the surgical field of view is typically deeper than the depth of field of view of the medical observation device. Therefore, in some cases, a certain amount of time is spent focusing on the area to be examined. To address this, a medical observation device with an autofocus (AF) function has been proposed (e.g., see Patent Document 1).

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication Text 2017 / 010157 Summary of the Invention

[0008] Technical issues

[0009] Simultaneously, at the surgical site, an insert (e.g., a fiber optic lens) connected to the imaging unit (e.g., a camera head) is inserted into the living tissue to perform imaging. In this case, the insert is sometimes removed from the living tissue during surgery to, for example, to clean blood, water droplets, dirt, etc., adhering to the lens distal to the insert. Therefore, when the insert is inserted into or removed from the living tissue, the image changes drastically, leading to unstable AF operation.

[0010] This technology was developed for such a situation and is intended to achieve stable AF operation when imaging the interior of a living organism.

[0011] Solution to the problem

[0012] An endoscope system according to a first aspect of the present technology includes: circuitry configured to:

[0013] The monitoring endoscope features include an imaging device and a mirror device coupled to the imaging device, wherein the features indicate whether the mirror device is inserted into a living body; when the features meet predetermined conditions, the focus of the endoscope system is controlled in a first manner; and when the features do not meet predetermined conditions, the focus of the endoscope system is controlled in a second manner different from the first manner.

[0014] According to a second aspect of the present invention, a non-transitory computer-readable medium stores a program that, when executed by a computer, causes the computer to perform processing comprising: monitoring features of an insertion part to be inserted into and removed from a living body, the insertion part being connected to an image sensor that generates an image in a medical imaging system; controlling the focus of the medical imaging system in a first manner if the features satisfy predetermined conditions; and controlling the focus of the medical imaging system in a second manner different from the first manner if the features do not satisfy predetermined conditions.

[0015] The method according to a third aspect of the present technology includes: monitoring features of an insertion part to be inserted into and removed from a living body, the insertion part being connected to an image sensor that generates an image in a medical imaging system; controlling the focus of the medical imaging system in a first manner when the features meet predetermined conditions; and controlling the focus of the medical imaging system in a second manner different from the first manner when the features do not meet predetermined conditions. Attached Figure Description

[0016] Figure 1 This is a block diagram illustrating an imaging system according to an embodiment of the present technology.

[0017] Figure 2 This is a block diagram showing an example configuration of a camera head, a display device, and an imaging control device.

[0018] Figure 3 This is a view showing an example of a method for inserting an insert into a living organism.

[0019] Figure 4 This is a flowchart illustrating the AF control process.

[0020] Figure 5 This is a view that shows an example of a captured image.

[0021] Figure 6 This is a view that shows an example of a captured image.

[0022] Figure 7 This is a view that shows an example of a captured image.

[0023] Figure 8 This is a view that shows an example of a captured image.

[0024] Figure 9 This is a view showing an example of an AF frame.

[0025] Figure 10 This is a diagram showing an example configuration of the support arm device.

[0026] Figure 11 This is a view that shows an example of a captured image.

[0027] Figure 12 This is a diagram showing an example of a computer configuration. Detailed Implementation

[0028] Embodiments of this technology will be described below. The description will be presented in the following order.

[0029] 1. Example

[0030] 2. Variation Example

[0031] 3. Other

[0032] <<1. Example>>

[0033] First, refer to Figures 1 to 9 An embodiment of this technology is described.

[0034] <Imaging System Configuration Example>

[0035] Figure 1 An example configuration of the imaging system 1 using this technology is shown.

[0036] Imaging system 1 is an endoscope system used in the medical field, for example, specifically for imaging and observation in living organisms. Imaging system 1 includes an insertion part 11, a light source device 12, a light guide 13, a camera head (imaging unit) 14, a first transmission cable 15, a display device 16, a second transmission cable 17, an imaging control device 18, a third transmission cable 19, a connector 20, and a connector 21.

[0037] For example, the insertion part 11 includes a rigid mirror or a flexible mirror (e.g., a fiber optic mirror). That is, the insertion part 11 is rigid or at least partially flexible and has an elongated shape for insertion into a living organism. The insertion part 11 includes an optical system equipped with one or more lenses to converge or focus an image of the object. The insertion part 11 and the camera head 14 can be integrated with each other.

[0038] The light source device 12 is connected to the first end of the light guide 13 and, under the control of the imaging control device 18, provides light to the first end of the light guide 13 to illuminate the inside of the living body.

[0039] The light guide 13 is detachably connected to the light source device 12 at its first end and detachably connected to the insertion part 11 at its second end. The light guide 13 then transmits light provided from the light source device 12 from the first end to the second end and provides the light to the insertion part 11. The light provided to the insertion part 11 is emitted from the distal end of the insertion part 11 and guided into the interior of the living organism. The light (object image) directed towards and reflected from the interior of the living organism is collected by the optical system in the insertion part 11.

[0040] The camera head 14 is detachably connected to the eyepiece unit 11A, which is the proximal end of the insertion section 11. Then, under the control of the imaging control device 18, the camera head 14 captures the object image converged by the insertion section 11 and outputs an image signal (RAW signal) generated through imaging. An example of an image signal is an image signal of 4K or higher.

[0041] Note that, below, images based on image signals will be referred to as captured images.

[0042] The first transmission cable 15 is detachably connected to the imaging control device 18 at one end via connector 20, and detachably connected to the camera head 14 at the other end via connector 21. Thereafter, the first transmission cable 15 transmits image signals, etc., output from the camera head 14 to the imaging control device 18, and simultaneously transmits each of the control signals, synchronization signals, clock signals, power signals, etc., output from the imaging control device 18 to the camera head 14.

[0043] Note that image signals, etc., transmitted from the camera head 14 to the imaging control device 18 via the first transmission cable 15 can be optical signals or electrical signals. This also applies to the transmission of control signals, synchronization signals, and clock signals from the imaging control device 18 to the camera head 14 via the first transmission cable 15.

[0044] The display device 16 displays an image based on a video signal from the imaging control device 18 under the control of the imaging control device 18.

[0045] The second transmission cable 17 is detachably connected at one end to the display device 16 and at the other end to the imaging control device 18. The second transmission cable 17 then transmits the video signal processed by the imaging control device 18 and the control signal output from the imaging control device 18 to the display device 16.

[0046] The imaging control device 18 includes a camera control unit (CCU) and a central processing unit (CPU). The imaging control device 18 controls the operation of the light source device 12, the camera head 14, and the display device 16 as a whole.

[0047] The third transmission cable 19 is detachably connected at one end to the light source device 12 and at the second end to the imaging control device 18. Subsequently, the third transmission cable 19 transmits control signals from the imaging control device 18 to the light source device 12.

[0048] <Configuration example of camera head 14, display device 16 and imaging control device 18>

[0049] Next, we will refer to Figure 2 This describes an example configuration of the camera head 14, the display device 16, and the imaging control device 18.

[0050] Note that, for ease of explanation, Figure 2 The connectors 20 and 21 that connect each of the imaging control device 18 and the camera head 14 via the first transmission cable 15, and the connector that connects each of the imaging control device 18 and the display device 16 via the second transmission cable 17 are omitted.

[0051] The camera head 14 includes: a lens unit 51, a lens drive unit 52, a lens position detection unit 53, an imaging processing unit 54, a communication unit 55, an inertial measurement unit (IMU) 56, and an input unit 57.

[0052] The lens unit 51 includes a plurality of lenses movable along the optical axis, and forms an object image converged by the insertion portion 11 on the imaging surface of the imaging processing unit 54. The lens unit 51 includes a focusing lens 61 and a zoom lens 62.

[0053] The focusing lens 61 includes one or more lenses and moves along the optical axis to adjust the focus of the camera head 14.

[0054] The zoom lens 62 includes one or more lenses and moves along the optical axis to adjust the angle of view of the camera head 14.

[0055] In addition, the lens unit 51 also includes a focusing mechanism (not shown) for moving the focusing lens 61 along the optical axis and an optical zoom mechanism (not shown) for moving the zoom lens 62 along the optical axis.

[0056] The lens drive unit 52 includes: an actuator 71 that operates the aforementioned focusing mechanism and optical zoom mechanism; and a driver 72 that drives the actuator 71. Furthermore, the lens drive unit 52 adjusts the focal point and viewing angle of the lens unit 51 under the control of the imaging control device 18.

[0057] The lens position detection unit 53 is configured using a position sensor such as an optical circuit breaker, and detects the position of the focusing lens 61 (hereinafter referred to as the focal position) and the position of the zoom lens 62 (hereinafter referred to as the zoom position). Subsequently, the lens position detection unit 53 outputs detection signals corresponding to the focal position and zoom position to the imaging control device 18 via the first transmission cable 15.

[0058] The imaging processing unit 54 includes a sensor chip on which an imaging element (not shown), a signal processing unit (not shown), and the like are integrated. The imaging element may include, for example, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS). This imaging element receives an image of the object formed by the insertion portion 11 and captured by the lens unit 51, and converts the image into an electrical signal. The signal processing unit performs signal processing (such as A / D conversion) on the electrical signal (analog signal) from the imaging element and outputs an image signal. Under the control of the imaging control device 18, the imaging processing unit 54 captures an image of the interior of a living organism and outputs an image signal (digital signal) generated through A / D conversion.

[0059] Note that the signal processing unit described above can be separated from the imaging element, rather than being formed as a whole.

[0060] The communication unit 55 serves as a transmitter to send image signals output from the imaging processing unit 54 to the imaging control device 18 via the first transmission cable 15. For example, the communication unit 55 includes a high-speed serial interface for transmitting image signals to the imaging control device 18 at a transmission rate of 1Gbps or higher via the first transmission cable 15.

[0061] The IMU 56 detects the acceleration and angular velocity of the camera head 14 and provides a detection signal indicating the detection result to the imaging control device 18.

[0062] The input unit 57 includes input devices such as buttons, switches, touch panels, etc., and detects user input. The input unit 57 provides input signals to the imaging control device 18 based on the user input.

[0063] The imaging control device 18 includes: a communication unit 101, a signal processing unit 102, a display control unit 103, a control unit 104, an input unit 105, an output unit 106, and a storage unit 107.

[0064] The communication unit 101 serves as a receiver for receiving image signals output from the camera head 14 (communication unit 55) via the first transmission cable 15. For example, the communication unit 101 includes a high-speed serial interface for transmitting image signals to the communication unit 55 at a transmission rate of 1Gbps or higher.

[0065] Under the control of the control unit 104, the signal processing unit 102 performs various types of processing on the image signal (RAW signal) output from the camera head 14 (communication unit 55) and received by the communication unit 101. The signal processing unit 102 includes: an image processing unit 111, a demodulation processing unit 112, a position estimation unit 113, a distance measurement unit 114, and an image recognition unit 115.

[0066] The image processing unit 111 performs RAW processing on the image signal (RAW signal) received by the communication unit 101, such as optical black subtraction processing and demosaic processing, and converts the processed RAW signal (image signal) into an RGB signal (image signal). Furthermore, the image processing unit 111 performs RGB processing on the RGB signal (image signal), such as white balance, RGB gamma correction, and YC conversion (converting the RGB signal into a luminance signal and a color difference signal (Y, Cb / Cr signal)). Additionally, the image processing unit 111 performs YC processing on the Y and Cb / Cr signals (image signals), such as color difference correction and noise reduction. The image processing unit 111 provides the image signal generated through image processing to the display control unit 103 and the control unit 104.

[0067] The demodulation processing unit 112 performs demodulation processing (e.g., autofocus (AF) processing) on ​​the image signal (e.g., Y, Cb / Cr signal) processed by the image processing unit 111 for controlling the camera head 14. Note that the image signal is not limited to Y or Cb / Cr signals, and can be any signal to which image processing can be applied, and can be a RAW signal if it has not undergone RAW processing. Furthermore, the image signal can be a luminance signal when a luminance signal (Y) is to be generated in RAW processing.

[0068] For example, demodulation processing unit 112 detects the contrast and frequency components of the image within a specified region based on pixel information (luminance signal (Y signal)) of each pixel in a specified region of a captured image frame captured by imaging processing unit 54. Subsequently, demodulation processing unit 112 calculates a focus evaluation value to assess the focus state of the captured image (the object image in the captured image) based on the detected contrast and frequency components. For example, demodulation processing unit 112 calculates the contrast of the image within the specified region, or the sum of the high-frequency components of the image within the specified region, as the focus evaluation value. Note that a higher focus evaluation value indicates better image focus alignment. Demodulation processing unit 112 provides demodulation information indicating the calculated focus evaluation value to control unit 104.

[0069] The position estimation unit 113 performs position estimation processing of the camera head 14 based on the image signal and detection signal from the IMU 56. For example, based on the image signal, the position estimation unit 113 creates an environment map (3D coordinate map) and estimates the position of the camera head 14 in the environment map using Visual Simultaneous Localization and Mapping (Visual-SLAM). Alternatively, for example, based on the image signal and detection signal from the IMU 56, the position estimation unit 113 creates an environment map and estimates the position of the camera head 14 in the environment map using IMU-SLAM. The position estimation unit 113 provides position information indicating the estimated position of the camera head 14 to the control unit 104.

[0070] Note, for example, that details of Visual-SLAM are described in Andrew J. Davison, “Real-time Simultaneous Localization and Mapping with a Single Camera”, Proceedings of the 9th IEEE International Conference on Computer Vision, Volume 2, 2003, pp. 1403-1410, or in JP 2011-95797A. Furthermore, for example, details of IMU-SLAM are described in JP 2017-185254 A.

[0071] The distance measurement unit 114 measures the distance to the object at the camera head 14 and provides distance information indicating the measurement result to the control unit 104.

[0072] Note that the object used as the target for distance measurement can be, for example, an organ or part of an organ in a living organism.

[0073] Furthermore, any method can be used as a distance measurement method. For example, the imaging processing unit 54 includes an imaging element or a time-of-flight (ToF) sensor, the imaging element including phase difference pixels, and the imaging processing unit 54 uses the imaging element or the ToF sensor to measure the distance to the object.

[0074] The image recognition unit 115 performs object recognition processing in the captured image based on the image signal. For example, the image recognition unit 115 performs recognition processing for predetermined objects such as blood, water droplets, or dirt in the captured image. Furthermore, for example, based on the object recognition results in the captured image, the image recognition unit 115 performs surgical scene recognition processing in a surgical procedure, such as a surgical process. The image recognition unit 115 provides the control unit 104 with recognition information indicating the object recognition results or the surgical scene recognition results.

[0075] Under the control of the control unit 104, the display control unit 103 uses on-screen display (OSD) processing, etc., and generates a video signal for display based on the image signal (Y, Cb / Cr signal) processed by the image processing unit 111. Subsequently, the display control unit 103 outputs the generated video signal to the display device 16 (display unit 151) via the second transmission cable 17.

[0076] The control unit 104 includes a CPU, etc., and outputs control signals via a first transmission cable 15, a second transmission cable 17, and a third transmission cable 19, thereby controlling the operation of the light source device 12, the camera head 14, and the display device 16, as well as controlling the overall operation of the imaging control device 18. The control unit 104 includes an insertion state detection unit 121, a focus control unit 122, and a zoom control unit 123.

[0077] The insertion state detection unit 121 detects insertion of the insertion part 11 into the insertion port and into the living body, or detects removal of the insertion part 11 from the insertion port and from the living body, based on at least one of the detection signal from the IMU 56, the image signal from the image processing unit 111, the demodulation information from the demodulation processing unit 112, the position information from the position estimation unit 113, or the distance information from the distance measurement unit 114. In other words, the insertion state detection unit 121 monitors features of an endoscope including a camera head 14 and an insertion part 11 coupled to the camera head 14, wherein these features indicate whether the insertion part 11 is inserted into the living body. For example, this feature can be monitored by analyzing one or more of the following: an image captured by an image sensor via the endoscope device, the position of the insertion tube, the tilt of the camera head, the shape of the distal port in the image captured by the image sensor via the endoscope device, and setting information from the support arm. Analyzing the image captured by the image sensor may include, for example, analyzing the color of the image, the color of the peripheral portion of the image, the brightness of the image, and the brightness of the peripheral portion of the image.

[0078] The focus control unit 122 operates the lens drive unit 52 to adjust the focal point of the lens unit 51 (change the focal point position). For example, the focus control unit 122 performs autofocus (AF) processing based on the focal point position detected by the lens position detection unit 53 and the demodulation information from the demodulation processing unit 112. Furthermore, the focus control unit 122 controls the AF operation based on at least one of the following: operation information from the input unit 57 or input unit 105, the insertion and removal detection results of the insertion unit 11 obtained by the insertion state detection unit 121, and the recognition information from the image recognition unit 115. For example, the focus control unit 122 changes the focus control conditions to control the focus control operation.

[0079] Note that, for example, continuous AF and single-shot AF are installed as AF functions of the imaging system 1. Continuous AF is basically a function that continuously (constantly) performs AF. For example, continuous AF is turned on or off according to an operation on an operation unit (not shown) provided in the input unit 57 or the input unit 105. For example, single-shot AF is a function that performs point AF in response to an operation on an operation unit (not shown) provided in the input unit 57 of the camera head 14.

[0080] The zoom control unit 123 operates the lens driving unit 52 to adjust the viewing angle of the lens unit 51 (change the zoom position).

[0081] The input unit 105 includes input devices such as buttons, switches, mice, keyboards, touch panels, etc., and receives user input. The input unit 105 provides an input signal to the control unit 104 according to the user input.

[0082] The output unit 106 includes, for example, a speaker, a printer, etc., and outputs various types of information.

[0083] The storage unit 107 stores programs to be executed by the control unit 104, information for processing by the control unit 104, etc. The storage unit 107 is a non-transitory computer-readable medium on which a program that causes the control unit 104 to execute processing when executed by a computer is stored.

[0084] The display device 16 includes a display unit 151.

[0085] The display unit 151 includes a display panel such as a liquid crystal panel, an organic electroluminescence (EL) panel, etc., and displays a display image based on a video signal from the imaging control device 18.

[0086] <Method of inserting the insertion portion 11>

[0087] Figure 3 An example of a method for inserting the insertion portion 11 of the imaging system 1 into a living body 202 is shown. In this example, the insertion portion 11 is inserted into the living body 202 using a trocar 201.

[0088] Specifically, the trocar 201 is inserted into the skin 202A of the living body 202. Subsequently, the insertion portion 11 is inserted from the insertion port 201A of the trocar 201, and the distal end of the insertion portion 11 protrudes from the distal port 201B of the trocar 201. With this configuration, the insertion portion 11 is inserted into the living body 202, and the position of the insertion portion 11 is stabilized.

[0089] <AF control processing>

[0090] Next, reference will be made to Figure 4The flowchart in the diagram describes the AF control processing performed by the imaging system 1.

[0091] For example, the process begins when the power supply to the imaging control device 18 is turned on, and ends when the power supply to the imaging control device 18 is turned off.

[0092] In step S1, the insertion state detection unit 121 begins to detect the insertion state of the insertion part 11. For example, the insertion state detection unit 121 begins to detect the insertion of the insertion part 11 into the living body or the removal of the insertion part 11 from the living body.

[0093] Here, inserting the insertion part 11 into a living body means, for example, inserting the distal end of the insertion part 11 from a predetermined insertion port toward the living body by a predetermined distance of 0 mm or more. Conversely, removing the insertion part 11 from the inside of the living body means, for example, removing the distal end of the insertion part 11 from the predetermined insertion port toward the inside of the living body by a predetermined distance of 0 mm or more.

[0094] Furthermore, the insertion port is an opening for inserting the insertion part 11 into a living body, and corresponds to, for example, the opening of an insertion aid (e.g., the insertion port 201A of the cannula 201), such as the body opening of an ear canal or nostril, or an opening in an incision on the patient's body.

[0095] Furthermore, when the insertion part 11 is inserted into a living body via a long tubular member such as the cannula needle 201, the insertion port for detecting the insertion of the insertion part 11 can be provided separately from the insertion port for detecting the removal of the insertion part 11. For example, Figure 3 The distal port 201B of the cannula 201 can be used to detect the insertion of the insertion part 11, and the insertion port 201A of the cannula 201 can be used to detect the removal of the insertion part 11.

[0096] Here, examples of methods for detecting the insertion of the insertion part 11 into a living body and removing the insertion part 11 from the living body will be described.

[0097] For example, the insertion state detection unit 121 detects the insertion and removal of the insertion section 11 based on the color information (or wavelength information) of the captured image.

[0098] Figures 5 to 8 Examples of captured images 251 to 254 are schematically shown, in which the insertion part 11 moves from a state of being inserted into the living body using the cannula needle 201 to a state of being removed from the living body.

[0099] Specifically, Figure 5 This is an example of a captured image 251 obtained with the insertion part 11 inserted into a living organism. Since most of the captured image 251 is occupied by internal organs, most of the captured image 251 is slightly reddish.

[0100] Figure 6 This is an example of a captured image 252, taken in order to remove the insertion part 11, with the distal end of the insertion part 11 slightly separated from the internal organs, compared to the time of the captured image 251. Since most of the captured image 252 is occupied by the internal organs, most of the captured image 252 is slightly reddish.

[0101] Figure 7 This is an example of a captured image 253 with the distal end of the insertion portion 11 slightly penetrating the interior of the cannula 201 from the distal port 201B. Since most of the central portion of the captured image 253, as viewed from the distal port 201B, is occupied by internal organs, it appears slightly reddish. Conversely, most of the peripheral portion of the captured image is occupied by the inner wall of the cannula 201, and light emitted from the distal end of the insertion portion 11 is reflected by the inner wall. Therefore, most of the peripheral portion of the captured image 253 appears white (or the silver of the cannula's inner wall). Furthermore, the peripheral portion of the captured image 253 may be overexposed.

[0102] Figure 8 This is an example of a captured image 254 taken with the distal end of the insertion portion 11 removed to the vicinity of the insertion port 201A of the cannula 201. In captured image 254, the distal port 201B in the central portion is smaller than the distal port 201B in captured image 253. Furthermore, since internal organs are barely visible from the distal port 201B, the central portion of captured image 254 is dark and black. Conversely, most of the peripheral portion of captured image 254 is occupied by the inner wall of the cannula 201, and light emitted from the distal end of the insertion portion 11 is reflected by the inner wall. Therefore, most of the peripheral portion of captured image 254 is white (or the silver of the inner wall of the cannula). Furthermore, the peripheral portion of captured image 254 may be overexposed. However, since the amount of light reflected by the inner wall is reduced around the distal port 201B, the color becomes darker than the color in captured image 253.

[0103] The insertion state detection unit 121 detects the insertion and removal of the insertion part 11 by using the occurrence of a large change in the color of the captured image caused by inserting and removing the insertion part 11 in this way. For example, the insertion and removal of the insertion part 11 is detected based on the color change of the central part and the peripheral part of the captured image.

[0104] Specifically, for example, when the peripheral portion of the captured image turns white, the insertion state detection unit 121 detects the insertion and removal of the insertion portion 11 based on the color change of the central portion. For example, the insertion state detection unit 121 determines that the insertion portion 11 has been removed when the central portion of the captured image changes from red to black. Conversely, for example, the insertion state detection unit 121 determines that the insertion portion 11 has been inserted when the central portion of the captured image changes from black to red.

[0105] Note that the insertion state detection unit 121 can detect the insertion and removal of the insert 11 based solely on the color change of the peripheral portion of the captured image. For example, the insertion state detection unit 121 determines that the insert 11 has been removed when the peripheral portion of the captured image changes from red to white. Conversely, the insertion state detection unit 121 determines that the insert 11 has been inserted when the peripheral portion of the captured image changes from white to red. This allows for a reduction in the computational load required to detect the insertion state of the insert 11.

[0106] Furthermore, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion section 11 based on the brightness of the captured image.

[0107] Specifically, when the inserter 11 is inserted into a living organism, the brightness of the captured image remains stable and essentially unchanged due to the absence of external light. Conversely, when the inserter 11 is removed from the living organism, the brightness of the captured image fluctuates and becomes unstable due to the movement of the inserter 11 or the influence of external light. Furthermore, when the distal end of the inserter 11 is located inside the cannula 201, light emitted from the distal end of the inserter 11 is reflected onto the inner wall of the cannula 201, resulting in overexposure of the peripheral portion of the captured image and causing significant variations in brightness.

[0108] Therefore, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 based on the change in brightness of the peripheral portion of the captured image. For example, the insertion state detection unit 121 determines that the insertion part 11 is removed when the brightness of the peripheral portion of the captured image changes significantly from a stable state. Conversely, for example, the insertion state detection unit 121 determines that the insertion part 11 is inserted when the brightness of the peripheral portion of the captured image changes significantly from an unstable state.

[0109] Note that a stable brightness state in the peripheral area of ​​the captured image indicates, for example, that the average brightness of the peripheral area of ​​the captured image fluctuates within a predetermined time period of less than a predetermined threshold. An unstable brightness state in the peripheral area of ​​the captured image indicates, for example, that the average brightness of the peripheral area of ​​the captured image fluctuates within a predetermined time period of more than a predetermined threshold. A large change in brightness in the peripheral area of ​​the captured image indicates, for example, that the change in the average brightness of the peripheral area of ​​the captured image exceeds a predetermined threshold.

[0110] Alternatively, the insertion and removal of the insertion unit 11 can be detected based on the change in brightness of the entire captured image, or based on the change in brightness of the portion of the captured image other than the peripheral portion, or based on the change in brightness of the peripheral portion and other portions of the captured image.

[0111] Furthermore, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 based on the position of the camera head 14 in the environment map estimated by the position estimation unit 113. For example, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 by tracking the movement (position change) of the camera head 14 relative to the insertion port to the living body in the environment map.

[0112] Note that, for example, the position estimation unit 113 can estimate the position of the distal end of the insertion part 11 instead of the position of the camera head 14, and the insertion state detection unit 121 can track the movement (position change) of the distal end of the insertion part 11 relative to the position of the insertion port to the living body in the environment map, thereby detecting the insertion and removal of the insertion part 11.

[0113] Furthermore, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 based on the acceleration of the camera head 14 detected by the IMU 56. For example, if the change in the acceleration of the camera head 14 meets a predetermined condition, the insertion state detection unit 121 determines that the insertion part 11 has been inserted or removed. For example, if the acceleration of the camera head 14 changes from a state where the acceleration is essentially zero (the camera head 14 is essentially stationary) to a state where the acceleration of the camera head 14 is above a predetermined threshold, the insertion state detection unit 121 determines that the insertion part 11 has been removed. Conversely, for example, if the acceleration of the camera head 14 changes from a state where the acceleration of the camera head 14 is above a predetermined threshold to a state where the acceleration of the camera head 14 is essentially zero (the camera head 14 is essentially stationary), the insertion state detection unit 121 determines that the insertion part 11 has been inserted.

[0114] Furthermore, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 based on the tilt of the camera head 14 detected by the IMU 56. For example, if the change in the tilt of the camera head 14 meets a predetermined condition, the insertion state detection unit 121 determines that the insertion part 11 has been inserted or removed. For example, if the tilt of the camera head 14 changes by a predetermined threshold from a state where the change in the tilt of the camera head 14 is substantially zero (a state where the camera head 14 is substantially stationary), the insertion state detection unit 121 determines that the insertion part 11 has been removed. Conversely, for example, if the change in the tilt of the camera head 14 changes by a predetermined threshold and thereafter the change in the tilt of the camera head 14 becomes substantially zero, the insertion state detection unit 121 determines that the insertion part 11 has been inserted.

[0115] Note that, for example, the IMU can be located at the distal end of the insertion part 11, and under conditions similar to those of the camera head 14, the insertion and removal of the insertion part 11 can be detected based on the acceleration or tilt of the distal end of the insertion part 11.

[0116] In addition, as mentioned above Figure 7 and Figure 8 As shown, the distal port 201B of the cannula 201 is approximately circular in the captured image. Therefore, for example, the image recognition unit 115 identifies the distal port 201B of the cannula 201 in the captured image, and the insertion state detection unit 121 detects the insertion or removal of the insertion portion 11 based on the shape of the distal port 201B in the captured image. For example, the insertion state detection unit 121 determines that the insertion portion 11 has been removed if the size of the distal port 201B gradually decreases and then disappears from the captured image. Conversely, the insertion state detection unit 121 determines that the insertion portion 11 has been inserted if the size of the distal port 201B gradually increases and then disappears from the captured image.

[0117] Furthermore, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion section 11 based on the focus evaluation value calculated by the demodulation processing unit 112.

[0118] Specifically, when the inserter 11 is inserted into a living organism, the distance to the object (e.g., to an internal organ) changes almost nothing, and therefore the focus assessment value remains stable and essentially unchanged. Conversely, when the inserter 11 is removed from the living organism, the movement of the inserter 11 or the object causes fluctuations in the distance to the object, resulting in unstable focus assessment values ​​with fluctuations. Furthermore, when the distal end of the inserter 11 moves within the cannula 201, the change in the distance to the object due to the movement of the inserter 11 leads to significant changes in the focus assessment value.

[0119] Therefore, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 based on changes in the focus evaluation value. For example, if the focus evaluation value changes significantly from a stable state, the insertion state detection unit 121 determines that the insertion part 11 has been removed. For example, if the focus evaluation value changes significantly from an unstable state, the insertion state detection unit 121 determines that the insertion part 11 has been inserted.

[0120] Note that a stable focus evaluation value indicates, for example, that the fluctuation range of the focus evaluation value within a previously predetermined time period is less than a predetermined threshold. An unstable focus evaluation value indicates, for example, that the fluctuation range of the focus evaluation value within a previously predetermined time period is greater than a predetermined threshold. A large change in the focus evaluation value indicates, for example, a change greater than a predetermined threshold.

[0121] Note that multiple frames (hereinafter referred to as AF frames) can be provided in the captured image, and the insertion and removal of the insertion part 11 can be detected based on the focus evaluation value detected in each AF frame.

[0122] Figure 9 An example of AF frame settings is shown. In this example, a total of 25 large AF frames (5 rows x 5 columns) are provided in the captured image. Additionally, a total of 16 small AF frames (4 rows x 4 columns) are provided within 9 AF frames in the central portion of the captured image. Therefore, in this example, a total of 169 AF frames are provided, including 25 large AF frames and 144 small AF frames. Note that, for example, large AF frames can be used to calculate AF evaluation values ​​using image contrast, and small AF frames can be used to calculate AF evaluation values ​​using phase difference information in the image plane phase difference.

[0123] Subsequently, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion unit 11 based on the change in the focus evaluation value of the peripheral portion relative to the focus evaluation value of the AF frame of the central portion of the captured image.

[0124] Note that the insertion state detection unit 121 may detect the insertion and removal of the insertion unit 11 based solely on the change in focus evaluation value of the central portion of the captured image, or solely on the change in focus evaluation value of the peripheral portion of the captured image.

[0125] Furthermore, for example, the insertion state detection unit 121 detects the insertion and removal of the insertion part 11 based on the distance to the object measured by the distance measurement unit 114. For example, if the distance to the object changes significantly from a stable state, the insertion state detection unit 121 determines that the insertion part 11 has been removed. Conversely, for example, if the distance to the object changes significantly from an unstable state, the insertion state detection unit 121 determines that the insertion part 11 has been inserted.

[0126] Note that a stable distance to the object indicates, for example, that the distance fluctuation range within a previously predetermined time period is less than a predetermined threshold. An unstable distance to the object indicates, for example, that the distance fluctuation range within a previously predetermined time period is greater than a predetermined threshold. A large change in the distance to the object indicates, for example, a distance change greater than a predetermined threshold.

[0127] Note that the insertion and removal detection method of the insertion unit 11 described above is an example, and other detection methods can be used. Furthermore, multiple detection methods can be combined with each other.

[0128] Back Figure 4 In step S2, the focus control unit 122 determines whether continuous AF is enabled. If it is determined that continuous AF is enabled, that is, if the continuous AF function is activated, the process proceeds to step S3.

[0129] In step S3, the focus control unit 122 restricts AF operation. For example, the focus control unit 122 stops AF operation until the insertion part 11 is inserted into the living body. With this configuration, for example, even when AF is continuously activated while the imaging control device 18 is powered on, AF is suppressed until the insertion part 11 is inserted into the living body.

[0130] Note that when continuous AF is set while the imaging control device 18 is powered on, AF operation can begin regardless of whether the insertion part 11 is inserted into the living body.

[0131] In step S4, the insertion state detection unit 121 determines whether the insertion part 11 is inserted into the living body. If it is determined that the insertion part 11 is inserted into the living body, the process proceeds to step S5.

[0132] In step S5, the focus control unit 122 begins normal AF operation. That is, the focus control unit 122 removes the restrictions on AF operation and operates the lens drive unit 52 based on the focal position detected by the lens position detection unit 53 and the demodulation information from the demodulation processing unit 112, thereby initiating the process of automatically adjusting the focal point of the lens unit 51.

[0133] Note that normal AF operation can begin after the predetermined conditions are met, rather than immediately after the insertion part 11 is inserted into the living body.

[0134] For example, the focus control unit 122 begins normal AF operation after a predetermined time period (e.g., a few seconds) elapses from when the insertion unit 11 is detected. Note that the time period until the start of normal AF operation can be a fixed value or a variable value. In the case of a variable value, the time period can be set based on, for example, user settings, the time period elapsed before the restriction on AF operation is lifted, preset conditions, etc.

[0135] Furthermore, for example, after detecting the insertion of the insertion unit 11, when the focus evaluation value is stable or the brightness (luminance) of the captured image is stable, the focus control unit 122 begins normal AF operation, for example, controlling the focus of the endoscope system in a first manner. Note that the state of stable focus evaluation value and stable brightness of the captured image are considered similar to the states described above.

[0136] Furthermore, for example, normal AF operation can be gradually started after the insertion part 11 is inserted into the living body.

[0137] For example, after detecting the insertion of the insertion unit 11, the focus control unit 122 slows down the AF operation for a predetermined period of time. Specifically, for example, after detecting the insertion of the insertion unit 11, the focus control unit 122 performs the AF operation while limiting the moving speed of the focusing lens 61 (i.e., the AF operation speed) to below a predetermined value for a predetermined period of time. Subsequently, after the predetermined period of time has elapsed, the focus control unit 122 removes the restriction on the moving speed of the focusing lens 61. Note that the duration for delaying the AF operation can be a fixed value or a variable value. In the case of a variable value, the duration is set according to, for example, user settings, the time elapsed before the restriction on the AF operation is removed, preset conditions, etc.

[0138] This is because, in the case of an endoscopic operator performing operations of the imaging system 1 such as the insertion and removal of the insertion part 11, removing the insertion part 11 from inside the living body and then re-inserting it, it may take time to confirm whether the captured image is the same as the image before removal. Therefore, even when the insertion part 11 is inserted into the living body, focusing is performed using any of the methods described above, which takes some time instead of being performed immediately.

[0139] The process then proceeds to step S6.

[0140] Conversely, if it is not determined in step S4 that the insertion part 11 is inserted into the living body, that is, if the insertion part 11 remains inserted into the living body or the insertion part 11 remains removed from the living body, the process in step S5 is skipped and the process proceeds to step S6.

[0141] In step S6, the insertion state detection unit 121 determines whether the insertion part 11 has been removed from the living body. If it is determined that the insertion part 11 has been removed from the living body, the process proceeds to step S7.

[0142] In step S7, the focus control unit 122 restricts AF operation.

[0143] For example, if AF operation continues even after the insertion part 11 has been removed from inside the living organism, the focal position will change significantly in order to focus on a certain location outside the living organism. Furthermore, for example, when cleaning the insertion part 11, the focal position will continuously change due to the adhesion of water droplets, etc. Therefore, for example, this causes continuous movement of the focusing lens 61 and generates unwanted motion noise or vibration, which may cause concern for doctors, nurses, endoscopy operators, patients, etc. In addition, this may raise concerns about premature degradation of the actuator 71 that drives the focusing lens 61.

[0144] To overcome these problems, for example, the focusing control unit 122 stops the AF operation when the insertion part 11 is removed from the living body, for example, when the insertion part 11 exits from the insertion port. In other words, when the insertion part 11 is removed, the focus of the endoscope system is controlled in a second manner different from the first manner (e.g., having a fixed focus position).

[0145] Meanwhile, when the insertion part 11 is inserted into a living body, the focal point is located very close to (proximal) an internal organ. Conversely, when the insertion part 11 is removed from the living body, the focal point can be located further away (distal) than when it was inserted, for example, with a high probability on a person or device in an operating room.

[0146] Therefore, for example, removing the insert 11 from inside the living organism and then re-inserting it with the focus on the far side will increase the blur of the captured image, leading to unstable focus evaluation values. Furthermore, the focusing lens 61 needs to be moved from the far side to the near side. Therefore, after re-inserting the insert 11, a longer time is required to adjust the focus.

[0147] Furthermore, for example, when the insertion part 11 needs to be removed, cleaned, and then reinserted, the endoscopic operator can adjust the insertion depth of the insertion part 11 to focus on a target location such as an internal organ. However, when the focal position changes significantly when the insertion part 11 is removed, the target location will not be in focus upon reinsertion, making it difficult for the endoscopic operator to determine how deep the insertion part 11 should be inserted.

[0148] To address this, for example, the focus control unit 122 fixes the focal position (the position of the focusing lens 61) to the position at the moment the insertion part 11 is removed from the living body or a predetermined reference position. Alternatively, for example, the focus control unit 122 limits the focal position operating range to a narrower range than the time before the insertion part 11 is removed from the living body. In this case, for example, the focus control unit 122 sets the limited focal position operating range based on the focal position or reference position at the time the insertion part 11 is removed from the living body.

[0149] For example, the reference position is set as the focal point (i.e., near) of an object at a predetermined short distance (e.g., a few millimeters to a few centimeters). Specifically, for example, the reference position is set as the focal point used when an internal organ in a living organism is in focus.

[0150] Alternatively, the reference position is set to the focal position immediately preceding the removal of the insertion part 11. For example, the focus control unit 122 continuously stores the focal positions over a previously predetermined time period. Subsequently, upon detecting the removal of the insertion part 11, the focus control unit 122 sets the focal position in a stable state prior to the removal of the insertion part 11 as the reference position. For example, the focus control unit 122 sets the average value of the focal position during a period when the focal position was stable before the removal of the insertion part 11 (e.g., a period in which the focal position fluctuation range is less than a predetermined threshold) as the reference position.

[0151] In this way, fixing the focus position or limiting the focus position operation range will allow for quick focusing on the desired target location (e.g., internal organs) when AF is restored.

[0152] The process then proceeds to step S8.

[0153] Conversely, if it is not determined in step S6 that the insertion part 11 has been removed from the living body, that is, if the insertion part 11 remains inserted in the living body or the insertion part 11 remains removed from the living body, the process of step S7 is skipped and the process proceeds to step S8.

[0154] In step S8, the focus control unit 122 determines whether to turn off the continuous AF based on the input signal from input unit 57 or input unit 105. If it is determined that the continuous AF is not turned off, the process returns to step S4.

[0155] Subsequently, the process of steps S4 to S8 is repeated until it is determined in step S8 that the continuous AF is turned off.

[0156] Conversely, if it is determined in step S8 that the continuous AF is turned off, the process proceeds to step S9. This is assumed, for example, that the user performs a continuous AF turn-off operation using input unit 57 or input unit 105.

[0157] In step S9, the focus control unit 122 stops the AF operation.

[0158] The process then proceeds to step S10.

[0159] Conversely, if it is determined in step S2 that continuous AF is off, the processing in steps S3 to S9 is skipped, and the process proceeds to step S10. This corresponds, for example, to the case where continuous AF is off when the power to the imaging control device 18 is on.

[0160] In step S10, the focus control unit 122 determines whether to activate the single-touch AF based on the input signal from the input unit 57. If it is determined that the single-touch AF is activated, the process proceeds to step S11. This is assumed to be, for example, the case where the user performs a single-touch AF activation operation using the input unit 57.

[0161] In step S11, the focusing control unit 122 performs normal AF operation. With this configuration, for example, the desired target location such as an internal organ becomes the focus.

[0162] The process then proceeds to step S12.

[0163] Conversely, if it is determined in step S10 that the single-touch AF is not activated, the process in step S11 is skipped, and the process proceeds to step S12.

[0164] In step S12, the focus control unit 122 determines whether to activate continuous AF based on the input signal from input unit 57 or input unit 105. If it is determined that continuous AF is not activated, the process returns to step S10.

[0165] After that, the process of steps S10 to S12 is repeated until it is determined in step S12 that the continuous AF is turned on.

[0166] Conversely, if it is determined in step S12 that the continuous AF is on, the process proceeds to step S13. This is assumed to be, for example, the case where the user performs a continuous AF on operation using input unit 57 or input unit 105.

[0167] In step S13, the insertion state detection unit 121 determines whether the insertion part 11 has been inserted into the living body. If it is determined that the insertion part 11 has been inserted into the living body, the process returns to step S5, and the processing in step S5 and subsequent processing are executed. That is, when continuous AF is turned on while the insertion part 11 is inserted into the living body, normal AF operation begins.

[0168] Conversely, if it is determined in step S13 that the insertion part 11 has not been inserted into the living body, the process returns to step S3, and the processing in step S3 and subsequent processing are performed. That is, when continuous AF is turned on while the insertion part 11 is not inserted into the living body, normal AF operation is restricted until the insertion part 11 is inserted into the living body.

[0169] As described above, stable AF operation can be achieved when imaging the interior of a living organism. For example, when continuous AF is engaged, when the insertion unit 11 is removed from and reinserted from the interior of the living organism, the same object as before removal can be quickly set as the focus.

[0170] <<2. Variation Example>>

[0171] In the following text, variations of the above embodiments of the present technology will be described.

[0172]

[0173] For example, such as Figure 10 As shown, when the camera head 14 and the insertion part 11 are moved while being supported by the support arm device 301, the insertion state detection unit 121 can detect the insertion and removal of the insertion part 11 based on the movement of the support arm device 301.

[0174] The support arm assembly 301 includes a base unit 311 serving as a base and an arm portion 312 extending from the base unit 311. In this example, the arm portion 312 includes: a plurality of connectors 321a to 321c; and a plurality of connectors 322a and 322b connected via connectors 321b.

[0175] The arm control device 302 drives the actuators disposed in joints 321a to 321c and controls the rotation angle of joints 321a to 321c to control the movement of arm 312. Subsequently, arm 312 is driven to control the position and orientation of camera head 14 and insertion part 11, allowing insertion part 11 to be inserted into the body of patient 304 on patient bed 303 from the desired direction via cannula 305.

[0176] Next, for example, the insertion state detection unit 121 estimates the position of the distal end of the insertion part 11 based on the information about the movement of the support arm device 301 (arm 312) output from the arm control device 302, thereby detecting the insertion and removal of the insertion part 11.

[0177] Furthermore, for example, if an insertion mode for inserting the insertion part 11 into the living body and a removal mode for removing the insertion part 11 from the living body are set in the support arm device 301 or the arm control device 302, the insertion state detection unit 121 can detect the insertion and removal of the insertion part 11 based on the setting information output from the support arm device 301 or the arm control device 302 when setting the insertion mode and the removal mode.

[0178] Furthermore, as described above, the insertion and removal of the insertion part 11 can be detected by combining multiple detection methods (e.g., a method using the brightness of the captured image and a method using focus evaluation values). However, combining multiple detection methods may result in different detection results for each individual detection method. To address this, for example, the detection methods can be prioritized based on factors such as detection accuracy and computational efficiency. The detection result obtained by the detection method with higher priority can be adopted.

[0179] Furthermore, detection methods with low detection accuracy and low computational cost (e.g., using methods that capture the brightness of an image) can be used frequently, while detection methods with high detection accuracy and high computational cost (e.g., using image recognition methods) can be used intermittently. For example, if the removal of the insertion part 11 is detected by a detection method with low computational cost, detection processing using a detection method with high computational cost can be performed intermittently (e.g., every 10 seconds). For example, if the insertion of the insertion part 11 is detected by a method with high computational cost, but the insertion of the insertion part 11 is not detected by a method with low computational cost, the detection result with high detection accuracy can be used to determine that the insertion part 11 has been inserted. In this case, after the insertion of the insertion part 11 is detected, the detection processing performed by the detection method with high computational cost is stopped.

[0180] Furthermore, for example, if the insertion of the insertion part 11 is not detected within a predetermined time period (e.g., within five minutes) after the removal of the insertion part 11 is detected, the detection result can be considered to be incorrect, thereby determining that the insertion part 11 has been inserted.

[0181] <Variations on AF operation>

[0182] For example, the focusing control unit 122 can control the AF operation based on the surgical flow recognized by the image recognition unit 115, etc. For example, the focusing control unit 122 can control the AF operation based on the presence or absence of bleeding, the on / off state of the electrosurgical unit, etc.

[0183] Furthermore, for example, if the image recognition unit 115 detects blood, water droplets, dirt, or other substances adhering to the lens of the insertion part 11 based on the captured image, the focus control unit 122 can stop the AF operation.

[0184] <Examples of User Interface>

[0185] For example, when the insertion part 11 is inserted into a living body, continuous AF can be easily set.

[0186] For example, when a single-touch AF operation is performed while the insertion part 11 is inserted into a living body, continuous AF can be automatically switched on after the single-touch AF operation is performed. Note that in this case, the period during which continuous AF is automatically switched on by the single-touch AF operation can be limited to a predetermined time period after the insertion part 11 is inserted into the living body (e.g., within a few minutes).

[0187] Furthermore, for example, continuous AF operation and single-touch AF operation can be performed by the same operating device. For example, a short press of the operating button can perform single-touch AF, while a long press of the operating button can switch the continuous AF setting from on to off or off to on.

[0188] Furthermore, for example, the display unit 151 can display the operation status of continuous AF under the control of the display control unit 103. For example, during normal continuous AF operation, letters indicating the operation status (e.g., AF-C) in a specific color (e.g., green) can be displayed in the corner of the captured image. When continuous AF operation is restricted, these letters change to warn the operator that AF operation is restricted. For example, these letters can change to a different color that provides good contrast with the image, such as white, can be enlarged, can flash, etc. Alternatively or optionally, other types of warnings can be used, such as audible warnings, etc.

[0189] In addition, such as Figure 11 As shown, a steadily focused captured image 352 can be displayed in a picture-in-picture (PinP) format on top of the captured image 351 in real-time display. Note that in Figure 11 In the example, the image of the far port 201B of the insertion part 11 appears in the captured image 351.

[0190] For example, captured image 352 is the latest (real-time) image among captured images captured in a stable focus state (e.g., a state where the fluctuation range of the focus evaluation value is less than a predetermined threshold within a predetermined time period). In this case, if a stable focus state is achieved at the current point in time, captured images 351 and 352 can be matched.

[0191] Alternatively, for example, if the insertion part 11 has been removed from the living body, the captured image captured in a stable focusing state immediately before the removal of the insertion part 11 is displayed as captured image 352.

[0192] Note that the captured image 352 is displayed when the insertion part 11 is removed from the living organism, but not when the insertion part 11 is inserted into the living organism. Alternatively, the captured image 352 may be displayed separately when the insertion part 11 is removed from the living organism. In any case, for example, the display of the captured image 352 begins when the removal of the insertion part 11 is detected, and stops when the insertion of the insertion part 11 is detected.

[0193] <Variations on the imaging system configuration example>

[0194] The configuration example of the imaging system 1 described above is an example and can be a variant. For example, some functions of the camera head 14 can be transferred to the imaging control device 18, or some functions of the imaging control device 18 can be transferred to the camera head 14.

[0195] For example, one or more of the insertion state detection unit 121, focus control unit 122, and zoom control unit 123 of the imaging control device 18 may be disposed in the camera head 14. In this case, the camera head 14 controls the AF operation.

[0196] Furthermore, for example, some or all of the functions of the image processing unit 111 and the demodulation processing unit 112 of the imaging control device 18 may be provided in the camera head 14.

[0197] <<3. Others>>

[0198] <Computer Configuration Example>

[0199] The aforementioned series of processes can be performed by hardware or software. In the case of performing these processes by software, a program included in that software is installed in the computer. In this document, "computer" includes computers incorporated into dedicated hardware, such as general-purpose personal computers, on which various types of functions can be performed by installing various programs, etc. As used herein, "computer" refers to circuitry that can be configured via the execution of computer-readable instructions, and that circuitry may include one or more local processors (e.g., CPUs), and / or one or more remote processors, such as cloud computing resources, or any combination thereof.

[0200] Figure 12 This is a block diagram illustrating an exemplary configuration of the hardware of a computer that performs the series of processes described above by a program.

[0201] In a computer, a central processing unit (CPU) 1001, a read-only memory (ROM) 1002, and a random access memory (RAM) 1003 are interconnected via a bus 1004.

[0202] Bus 1004 is also connected to input / output interface 1005. Input / output interface 1005 is connected to input unit 1006, output unit 1007, storage unit 1008, communication unit 1009, and driver 1010.

[0203] Input unit 1006 includes a keyboard, mouse, microphone, etc. Output unit 1007 includes a display, speaker, etc. Storage unit 1008 includes a hard disk, non-volatile memory, etc. Communication unit 1009 includes a network interface, etc. Driver 1010 drives removable media 1011 including disks, optical disks, magneto-optical disks, semiconductor memories, etc.

[0204] On a computer with such a configuration, the above series of processes are performed by the following operation: the CPU 1001 loads the program stored in the storage unit 1008 into the RAM 1003 via, for example, the input / output interface 1005 and the bus 1004, and executes the program.

[0205] The program executed by the computer (CPU 1001) can be stored in the removable medium 1011 and provided, for example, as a packaging medium. Alternatively, the program can be provided via wired or wireless transmission media, including local area networks, the Internet, and digital satellite broadcasting.

[0206] On a computer, by attaching the removable medium 1011 to the drive 1010, a program can be installed in the storage unit 1008 via the input / output interface 1005. Alternatively, the program can be received at the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0207] Note that a program executed by a computer may be a program that is processed in a time series in the order described in this specification, or it may be a program that is processed in parallel or called at necessary time intervals, such as when necessary.

[0208] Furthermore, in this specification, "system" refers to a collection of multiple components (devices, modules (parts), etc.). In other words, all components can be housed in the same housing, but they do not necessarily have to be in the same housing. Therefore, multiple devices housed in separate housings and connected via a network can constitute a system. A device containing multiple modules in a single housing can also be a system.

[0209] Note that the embodiments of this technology are not limited to the above embodiments, but can be modified in various ways within the scope of this technology.

[0210] For example, this technology can be configured as a cloud computing platform, in which a function is collaboratively shared among multiple devices via a network for processing. Alternatively, this technology can be configured as a server or IP converter in a hospital, in which a function is collaboratively shared among multiple devices via a network for processing.

[0211] Furthermore, each step described in the flowchart above can be performed on a single device or shared by multiple devices for processing.

[0212] Furthermore, in cases where a step includes multiple processing stages, the multiple processing stages included in a step can be executed on a single device or shared by multiple devices for processing.

[0213] <Configuration Combination Example>

[0214] This technology can also be configured as follows.

[0215] (1) Endoscopic system, including:

[0216] An insertion state detection unit is configured to detect whether an insertion portion equipped with an optical system and connected to an imaging unit has been inserted into a living organism; and

[0217] The focus control unit is configured to control the focus control operation of the imaging unit based on the detection results of the insertion state.

[0218] (2) Based on the endoscopic system in (1),

[0219] The focusing control unit restricts focusing control operations when the insertion part is removed from inside the living organism.

[0220] (3) Based on the endoscope system in (2),

[0221] In the case where the insertion part is removed from the living body, the focusing control unit fixes the focal position of the imaging unit.

[0222] (4) Based on the endoscope system in (2),

[0223] In the case of removing the insert from inside the living organism, the focus control unit controls the focus control operation so that the operating range of the focal position of the imaging unit is within a predetermined range that is narrower than the range before the insert was removed.

[0224] (5) Based on the endoscope system in (2),

[0225] The focusing control unit fixes the focal position of the imaging unit at the position before the insertion part is removed from the inside of the living organism.

[0226] (6) Based on the endoscope system in (2),

[0227] The focusing control unit controls the operation of the focusing control, such that the operating range of the focal position of the imaging unit is within a predetermined range, including the position before the insertion part is removed from the inside of the living body, based on the position before the insertion part is removed from the inside of the living body.

[0228] (7) Based on the endoscope system in (2),

[0229] The focusing control unit fixes the focal position of the imaging unit at the object focusing position at a predetermined short distance, or, based on the object focusing position, narrows the operating range of the focal position to a narrower range than before the insertion part was removed.

[0230] (8) Based on the endoscope system in (2),

[0231] In cases where the insertion part is removed from the living organism, the focusing control unit stops focusing control operations.

[0232] (9) An endoscope system based on any one of (2) to (8),

[0233] When the insertion part is inserted into the living body, the focusing control unit removes the restriction on focusing control operation.

[0234] (10) Based on the endoscopic system in (9),

[0235] The focusing control unit removes the restriction on focusing control operation after a predetermined time period has elapsed since the insertion part was inserted into the living body.

[0236] (11) An endoscope system according to any one of (2) to (8),

[0237] Specifically, when the focus evaluation value used to assess the focus state of the captured image obtained by the imaging unit is stable, or when the brightness of the captured image is stable after the inserter is inserted into the living body, the focus control unit removes the restriction on the focus control operation.

[0238] (12) An endoscope system according to any one of (1) to (11),

[0239] The focusing control unit slows down the focusing control operation speed for a predetermined period of time after the insertion part is inserted into the living body.

[0240] (13) An endoscope system according to any one of (1) to (12),

[0241] The insertion state detection unit detects the insertion of the insertion part into the insertion port of the living organism and the removal of the insertion part from the insertion port.

[0242] (14) An endoscope system according to any one of (1) to (13),

[0243] The insertion state detection unit detects the insertion and removal of the insertion part based on the captured image obtained by the imaging unit.

[0244] (15) Based on the endoscopic system of (14),

[0245] The insertion state detection unit detects the insertion and removal of the insertion part based on the changes in brightness of the captured image.

[0246] (16) Based on the endoscopic system of (15),

[0247] The insertion state detection unit detects the insertion and removal of the insertion part based on the brightness change of the surrounding part of the captured image.

[0248] (17) An endoscope system according to any one of (14) to (16),

[0249] The insertion state detection unit detects the insertion and removal of the insertion part based on changes in the focus evaluation value used to evaluate the focus state of the captured image.

[0250] (18) An endoscope system according to any one of (14) to (17),

[0251] The insertion state detection unit detects the insertion and removal of the insertion part based on the color changes of the captured image.

[0252] (19) An endoscope system according to any one of (13) to (18) further includes

[0253] The image recognition unit identifies the insertion port in the image captured by the imaging unit.

[0254] The insertion state detection unit detects the insertion and removal of the insertion part based on the shape of the insertion port in the captured image.

[0255] (20) An endoscope system according to any one of (1) to (19),

[0256] The insertion state detection unit acquires measurement distance information representing the distance to the object, and detects the insertion and removal of the insertion part based on changes in the measurement distance information.

[0257] (21) An endoscope system according to any one of (1) to (20) further includes

[0258] The position estimation unit is configured to estimate the position of at least one of the imaging unit and the insertion unit.

[0259] The insertion state detection unit detects the insertion and removal of the insertion part based on the positional changes of at least one of the imaging unit and the insertion part.

[0260] (22) An endoscope system according to any one of (1) to (21),

[0261] The insertion state detection unit detects the insertion and removal of the insertion part based on the output of the arm that controls the position and orientation of the imaging unit.

[0262] (23) An endoscope system according to any one of (1) to (22),

[0263] The insertion state detection unit detects the insertion and removal of the insertion part based on the change in acceleration of at least one of the imaging unit and the insertion part.

[0264] (24) An endoscope system according to any one of (1) to (23),

[0265] The insertion state detection unit detects the insertion and removal of the insertion part based on the change in tilt of at least one of the imaging unit and the insertion part.

[0266] (25) An endoscope system according to any one of (1) to (24) further includes

[0267] The display control unit is configured to control the display unit to display a first captured image obtained by the imaging unit before the insertion part is removed from the living body when the insertion part is removed from the living body.

[0268] (26) Based on the endoscopic system of (25),

[0269] The display control unit controls the display unit to display the first captured image together with the real-time second captured image obtained by the imaging unit.

[0270] (27) Based on the endoscopic system in (26),

[0271] In the case where the insertion part is inserted from inside the living body, the display control unit controls the display unit to stop displaying the first captured image.

[0272] (28) An endoscope system according to any one of (1) to (27),

[0273] In the case where the insertion part is inserted into the living body, when the focusing control operation is performed as a point operation, the focusing control unit performs control to continuously perform focusing control.

[0274] (29) An endoscope control method, comprising:

[0275] The insertion status is detected, indicating whether the insertion part, equipped with an optical system and connected to the imaging unit, has been inserted into the living organism; and

[0276] The focusing control operation of the imaging unit is controlled based on the detection results of the insertion state.

[0277] (30) An imaging control device, comprising:

[0278] An insertion state detection unit is configured to detect whether an insertion portion equipped with an optical system and connected to an imaging unit has been inserted into a living organism; and

[0279] The focus control unit is configured to control the focus control operation of the imaging unit based on the detection results of the insertion state.

[0280] <Variations on the configuration combination example>

[0281] This technology can also be configured as follows.

[0282] (31) An endoscope system comprising:

[0283] The circuit is configured as follows:

[0284] Features of a monitoring endoscope, the endoscope including an imaging device and a scleral device coupled to the imaging device, wherein features indicate whether the scleral device is inserted into a living body;

[0285] When the features meet predetermined conditions, the focus of the endoscope system is controlled in a first manner; and

[0286] If the features do not meet the predetermined conditions, the focus of the endoscope system is controlled in a second manner different from the first manner.

[0287] (32) The endoscope system according to (31), wherein the first mode includes the focus of the autofocusing endoscope system.

[0288] (33) The endoscope system according to (31), wherein the first method includes setting the focus of the endoscope system to a fixed focus position.

[0289] (34) The endoscope system according to (33) wherein the fixed focal position is set to the focal position just before the feature satisfies the predetermined conditions.

[0290] (35) An endoscope system according to any one of (31) to (34), wherein the features are features of an image captured by an image sensor via a mirror device.

[0291] (36) According to the endoscope system of (35), the characteristic is the color of the image.

[0292] (37) According to the endoscope system of (35), the characteristic is the color of the peripheral part of the image.

[0293] (38) According to the endoscope system of (35), the characteristic is the brightness of the image.

[0294] (39) The endoscope system according to claim (35), wherein the feature is the fluctuation of brightness in the peripheral portion of the image within a predetermined time period.

[0295] (40) An endoscope system according to any one of (31) to (39), wherein the feature is the position of the insertion tube of the endoscope system.

[0296] (41) An endoscope system according to any one of (31) to (40), wherein the characteristic is the tilt of the camera head of the endoscope system.

[0297] (42) An endoscope system according to any one of (31) to (41), wherein the characteristic is the acceleration of the camera head of the endoscope system.

[0298] (43) An endoscope system according to any one of (31) to (42), wherein the feature is that the shape of the distal port in the image is captured by an image sensor via a mirror device.

[0299] (44) An endoscope system according to any one of (31) to (43), wherein the feature is the focus evaluation value of the endoscope system.

[0300] (45) An endoscope system according to any one of (31) to (44), wherein the feature is setting information from the support arm of the endoscope system.

[0301] (46) An endoscope system according to any one of (31) to (45), wherein the circuitry is further configured to display a first captured image obtained by the imaging device before the features meet the predetermined conditions, provided that the features meet the predetermined conditions.

[0302] (47) The endoscope system according to (46), wherein the circuit is further configured to stop displaying the first captured image and display a real-time captured image if the features do not meet predetermined conditions.

[0303] (48) An endoscope system according to any one of (31) to (47), wherein, after a feature satisfies a predetermined condition and then stops satisfying the predetermined condition, the circuit gradually controls the focus of the endoscope system in a second manner.

[0304] (50) A non-transitory computer-readable medium on which a program is stored, which, when executed by a computer, causes the computer to perform a process comprising the following steps:

[0305] The monitoring system will identify the features of the insertion site, which will be inserted into and removed from the living body, and will be connected to an image sensor that generates images in a medical imaging system.

[0306] When the features meet predetermined conditions, the focus of the medical imaging system is controlled in a first manner; and

[0307] If the features do not meet the predetermined conditions, the focus of the medical imaging system is controlled in a second manner different from the first manner.

[0308] (60) A method comprising:

[0309] The monitoring system will identify the features of the insertion site, which will be inserted into and removed from the living body, and will be connected to an image sensor that generates images in a medical imaging system.

[0310] When the features meet predetermined conditions, the focus of the medical imaging system is controlled in a first manner; and

[0311] If the features do not meet the predetermined conditions, the focus of the medical imaging system is controlled in a second manner different from the first manner.

[0312] Note that the effects described here are provided for illustrative purposes only and are not intended to be limiting. Other effects may also be considered.

[0313] Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alterations can occur depending on design requirements and other factors, as long as they are within the scope of the appended claims or their equivalents.

[0314] List of reference numerals

[0315] 1 Imaging System

[0316] 11 Insertion section

[0317] 14-camera lens

[0318] 18 Imaging Control Devices

[0319] 51 lens units

[0320] 52 Lens Drive Section

[0321] 53 Lens Position Detection Unit

[0322] 54 imaging processing units

[0323] 61 Focusing Lens

[0324] 71 actuator

[0325] 102 Signal Processing Unit

[0326] 103 Display Control Unit

[0327] 104 control unit

[0328] 111 Image Processing Unit

[0329] 112 Demodulation Processing Unit

[0330] 113 location estimation unit

[0331] 114 distance measurement units

[0332] 115 Image Recognition Unit

[0333] 121 Insertion Status Detection Unit

[0334] 122 Focusing Control Unit

[0335] 151 display units

[0336] 201 cannula

[0337] 201A Insert Port

[0338] 201B Remote Port

[0339] 301 support arm device

[0340] 302 Arm Control Device

[0341] 312 Arm.

Claims

1. An endoscope system comprising: circuitry configured to: monitor a feature of an endoscope, the endoscope including an imaging device and a scope coupled to the imaging device, wherein the feature is a feature of an image captured by an image sensor via the scope, and indicates whether the scope is inserted into a living body; control a focus of the endoscope system in a first manner if the feature satisfies a predetermined condition; and control the focus of the endoscope system in a second manner different from the first manner if the feature does not satisfy the predetermined condition; wherein the predetermined condition includes that the scope is detected to be inserted into the living body, and a focus evaluation value or a brightness of the captured image has stabilized, wherein the first manner includes autofocusing the focus of the endoscope system, and the second manner includes setting the focus of the endoscope system to a fixed focus position.

2. The endoscope system of claim 1, wherein, the fixed focus position is set to a focus position just before the feature satisfies the predetermined condition.

3. The endoscope system of claim 1, wherein, the feature is a color of the image.

4. The endoscope system of claim 1, wherein, the feature is a color of a peripheral portion of the image.

5. The endoscope system of claim 1, wherein, the feature is a brightness of the image.

6. The endoscope system of claim 1, wherein, the feature is a fluctuation in brightness in a peripheral portion of the image within a predetermined period of time.

7. The endoscope system of claim 1, wherein, the feature is a position of an insertion tube of the endoscope system.

8. The endoscope system of claim 1, wherein, the feature is a tilt of a camera head of the endoscope system.

9. The endoscope system of claim 1, wherein, the feature is an acceleration of the camera head of the endoscope system.

10. The endoscope system of claim 1, wherein, the feature is a shape of a distal port in an image captured by an image sensor via the scope.

11. The endoscope system of claim 1, wherein, the feature is a focus evaluation value of the endoscope system.

12. The endoscope system of claim 1, wherein, the feature is setting information from a support arm of the endoscope system.

13. The endoscope system of claim 1, wherein, the circuitry is further configured to, if the feature satisfies the predetermined condition, display a first captured image obtained by the imaging device just before the feature satisfies the predetermined condition.

14. The endoscope system of claim 13, wherein, the circuitry is further configured to, if the feature does not satisfy the predetermined condition, stop displaying the first captured image and display a real-time captured image.

15. The endoscope system of claim 1, wherein, after the feature satisfies the predetermined condition and then stops satisfying the predetermined condition, the circuitry gradually controls the focus of the endoscope system in the second manner.

16. A non-transitory computer readable medium having stored thereon a program which, when executed by a computer, causes the computer to perform a process comprising: monitoring a feature of an insertion portion to be inserted into and removed from a living body, the insertion portion to be connected to an image sensor that generates an image in a medical imaging system, the feature being a feature of an image captured by an image sensor via a scope of the insertion portion; controlling a focus of the medical imaging system in a first manner if the feature satisfies a predetermined condition; and controlling the focus of the medical imaging system in a second manner different from the first manner if the feature does not satisfy the predetermined condition. ​ wherein the predetermined condition comprises that the mirror device is detected to be inserted into the living body and a focus evaluation value or brightness of the captured image has been stabilized, wherein the first mode comprises automatically focusing the focal point of the medical imaging system and the second mode comprises setting the focal point of the medical imaging system to a fixed focal point position.

Images