Endoscope focus control device, focus control method, and focus control program

Abstract

This endoscope focus control device is provided with: an image feature assessment unit that assesses a feature of an image obtained by an imaging device provided at a distal end section of an endoscope; and a determination unit that assesses an observation target site to be observed by an operator from the image on the basis of a positional change over time in the captured image of a feature section in the image detected by the image feature assessment unit and the consistency of operation content in which the operator changes the imaging range of the imaging device with respect to the endoscope, and determines the observation target site as a focus target.

Description

Endoscope focus control device, focus control method, and focus control program

[0001] The present invention relates to an endoscope focus control device, a focus control method, and a focus control program that facilitate focusing.

[0002] An endoscope is a device that is inserted into the body and enables observation of diseased areas that cannot be seen from the outside. An endoscope has a long, thin, flexible insertion section that is inserted into, for example, a patient's body cavity. The endoscope captures images of various parts of the body using an imaging element provided at the tip of the insertion section. The images captured by the endoscope (endoscopic images) are supplied to a video processor and processed. The endoscopic images from the video processor are displayed on a monitor display screen.

[0003] Although the captured image is obtained by a pan-focus optical system, some endoscopes have an autofocus control function to obtain an image in focus depending on the distance to the object. An example of an autofocus control technology applied to digital cameras is disclosed in Japanese Patent Laid-Open Publication No. 2005-31200 (hereinafter referred to as Patent Document 1). In the technology of Patent Document 1, if the subject is lost during pattern-driven AF control (normal control state), pattern-driven AF control (extended control state) continues until at least a predetermined time has elapsed.

[0004] Japanese Patent Application Laid-Open No. 2005-31200

[0005] However, with an endoscope, it is relatively difficult to capture a desired observation site within the imaging range. Therefore, even if the technology of Patent Document 1 is used, it is not always possible to reliably focus on the desired observation site. An object of the present invention is to provide an endoscope focus control device, focus control method, and focus control program that can reliably focus on the desired observation site by performing autofocus control using an operation to, for example, track and observe the observation site.

[0006] An endoscope focus control device according to one aspect of the present invention includes an image feature determination unit that determines the features of an image obtained by an imaging device provided at the tip of the endoscope, and a determination unit that determines the observation target area that the operator intends to observe from the image based on the positional change over time in the captured image of the image feature detected by the image feature determination unit and the consistency of the operation content of the operator using the endoscope to change the imaging range of the imaging device, and determines the observation target area as the target for focusing.

[0007] A focus control method according to one aspect of the present invention determines the characteristics of an image obtained by an imaging device provided at the tip of an endoscope, determines the area to be observed that the operator intends to observe from the image based on the positional change over time in the captured image of the characteristic part of the image detected by the determination and the consistency between the operation content of the operator using the endoscope to change the imaging range of the imaging device, and determines the area to be observed as the target for focusing.

[0008] A focus control program according to one aspect of the present invention causes a computer to execute the following steps: determine the characteristics of an image obtained by an imaging device provided at the tip of an endoscope; determine the observation area that the operator intends to observe from the image based on the positional change over time in the captured image of the characteristic part of the image detected by the determination and the consistency between the operation content of the operator using the endoscope to change the imaging range of the imaging device; and determine the observation area as the target for focusing.

[0009] An endoscope focus control device according to another aspect of the present invention includes a determination unit that determines an operation performed by an operator on the endoscope to change the imaging range of the imaging device based on time-series changes in images obtained sequentially in time series by an imaging device provided at the tip of the endoscope, and a determination unit that determines an observation target area that the operator intends to observe from the image based on the determination result of the operation, and determines an area including the observation target area within the imaging range of the imaging device as a focus area to be focused.

[0010] A focus control method according to another aspect of the present invention determines the operation content of an operator of an endoscope that changes the imaging range of the imaging device based on time-series changes in images obtained sequentially in time series by an imaging device provided at the tip of the endoscope, determines the observation target area that the operator intends to observe from the image based on the determination result of the operation content, and determines an area including the observation target area within the imaging range of the imaging device as a focus area to be focused.

[0011] A focus control program according to another aspect of the present invention causes a computer to execute the following steps: determine the operation content of an operator of the endoscope to change the imaging range of the imaging device based on time-series changes in images obtained sequentially in time series by an imaging device provided at the tip of the endoscope; determine the observation target area that the operator intends to observe from the image based on the determination result of the operation content; and determine an area including the observation target area within the imaging range of the imaging device as a focus area to be focused on.

[0012] An endoscope focus control device according to another aspect of the present invention includes an image feature determination unit that determines the features of an image obtained by an imaging device provided at the tip of the endoscope; a determination unit that determines the operation content of an operator of the endoscope to change the imaging range of the imaging device based on a position change within the captured image of the image feature portion detected by the image feature determination unit; and a determination unit that determines an observation target area that the operator intends to observe from the image based on the determination result of the operation content, and determines an area including the observation target area that includes the image feature portion within the imaging range of the imaging device as a focus area to be focused.

[0013] A focus control method according to another aspect of the present invention determines the characteristics of an image obtained by an imaging device provided at the tip of an endoscope, determines the operation content of the operator using the endoscope to change the imaging range of the imaging device based on the position change within the captured image of the image characteristic part detected by the determination, determines the observation target area that the operator intends to observe from the image based on the determination result of the operation content, and determines an area within the imaging range of the imaging device that includes the observation target area including the image characteristic part as the focus area to be focused.

[0014] A focus control program according to another aspect of the present invention causes a computer to execute the following steps: determine the characteristics of an image obtained by an imaging device provided at the tip of an endoscope; determine the operation content of an operator using the endoscope to change the imaging range of the imaging device based on a position change within the captured image of the image characteristic part detected by the determination; determine an observation target area that the operator intends to observe from the image based on the determination result of the operation content; and determine an area including the observation target area that includes the image characteristic part within the imaging range of the imaging device as a focus area to be focused.

[0015] According to the present invention, by determining the autofocus target according to the observation target area, for example, by utilizing an operation to follow and observe it, and performing control thereon, it is possible to reliably focus on the desired observation target area.

[0016] Fig. 1 is a block diagram showing an endoscope focus control device according to an embodiment of the present invention; Fig. 2 is a flowchart for explaining the operation of the embodiment; Fig. 3 is an explanatory diagram showing the state of an endoscopic examination; Fig. 4 is an explanatory diagram showing the state of an endoscopic examination; Fig. 5 is a flowchart showing an example of a specific operation flow of step S4 in Fig. 2; Fig. 6 is an explanatory diagram for explaining an imaging range indicated by a thick dashed line; Fig. 7 is an explanatory diagram for explaining an example when multiple object patterns exist and when the observation target portion does not have a specific pattern;

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0018] 1 is a block diagram showing an endoscope focus control device according to one embodiment of the present invention. This embodiment determines the operation of the endoscope to capture an observation target region, for example, at the center of the imaging range, and performs autofocus control based on the determination result, thereby enabling accurate focusing on the desired observation target region.

[0019] Endoscopes are required not only for imaging objects with clear features, such as faces, but also for imaging objects with unclear features. Therefore, even if autofocus technology, such as that used in cameras or smartphones, is adopted to focus on specific objects like faces, it is not always possible to reliably focus on the desired observation target area. Furthermore, when imaging with an endoscope, it is not possible to move internal organs or other objects to obtain an appropriate composition. Not only is it not possible to image the observation target area with the optimal composition, but it may also be difficult to easily position it within the imaging range.

[0020] A characteristic of endoscopes is that the control section and the imaging section (the imaging device 22, described below), which is often located near the tip, are located far apart, preventing the operator's movements from being directly transmitted to the tip of the insertion section. Furthermore, there is little feedback on the tip's movement in response to the operation, making intuitive operation difficult. Furthermore, the insertion section moves while in contact with internal structures. However, the imaging range of the imaging section located at the tip of the insertion section does not capture the state in which the insertion section is in contact with the structure. Therefore, even when viewing captured images, it is impossible to identify factors impeding the movement of the insertion section, making the movement of the insertion section itself difficult. This severely restricts framing during endoscopic imaging. In other words, the operator may not be able to capture the object they want to see as intended, making focusing difficult and resulting in significant time-consuming observations and examinations. While this situation is particularly prevalent with flexible endoscopes, similar problems can also occur with rigid endoscopes, which have a smaller flexible section.

[0021] Therefore, in this embodiment, it is possible to reflect the operator's intention and focus on a desired observation target area.

[0022] 1, the endoscope focus control device includes an observation target region determination unit 10 to which an endoscopic image from an endoscope 20 is input. Although a processing circuit for displaying the endoscopic image is not shown in Fig. 1, an endoscopic image whose focus has been controlled by the endoscope focus control device of this embodiment is sent to a video processor (not shown) for image processing, thereby making it possible to display an endoscopic image focused on the observation target region on a monitor.

[0023] The endoscope 20 has a thin, flexible insertion section that is inserted into a body cavity of a subject. A tip section 21 is provided at the tip of the insertion section. A control section 25, on which various controls are arranged, is provided at the proximal end of the insertion section. The insertion section has a bending section (not shown) that is composed of multiple bending pieces and is freely bendable. By operating the control section 25 to drive the bending section, the orientation of the tip section 21 can be adjusted in any direction. The control section and the bending section are connected by multiple wires or the like, and pulling or loosening these wires is transmitted to the tip. However, in recent years, types have been proposed in which an actuator at the tip is electrically connected and operated by a switch. If the control section is, for example, a lever or knob, the degree of bending can be changed by the amount of rotation or operation. This amount of rotation or operation may be detected electrically using a sensor or the like, or the amount of movement corresponding to the amount of rotation or operation may be detected from changes in the image obtained as an image result by an imaging section (imaging device 22) at the tip. Since the operator performs the operation while viewing this image, the image changes (changes over time) in accordance with the operator's intention.

[0024] An imaging device 22 is provided at the distal end 21 of the insertion portion. The imaging device 22 has an imaging element 23 and an optical system 24 that guides an optical image of a subject to the imaging surface of the imaging element. The imaging element is composed of a CCD, CMOS sensor, or the like, and photoelectrically converts the optical image of the subject from the optical system 24 to obtain an image of the subject (image signal). The optical system 24 is equipped with lenses and apertures (not shown) for zooming and focusing, and is equipped with a zoom (magnification change) mechanism, focus, and aperture mechanisms (not shown) that drive these lenses. Image information of the images (endoscopic images) obtained sequentially in time series by the imaging device 22 is supplied to the observation target region determination unit 10.

[0025] In an examination or treatment using the endoscope 20, a physician operates the endoscope 20 to insert the insertion portion of the endoscope 20 into a lumen of the human body. The physician operates a bending knob (not shown) or the like provided on the operation unit 25 to bend the bending portion or move the insertion portion forward or backward, thereby moving the tip portion 21 of the insertion portion to the site to be observed. In an endoscopic examination, the insertion portion needs to be inserted according to the state of the lumen, which may be narrow or curved, and it is relatively difficult to move the tip portion 21 to the desired position and orient it in the desired direction. For this reason, complex operations may be required for the physician to focus on the desired site to be observed and obtain an observation image of good quality.

[0026] Therefore, the observation target portion determination unit 10 determines the observation target portion by determining the characteristics of the image obtained by the imaging device 22, and by determining changes over time in the characteristics, and by determining the operation by the operator on the tip portion 21 where the imaging device 22 is disposed. In this embodiment, the determination result of the observation target portion determination unit 10 is used to automatically control the focus of the imaging device 22.

[0027] The observation target portion determination unit 10 includes an image determination unit 11, an operation determination unit 12, a focus area determination unit 13, and a temporary recording unit 16. Each unit of the observation target portion determination unit 10 may be configured by a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc. Each unit of the observation target portion determination unit 10 may operate and control each unit according to a program stored in a memory (not shown), or may realize some or all of its functions using a hardware electronic circuit.

[0028] The image from the endoscope 20 is provided to the temporary recording unit 16 and image determination unit 11 of the observation target region determination unit 10. The temporary recording unit 16 is configured with a predetermined memory medium and temporarily records the endoscopic image. The image determination unit 11 performs various image analysis processes on the endoscopic image to sequentially determine the characteristics of each part in the image, such as various patterns in the endoscopic image. The image determination unit 11 provides the determination results to the operation determination unit 12 and the focus area determination unit 13.

[0029] The operation determination unit 12 includes an operation content determination unit 14 and an object determination unit 15. Based on the determination result of the image determination unit 11, the operation content determination unit 14 detects time-series changes in the features of each part in the endoscopic image (e.g., the pattern of a lesion, etc.) and determines the operation content of the operator that changes the imaging range of the imaging device 22. That is, the operation content determination unit 14 determines the operation content of the operator that changes the imaging range of the imaging device 22 to move or change the orientation of the distal end portion 21 of the endoscope. Note that the operator moves the entire endoscope 20 or operates levers, switches, etc. (not shown) of the operation unit 25 to move or change the orientation of the distal end portion 21.

[0030] For example, when an endoscope is inserted into a lumen, the tip 21 of the endoscope 20 moves, and the endoscopic image changes so that the image pattern at the center of the image moves toward the periphery. When the endoscope is removed, the endoscopic image changes so that the image pattern at the periphery moves toward the center. Furthermore, with flexible endoscopes, in addition to insertion and removal operations, an operation to bend the tip may also be performed. During this bending operation, the image pattern in the endoscopic image changes position in the up, down, left, and right directions on the XY two-dimensional coordinate system of the image over time.

[0031] The operation content determination unit 14 determines the changes in each part in the images obtained sequentially in time series by comparing previous and next images and analyzing motion vectors based on the determination results of the image determination unit 11, and determines the operation content including the type and amount of operation performed by the surgeon, such as inserting or removing the insertion unit (tip portion 21) or bending the tip portion 21 to change its orientation. Note that the operation content determination unit 14 may determine the operation content through AI (artificial intelligence) processing using an inference model for determining the operation content.

[0032] Note that the endoscopic image changes not only due to the movement of the insertion portion (tip portion 21) but also due to the movement of the subject, such as an organ, etc. However, the image changes due to the movement of the insertion portion and the image changes due to the movement of the organ, etc. in different ways, and the operation content determination unit 14 can detect the image changes due to the movement of the insertion portion by image analysis.

[0033] The target determination unit 15 determines the target area to be observed that the surgeon wishes to observe based on the determination result of the image determination unit 11 and the determination result of the operation content of the surgeon by the operation content determination unit 14. For example, depending on the operation content of the surgeon, the target determination unit 15 may determine that the area captured in the center of the image is the target area to be observed. Also, for example, if the operation content of the surgeon indicates a specific direction, the target determination unit 15 may determine that the area located in that direction is the target area to be observed. Also, for example, if a lesion is captured in the image, the target determination unit 15 may determine that the lesion is the target area to be observed.

[0034] Due to the framing constraints described above, it is not always possible to capture the observation target region that the surgeon wishes to observe at the center of the image. However, since the operator will devise operations to position the observation target region at the center of the image, the target determination unit 15 may identify the observation target region by determining such operations based on the determination result of the operation content determination unit 14.

[0035] Although the example in which the operation determination unit 12 determines the operation content and the observation target region based on the endoscopic image has been described, the operation determination unit 12 may determine the operation content and the observation target region based on other information. In this embodiment, the operation detection unit 26 acquires operation information based on the operation of the operation unit 25. The operation determination unit 12 may determine the operation content of the surgeon and the observation target region using the operation information from the operation detection unit 26.

[0036] The operation detection unit 26 may acquire operation information using various sensors. For example, the operation detection unit 26 may acquire operation information by detecting the amount of operation of a curved knob provided on the operation unit 25. Alternatively, for example, the operation detection unit 26 may acquire operation information by detecting the insertion length of the insertion unit. Alternatively, the operation detection unit 26 may acquire operation information by detecting an operation of the operator to move and change the orientation of the distal end portion 21 using a known endoscope insertion shape observation device that detects the insertion shape of the insertion unit using a transmitting / receiving coil provided at the distal end portion 21 and an external fixed position.

[0037] Furthermore, the operation detection unit 26 may perform voice recognition processing on the voice acquired using a microphone that acquires ambient voice to acquire the operation content of the operator that changes the imaging range of the imaging device 22. If the operator utters, for example, "more to the right" when moving the tip end portion 21, the operation detection unit 26 can acquire the operation content from the content of the uttered voice.

[0038] The operation determination unit 12 may always determine the operation content and the observation target region during endoscopic imaging, or may perform these determinations only during a specific period. For example, the operation determination unit 12 may perform the determination when an attempt is made to transition from a state in which the endoscope is performing screening to a state in which a specific region (observation target region) is observed. The operation determination unit 12 may also perform the determination when approaching the observation target region and zooming in on the specific region (observation target region) to observe the region. The operation determination unit 12 may determine whether to perform the determination of the operation content and the observation target region based on the determination result of the image determination unit 11 or the operation information of the operation detection unit 26. For example, the operator may operate the operation unit 25 to instruct the start of the determination. If the operation determination unit 12 does not perform the determination, it may output a signal indicating that focus control is not required to the focus control unit 30. That is, as will be described later, when the operation determination unit 12 obtains a determination result of the operation content indicating that the state has transitioned from endoscopic screening to a state of observing the above-mentioned observation target area, focus control for the focus area may be performed.

[0039] The focus area determination unit 13 is given the determination result from the image determination unit 11 and the estimation result of the observation target region from the operation determination unit 12, and sets a focus area, which is a target region for focusing, in the endoscopic image. The focus area indicates the region for focusing within the imaging range of the imaging device 22. The focus area determination unit 13 determines that the region within the imaging range in which the observation target region obtained by the target determination unit 15 exists is the focus area to be set. Information on the focus area determined by the focus area determination unit 13 is given to the focus control unit 30.

[0040] The focus control unit 30 controls the imaging device 22 so that the focus area set by the focus area determination unit 13 is brought into focus. The focus control unit 30 controls the focusing mechanism of the optical system 24 to bring the focus area into focus. Note that the focus control unit 30 does not perform focus control during a period in which the operation determination unit 12 indicates that focus control is not necessary. Note that the focus control unit 30 may be configured to prioritize focusing on the specified focus area over other areas, or may assign a predetermined weight to focusing on the focus area.

[0041] Next, the operation of the embodiment configured as above will be described with reference to Figures 2 to 7. Figure 2 is a flowchart for explaining the operation of the embodiment, and Figures 3 and 4 are explanatory diagrams showing the state of endoscopic examination. Figure 5 is a flowchart showing an example of a specific operation flow of step S4 in Figure 2.

[0042] In the following description, the act of imaging a relatively wide range to search for a lesion or the like is referred to as "screening," and the act of imaging a specific area in detail by, for example, imaging close to the specific area or zooming in to capture the specific area for confirmation is referred to as "observation." For example, the following description will be given assuming that pan focus or autofocus control at the center of the imaging range is performed during screening, and focus control is performed by the operation determination unit 12 during observation, but this is not limited to this.

[0043] 3 shows an examination performed by an endoscopic system 40 including an observation target portion determination unit 10. An operator OP inserts the insertion portion 21a of an endoscope 20 into a lumen L. The endoscope 20 is connected to a video processor VP and the observation target portion determination unit 10 via a cable 42. An endoscopic image from an imaging device 22 (see FIG. 1) disposed at the tip 21 of the endoscope 20 is supplied to the video processor VP and the observation target portion determination unit 10. The video processor VP performs predetermined image signal processing on the input endoscopic image, and then supplies the endoscopic image to a monitor 41 to display it.

[0044] 2, imaging, display, and image evaluation are started. That is, imaging is started by the imaging device 22 disposed at the distal end 21 of the insertion section 21a, an endoscopic image is supplied to the video processor VP, and an image of the inside of the lumen L is displayed on the monitor 41. When screening starts, imaging is performed, for example, with a pan focus (S2).

[0045] In the example of Fig. 3, the inside of the lumen L is imaged by the imaging device 22 disposed in the tip portion 21 of the insertion section 21a inserted into the lumen L, and an image P1 of the inside of the lumen L is displayed on the display screen of the monitor 41. Note that the image P1a shows a dark area deep within the lumen L. The operator OP screens the lumen L by moving the tip portion 21 while viewing the endoscopic image displayed on the monitor 41, as indicated by the arrow representing the line of sight.

[0046] The operation determination unit 12 determines whether focus control is necessary in S3. For example, assume that an operation is performed to observe a specific area (observation target area) from an endoscopic screening state, or to capture an image of the observation target area from a close position and take a close-up. In this case, the operation determination unit 12 determines that focus control is necessary, for example, based on the determination result of the image determination unit 11. Then, focus control is performed in S4. If the operation determination unit 12 determines that focus control is not necessary (NO in S3), it proceeds to S5 and determines whether the endoscopic examination has ended.

[0047] The operation determination unit 12 may determine whether a desired part to be observed exists and whether focus control is necessary, taking into consideration the time for which the specific part is being imaged. For example, when the same part has been imaged for one second or more, the operation determination unit 12 may determine that the part to be observed exists within the imaging range and that focus control is necessary.

[0048] Fig. 4 shows an example of observation. Assume that the operator OP is performing screening by inserting the insertion section 21a deep into the lumen L. Assume that the operator OP has discovered a site deep within the lumen L that may be a lesion OL, and approaches the site from the screening state, for example by zooming in, to enlarge and observe the site (the site to be observed). In Fig. 4, an endoscopic image P2 immediately before transitioning to observation is displayed on the display screen of the monitor 41. An image P2a of a dark area deep within the lumen L is displayed in the center of the endoscopic image P2, and an image P2b of the lesion OL is displayed at the edge of the image.

[0049] The operator OP advances the distal end portion 21 deeper into the lumen L and operates the bending knob 25a to orient the distal end portion 21 in the direction indicated by the arrow in Figure 4, attempting to position the lesion OL at the center of the imaging range. However, as in the example of Figure 4, there are cases where the insertion section 21a comes into contact with a protrusion or fold on the lumen wall of the lumen L, making it difficult to orient the distal end portion 21 toward the lesion OL. Even in such cases, in this embodiment, the lesion OL can be imaged in focus and displayed on the monitor 41. In some cases, it is also possible to enlarge and display the image of the lesion OL at the center of the image.

[0050] That is, the operation determination unit 12 determines that focus control is necessary using the determination result of the image determination unit 11 and the operation information of the operation detection unit 26 (YES in S3). The operation determination unit 12 determines the operation content in S4. For example, the operation content determination unit 14 obtains operation content that indicates that the operator OP has devised an operation to position the object at the center of the image (the center of the imaging range). The object determination unit 15 determines that the object is the observation target site. In this case, the focus area determination unit 13 sets the area where the object is likely to be present as the focus area. In this way, the focus control unit 30 controls the optical system 24 of the imaging device 22 so that the focus area is focused. As a result, the captured image of the lesion OL is focused, and the image P2b on the monitor 41 becomes an image that is easy to view.

[0051] If the endoscopic examination has not ended (NO in S5), the process returns to S2 and screening continues. If it is determined that the endoscopic examination has ended (YES in S5), the process ends.

[0052] Next, an example of the focus control in step S4 will be specifically described with reference to FIGS.

[0053] In the following description, an example will be described in which autofocus control is performed by determining an operation to move the imaging range in accordance with the movement of the observation target portion toward the periphery of the imaging range and position the observation target portion in the center of the imaging range, but the transition to autofocus control is not limited to this case. For example, autofocus control may be performed based on an operation to position the observation target portion located on the periphery of the imaging range in the center of the imaging range, or may be performed based on a zoom operation or the like for observing the observation target portion located in the center of the imaging range.

[0054] 5, imaging by the endoscope 20 and determination by the observation target portion determination unit 10 begin in S11. The endoscopic image acquired by the endoscope 20 is temporarily recorded in the temporary recording unit 16. The imaging device 22 is normally autofocus (AF) controlled so as to focus at the center of the imaging range (S12). In S13, the operation content determination unit 14 determines whether the object pattern X has moved to the periphery of the imaging range while the tip portion 21 is fixed, i.e., while the operator OP is not performing any operation on the endoscope.

[0055] As described above, the image of the endoscopic image changes depending on the movement of the distal end 21 and the movement of the organ, which is the subject of the image. In this embodiment, the focus area is determined by determining the operation of the operator OP, which follows the movement of the observation target area. In S13, the operation determination unit 12 determines whether or not the image has changed due to the movement of the organ, rather than due to the operation of the operator OP. If the image change is due to the operation of the operator OP (NO in S13), the process proceeds to S20. If the change in the object pattern X is due to the movement of the organ, the operation determination unit 12 determines the operation content in S14.

[0056] Fig. 6 is an explanatory diagram illustrating the imaging range in this case using a thick dashed line. An object pattern X such as a lesion exists in the lumen wall PL, which is the subject, and Fig. 6 shows a state in which the object pattern X has moved to the upper right edge of the imaging range IRb due to movement of an organ or the like. In order to observe the object pattern X, the operator OP performs various operations to position the object pattern X in the center of the imaging range. The operation content determination unit 14 determines the operation content of such an operation by the operator OP.

[0057] For example, the operator OP operates the operation unit 25 to bend the bending portion of the insertion unit 21a, thereby changing the imaging range in accordance with the movement of the object pattern X in the captured image. The operation content determination unit 14 determines, based on the image change in this case, that the operator OP has performed an operation of operating the bending knob 25a to bend the insertion unit 21a, thereby changing the imaging range in accordance with the movement of the object pattern X.

[0058] 6 shows that the imaging range has changed in the direction of the arrow to the imaging range IRa indicated by the thick solid line. If there is an image change accompanying such a change in imaging range IRa (YES in S14), the object determination unit 15 determines the object pattern X as the main subject (the area to be observed) based on the determination result of the operation content determination unit 14 and the output of the image determination unit 11, and the focus area determination unit 13 sets the area within the imaging range where the object pattern X exists as the focus area. The focus control unit 30 controls the optical system 24 of the imaging device 22 so that the image is focused on the set focus area (S15). If there is no image change as described above (NO in S14), the operation determination unit 12 proceeds to S20.

[0059] That is, in this embodiment, by determining the operation of the operator OP following the object pattern X, focusing is performed by using the region of the imaging range corresponding to the object pattern X as the focus area. In other words, if there is an image feature determination unit that determines the characteristics of an image obtained by the imaging unit (imaging device 22) at the tip of the endoscope, it is possible to detect the positional change over time (upward, sideward, etc.) of the characteristic part of the image (object pattern X) detected by the image feature determination unit in the captured image. When the operator performs an operation to follow the object pattern X, it is considered that the operator is trying to somehow view that characteristic part. Therefore, if the operation performed by the operator to change the imaging range of the imaging device 22 with respect to the endoscope at that time is known, the operator's intention can be understood. In other words, the observation target area that the operator intends to observe can be determined from the image based on the information on the positional change over time in the image of the object pattern X and the operation content of the operator's operation to change the imaging range while observing and following this positional change (object movement and operator operation). This target observation area is the area that includes the image feature within the imaging range of the imaging device and is the target of focusing, so a focus area is determined so that this area can be focused on. Whether the operator's operation is a movement that tracks the feature of the object in the image can be detected by changes in the image, or the operation status of the operation unit can be detected using sensor information or the like. It can also be expressed as follows: the target observation area that the operator intends to observe is determined from the image based on the consistency between the temporal change in the image feature obtained by the imaging unit and the operation content of the operator using the endoscope to change the imaging range of the imaging device 22.

[0060] Therefore, even if the distal end 21 does not move to the desired position and direction in response to the operation of the operator OP, it is possible to determine the object pattern X as the observation target area and to focus on it. For example, as shown in Figure 6, even if the object pattern X is not present in the center of the imaging range IRa, it is possible to capture an image with the object pattern X in focus.

[0061] Next, in S16, the observation target portion determination unit 10 determines whether or not the object pattern X has been captured. If the object pattern X has been captured (YES in S16), the focus area determination unit 13 continues to set the area including the object pattern X as the focus area and focuses on the object pattern X (S17). If the object pattern X has not been captured (NO in S16), the operation determination unit 12 determines whether or not the tip end 21 has moved (S18) when the operation determination unit 12 determines that the tip end 21 has moved (YES in S18), and determines that observation of the object pattern X has ended, and proceeds to S20. On the other hand, if the operation determination unit 12 determines that the tip end 21 has not moved (NO in S18), it determines that the object pattern X has been lost (disappeared), and waits with the focus fixed, for example, by lying in wait in the area where the object pattern X has been lost (S19).

[0062] The lost area can be rewritten as a position or direction in the image where pattern X disappears from the image captured by the endoscope's imaging unit, with the center of the image serving as the origin. The location or direction in the image where pattern X disappears is determined, with the expectation that it will reappear from that direction. The internal structure of the human body viewed during an endoscopic examination may undergo regular, repetitive movements due to pulsation or peristalsis. Furthermore, parts of the human body supporting the endoscopic tube (the lumen wall or insertion tube) may also move in a similar manner, repeating the same movement and appearing again. This waiting technique is also effective. In this case, if the target area the operator is attempting to observe is not detected within the image, it can be expressed as setting a waiting area within the image for focusing, following the process by which the target area disappears from the image. To determine the positional change of pattern X within the image over time, the acquired images can be sequentially recorded in a memory or other device. By comparing these images and determining the positional movement of the recorded characteristic image parts, the positional change of pattern X within the image over time can be determined. To determine the positional change over time of the observation target region that the operator is attempting to observe, an endoscope focus control system is provided that includes a memory capable of recording acquired images in chronological order, and an analysis unit (a function of the image determination unit 11 in FIG. 1 ) that compares each recorded image taking into account the timing of imaging (e.g., by plotting a graph showing time on the horizontal axis and position in the image on the vertical axis, analyzing the pattern of the graph, and determining whether the change is repetitive or a movement in a specific direction, etc.) to analyze the direction and pattern of the positional change of the observation target region within the image. In S13, the operation determination unit 12 determines whether the image change was due to organ movement rather than due to the operator's operation. However, in S13, the direction and pattern of the positional change over time within the image, as described herein, may also be analyzed. If it is possible to determine whether the change is repetitive or a movement in a specific direction, the operator may be advised to wait without performing any operation if the change is repetitive, or if the movement is a movement in a specific direction, the operator may be advised to observe the direction in which the imaging unit observes the change.

[0063] In the next step S20, the operation determination unit 12 determines whether or not the endoscope (insertion section 21a) has been inserted or removed. If the endoscope has not been inserted or removed, the operation determination unit 12 returns the process to S19, and if the endoscope has been inserted or removed, the process returns to S5 in Fig. 2. In this way, when the endoscope is inserted into a lumen to observe the region to be observed, an image is automatically captured with the region to be observed in focus.

[0064] 7 is an explanatory diagram for explaining an example in which a plurality of object patterns exist and an observation target portion does not have a specific pattern. In Fig. 7, the imaging range of the imaging device 22 is indicated by a rectangular frame, as in Fig. 6.

[0065] The upper left column of Fig. 7 shows an example in which multiple object patterns X1, X2, and X3, such as lesions, exist within imaging range IR1. Here, assume that an operator, such as a doctor, attempts to observe object pattern X1 by moving or changing the direction of tip 21 to change the imaging range from imaging range IR1 to imaging range IR2, as indicated by the arrow in the upper right column of Fig. 7. At this stage, lesion X2 is located in the center of imaging range IR2, and it is not possible to focus on object pattern X1.

[0066] However, suppose an operator, such as a doctor, further performs an operation to move or change the orientation of the tip 21. The lower left column of FIG. 7 illustrates this state, where the imaging range IR3 is not fixed in one direction but is readjusted and moves in small increments, as indicated by the arrow in the lower left column of FIG. 7. In this case, the operation content determination unit 14 determines that the tip 21 is not moving to the desired position or direction, even though the operator is attempting to position the tip 21 in a position and direction appropriate for imaging the observation target region. In other words, the operation content determination unit 14 determines that the operator is attempting to further move the imaging range in the direction indicated by the arrow in the upper right column of FIG. 7, determines that the object pattern X1 present in this direction is the observation target region, and sets the area including the object pattern X1 as the focus area. This makes it possible to focus on the object pattern X1 even if the object pattern X1 is not located in the center of the imaging range.

[0067] The lower right column of FIG. 7 shows an example in which no object pattern, such as a lesion, exists. The lower right column of FIG. 7 shows that folds inside the lumen are included in the imaging range, but no object pattern, such as a lesion, exists. Even when no pattern, such as a lesion, exists, an operator, such as a doctor, may wish to observe a specific region as the observation target region. Even in such cases, in this embodiment, the operation content determination unit 14 determines the operation of the operator to track and observe the observation target region based on the image change based on the operator's operation or the operation information based on the operation of the operation unit 25. The object determination unit 15 determines an area of ​​a predetermined size and in a direction based on the operator's operation as the focus area. In this way, even when no object pattern exists, it is possible to focus on the observation target region that the operator wishes to observe.

[0068] In this manner, in this embodiment, the state in which the operator is operating the endoscope is detected, the operator's intention is inferred, the region of the observation target that the operator wishes to observe is identified, and control is performed to bring the focus to that region. This makes it easier to capture images that reflect the operator's intentions, and reduces the time required for observation and examination.

[0069] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0070] Furthermore, among the technologies described herein, many of the controls and functions, primarily those illustrated in flowcharts, can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. The program can be recorded or stored, in whole or in part, as a computer program product on a portable medium such as a flexible disk, CD-ROM, or nonvolatile memory, or on a storage medium such as a hard disk or volatile memory, and can be distributed or provided at the time of product shipment, via a portable medium, or via a communication line. A user can easily realize the endoscope focus control device of this embodiment by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.

Claims

1. An endoscope focus control device, comprising: an image feature determination unit that determines features of an image obtained by an imaging device provided at a distal end of an endoscope; and a determination unit that determines an observation target site that the operator intends to observe from the image based on the temporal positional change of a feature portion of the image detected by the image feature determination unit within the captured image and the consistency between the operation content for changing the imaging range of the imaging device with respect to the endoscope, and determines the observation target site as a focusing target.

2. The endoscope focus control device according to claim 1, wherein when the observation target site that the operator intends to observe is not detected in the image, a waiting area for focusing can be set in the image according to the disappearance process of the target site outside the image.

3. The endoscope focus control device according to claim 1, further comprising: a memory capable of recording images obtained from the imaging device in time series to determine the temporal positional change of the observation target site; and an analysis unit that analyzes the direction and positional change pattern of the positional change of the observation target site in the image by comparing the recorded images.

4. The endoscope focus control device according to claim 3, wherein the analysis unit reads and compares images from the memory to analyze the direction and positional change pattern of the positional change of the observation target site in the image, graphs with time on the horizontal axis and in-image position on the vertical axis to analyze the pattern, and further determines whether the positional change of the observation target site is a repetitive type or a moving type in a specific direction.

5. A focus control method, comprising: determining features of an image obtained by an imaging device provided at a distal end of an endoscope; determining an observation target site that the operator intends to observe from the image based on the temporal positional change of a feature portion of the image detected by the determination within the captured image and the consistency between the operation content for changing the imaging range of the imaging device with respect to the endoscope, and determining the observation target site as a focusing target.

6. A focus control program for causing a computer to execute a procedure of determining features of an image obtained by an imaging device provided at a distal end of an endoscope, determining an observation target site that the operator intends to observe from the image based on a temporal position change of a feature portion of the image detected by the determination in the captured image and consistency between the operation content for changing the imaging range of the imaging device of the endoscope by the operator, and determining the observation target site as a focusing target.

7. An endoscope focus control device comprising: a determination unit that determines operation content for changing the imaging range of the imaging device of the endoscope by the operator based on a temporal change of images sequentially obtained in time series by the imaging device provided at the distal end of the endoscope; and a determination unit that determines an observation target site that the operator intends to observe from the image based on the determination result of the operation content, and determines an area including the observation target site in the imaging range of the imaging device as a focus area to be focused.

8. The endoscope focus control device according to claim 7, wherein the determination unit determines the operation content based on not only the image obtained by the imaging device but also operation information based on an operation on the endoscope.

9. The endoscope focus control device according to claim 7, further comprising a focus control unit that performs focus control on the focus area when a determination result of the operation content indicating that a transition has been made from a screening state by the endoscope to an observation state of the observation target site is obtained.

10. The endoscope focus control device according to claim 7, further comprising a focus control unit that focuses on the focus area when the observation target site is imaged in a state where no operation by the operator is applied to the endoscope.

11. The endoscope focus control device according to claim 7, wherein the determination unit determines the operation content for changing the imaging range of the imaging device of the operator corresponding to the temporal change after determining the temporal change of the image in a state where no operation by the operator is applied to the endoscope, thereby specifying the observation target site and making a determination to start focus control for focusing on the observation target site.

12. The determination unit determines the operation content based on the temporal change of the pattern in the image obtained by the imaging device. The endoscope focus control device according to claim 7 is characterized in that.

13. The determination unit determines, as the operation content, an operation in which the operator moves the imaging range in response to the movement of the observation target site with respect to the imaging range. The endoscope focus control device according to claim 7 is characterized in that.

14. The determination unit determines, as the operation content, a bending operation on the endoscope in which the operator attempts to position the observation target site at the center of the imaging range. The endoscope focus control device according to claim 13 is characterized in that.

15. The observation target site is a lesion site. When there are a plurality of the lesion sites within the imaging range, the determination unit determines, based on the bending operation, one of the plurality of lesion sites as the observation target site. The endoscope focus control device according to claim 14 is characterized in that.

16. Based on the temporal change of the images sequentially obtained in time series by the imaging device provided at the distal end of the endoscope, determine the operation content for changing the imaging range of the imaging device by the operator with respect to the endoscope. Based on the determination result of the operation content, determine the observation target site that the operator attempts to observe from the image, and determine the area including the observation target site in the imaging range of the imaging device as the focus area to be focused. A focus control method is characterized in that.

17. A focus control program for causing a computer to execute a procedure of determining the operation content for changing the imaging range of the imaging device by the operator with respect to the endoscope based on the temporal change of the images sequentially obtained in time series by the imaging device provided at the distal end of the endoscope. Based on the determination result of the operation content, determine the observation target site that the operator attempts to observe from the image, and determine the area including the observation target site in the imaging range of the imaging device as the focus area to be focused.

18. An endoscopic focus control device, comprising: an image feature determination unit that determines features of an image obtained by an imaging device provided at a distal end of an endoscope; a determination unit that determines an operation content for changing an imaging range of the imaging device with respect to the endoscope based on a positional change of a feature portion of the image detected by the image feature determination unit within the captured image; and a determination unit that determines an observation target site that the operator intends to observe from the image based on the determination result of the operation content, and determines a region including the observation target site including the image feature portion within the imaging range of the imaging device as a focus area to be focused on.

19. A focus control method, comprising: determining features of an image obtained by an imaging device provided at a distal end of an endoscope; determining an operation content for changing an imaging range of the imaging device with respect to the endoscope based on a positional change of a feature portion of the image detected by the determination within the captured image; determining an observation target site that the operator intends to observe from the image based on the determination result of the operation content; and determining a region including the observation target site including the image feature portion within the imaging range of the imaging device as a focus area to be focused on.

20. A focus control program for causing a computer to execute a procedure of: determining features of an image obtained by an imaging device provided at a distal end of an endoscope; determining an operation content for changing an imaging range of the imaging device with respect to the endoscope based on a positional change of a feature portion of the image detected by the determination within the captured image; determining an observation target site that the operator intends to observe from the image based on the determination result of the operation content; and determining a region including the observation target site including the image feature portion within the imaging range of the imaging device as a focus area to be focused on.

Images