ROBUST AUTOFOCUS FOR DEEP CHANNEL SCENES

Abstract

The disclosure relates to techniques for controlling a medical visualization system, in particular for supporting autofocus functionality when imaging regions of interest in and around deep canals or at "edges" in the site. The disclosure describes techniques for autofocus assistance functionality based on an automatically determined region of interest. An adaptive, confidence-based decision algorithm is used, which is optionally adapted depending on the device parameters of a microscope of the medical visualization system and the temporal change in the xy position of the region of interest in order to maximize the robustness of the autofocus while keeping the response time to a minimum. The decision algorithm exhibits a dependence on the temporal change in the xy position as well as on the defocus value.

Description

TECHNICAL FIELD

[0001] Various examples of the present invention relate to an autofocus assistance functionality of a medical visualization system with a microscope. In particular, various examples of the invention relate to a trigger criterion for focusing on a region of interest. BACKGROUND

[0002] Medical visualization systems, such as surgical microscopy systems (Surgical Microscopy Systems), are used in a variety of applications. One example is neurosurgery. In neurosurgery, the goal is often to reach a deep-seated tumor or aneurysm using a narrow and deep channel. The tumor or aneurysm is to be surgically removed with as little disruption to the surrounding tissue as possible.

[0003] In order to appropriately display regions in the deep canal during surgical intervention using a medical visualization system, it is necessary to control one or more components of the medical visualization system for focusing.

[0004] For example, DE 10 2015 117 824 A1 discloses techniques for implementing autofocus based on a height map. To do so, the user selects an image point or image area to focus on. Such a region of interest can be, for example, the tip of an instrument. Such a technique allows the user to precisely define the region of interest. However, sometimes scenarios are desirable in which the region of interest is identified automatically—for example, to avoid interrupting the workflow during a surgical procedure.

[0005] One technique for automatically identifying the region of interest is the detection of relevant objects in the microscope images, such as the tip of an instrument. For example, the optical flow between two consecutive images could be considered, and based on this optical flow information, the region of interest can then be determined. However, external navigation systems can also be used to track the position of, for example, a marker. The region of interest can then be defined relative to the marker.

[0006] However, it has been found that such techniques, which automatically identify the region of interest, have limited accuracy, particularly for scenes with deep channels, or can lead to undesirable system behavior. For example, it has been observed that the focus may suddenly jump back and forth. Even for nearly identical scenes, very different focus values ​​are set. Particularly with continuous focusing, this leads to the undesirable behavior that strong z-adjustments are implemented by the autofocus assist functionality. The focus value suddenly "jumps" back and forth. The problem is exacerbated by poor image quality, e.g., limited brightness or occlusion of part of the scene, e.g., by an instrument.

[0007] US 2023 / 0 248 464 A1 discloses a surgical microscope system and a system, method, and computer program for a microscope of a surgical microscope system. The system is configured to determine an anatomical feature of interest within a region of interest, to detect the position of the anatomical feature in question in image sensor data, and to trigger an autofocus function of the microscope to focus on the position of the anatomical feature.

[0008] DE 10 2015 103 426 A1 discloses a microscope system and a method for the automated alignment of a microscope. The microscope system comprises a microscope with an observer beam path including a camera, a detection unit for detecting the focal depth set on the microscope, and an image processing unit that determines an image section representing the bottom of the deep channel in the electronic image based on information about the position of the bottom of a deep channel and the received focal depth in an electronic image.

[0009] DE 10 2017 110 816 A1 discloses an optical observation device and a method for efficiently executing an automatic focusing algorithm. The method comprises detecting a position of an optical observation device at a first time. It further comprises, based on the position of the optical observation device relative to the position of the object, detecting a starting distance between the optical observation device and an observation area on the object and executing an automatic focusing algorithm with the starting distance as the initial focus distance. SUMMARY

[0010] Therefore, there is a need for techniques for controlling a medical visualization system that address or mitigate at least some of the aforementioned drawbacks and limitations. In particular, there is a need for techniques for determining a defocus value for scenes with a deep channel, such as in neurosurgery. There is a need for improved autofocus assistance functionality that can provide focus for automatically determined regions of interest.

[0011] This problem is solved by the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0012] Techniques for providing autofocus assistance functionality for a medical visualization system are disclosed below. In particular, these techniques provide particularly robust focusing on an automatically determined region of interest. These techniques enable particularly robust focusing in scenes with a deep channel, such as in neurosurgery.

[0013] To achieve this, the techniques disclosed herein consider several factors related to an autofocus trigger criterion. The multiple factors describe the confidence with which it can be assumed that the region of interest has been correctly determined (and, for example, corresponds to a user request). Thus, a confidence-based, adaptive decision algorithm is provided for the autofocus functionality. If a corresponding autofocus trigger criterion is met, focusing occurs based on a defocus value determined in relation to the region of interest.

[0014] A controller for a medical visualization system is disclosed. The medical visualization system comprises a microscope. The controller is configured to obtain a sequence of microscope images. The microscope images are acquired by the microscope. The controller is also configured to identify a region of interest in the sequence of microscope images. Furthermore, the controller is configured to determine a change in an xy position of the region of interest. This can, in particular, be a temporal change, i.e., describe the strength of the change in the xy position as a function of time. Furthermore, the controller is configured to determine at least one z position of the region of interest. The controller is also configured to determine a defocus value of the region of interest based on the z position of the region of interest.The controller is further configured to determine whether an autofocus trigger criterion is met. The autofocus trigger criterion depends, for example, on the temporal change in the xy position and the defocus value. The controller is also configured to selectively control a component of the medical visualization system for focusing based on the defocus value, with this selective control taking place at least as a function of the autofocus trigger criterion.

[0015] In addition to the autofocus trigger criterion discussed above, which depends on the temporal change in the xy position and the defocus value, it would be conceivable to consider additional autofocus trigger criteria. For example, it could be considered whether a user has requested the triggering of a focus via a user input.

[0016] For example, it would be conceivable for the region of interest to be determined using object localization based on the microscope images. However, it would also be conceivable for the region of interest to be determined based on instrument tracking by an external navigation system that communicates with the medical visualization system as a separate component; then, the previously determined or specified region of interest can simply be searched for in the microscope images or mapped onto the microscope images.

[0017] The x-direction and y-direction refer to directions perpendicular to the main ray of a beam path of an optical channel of the microscope. The xy-position can also be referred to as the lateral position. The x-direction and y-direction, or the change in the xy-position, can therefore, in particular, define a change in the position of the region of interest in an image plane of the microscope images. The z-direction is oriented parallel to the main ray. The z-position can also be referred to as the depth position.

[0018] The temporal change of the xy-position (also referred to herein as Δp) can therefore describe a rate of change of the xy-position as a function of time. A measure of the temporal change of the xy-position can be determined, which is greater the more lateral movement is observed in the region of interest. In principle, the metric to be used in connection with the determination of the temporal change of the xy-position is variable. For example, extreme positions in the x-direction and / or in the y-direction could be observed during a predefined time interval and then a difference between the extreme positions determined. However, a time derivative could also be determined, for example for a specific point in time. The time derivative could also be determined for a given time interval and then the values ​​of the time derivative can be averaged.

[0019] The defocus value (also referred to as Δf) corresponds to the difference between the current focal plane of the microscope's optical channel and the z-position of the region of interest. The defocus value is therefore a local change in the z-position.

[0020] The use of the autofocus trigger criterion, which depends on both the temporal change in the xy position and the defocus value, is based on the realization that relatively small temporal changes in the xy position of the region of interest and a simultaneous significant change in the z position compared to the actual value, i.e., a large defocus value, can lead to unwanted focusing behavior. In such a case (Δp less than the threshold value and Δf greater than the threshold value), a particularly small or even barely perceptible lateral shift of the region of interest would nevertheless result in a significant change in the focus position of the microscope's optical channel. For an observer, the focus would therefore jump, while the region of interest would not shift or shift only slightly.Such a situation can occur particularly in certain scenes depicting a deep channel, namely when the region of interest shifts along the walls of the deep channel. By using an autofocus trigger criterion that considers both the temporal change in the xy position and the defocus value, such system behavior can be better controlled.

[0021] According to the invention, the triggering criterion depends on a ratio of the temporal change in the xy position to the defocus value. It would therefore be conceivable that, in the case of a larger temporal change in the xy position, a larger defocus value than is acceptable for fulfilling the autofocus triggering criterion would also be assumed. If, in contrast, the temporal change in the xy position is particularly small (relative to a specific defocus value), it can be assumed that the autofocus triggering criterion is not met. In other words, for example, a threshold comparison can be carried out between the ratio of the temporal change in the xy position to the defocus value and a predetermined threshold. Such a threshold can be fixed. However, it would also be conceivable for the threshold to be determined variably, for example depending on one or more imaging parameters of the microscope.

[0022] By taking into account the ratio of the temporal change of the xy position to the defocus value, it is possible to react flexibly to different scenarios related to a time-varying positioned area of ​​interest.

[0023] In particular, it may be unnecessary to use, for example, two fixed thresholds for the temporal change of the xy position on the one hand and the defocus value on the other. Instead, based on the ratio of the temporal change of the xy position to the defocus value, different scenarios related to the time-varying region of interest can be dynamically addressed.

[0024] The ratio of the temporal change in the xy position to the defocus value can be indicative of the confidence in determining the region of interest. This is based on the realization that in typical surgical procedures, for example, in neurosurgery, it is rather unusual for the region of interest relevant to the surgeon to move back and forth between two points, for example, at the floor of the deep canal and another point, for example, on the wall of the deep canal. However, such a scenario corresponds to a relatively small temporal change in the xy position in relation to a relatively large defocus value.

[0025] In other words, it would generally be conceivable that the trigger criterion depends on a confidence value in determining the region of interest.

[0026] The confidence value can, for example, depend on the ratio of the temporal change in the xy position to the defocus value. In other words, this means that a higher (lower) confidence can be assumed when determining the region of interest if the ratio of the temporal change in the xy position to the defocus value is large (small).

[0027] Techniques were described above in which the defocus value is determined and taken into account as part of the autofocus triggering criterion. Alternatively or additionally, a temporal change in the z-position (also referred to as Δz) can also be taken into account as part of the autofocus triggering criterion. In other words, it would be possible for the control to still be set up to determine the temporal change in the z-position. It can therefore be determined how much the z-position of the area of ​​interest varies as a function of time. Here, too, different metrics for determining the temporal change in the z-position are conceivable. For example, it would be conceivable to determine a minimum z-position and a maximum z-position during a predetermined time interval. It would then be possible to determine the difference between the minimum z-position and the maximum z-position as the temporal change in the z-position.The difference between exactly two z-positions could also be determined. The time derivative of the z-position over a time interval could be determined and then averaged.

[0028] For example, if the z-position of the region of interest varies greatly as a function of time (say, during a certain time interval), a lower confidence in determining the region of interest can be assumed and it can be assumed that the autofocus trigger criterion is not met.

[0029] For example, it would be conceivable to perform two threshold comparisons. A first threshold comparison can be performed between the ratio of the temporal change in the xy position to the defocus value using a first predefined threshold; a second threshold comparison can be performed between the temporal change in the z position and a corresponding predefined second threshold. Only if both threshold comparisons produce a positive result can the autofocus trigger criterion be considered fulfilled.

[0030] However, other links between the temporal change of the xy position, the defocus value, and the temporal change of the z position in connection with the autofocus trigger criterion would also be conceivable.

[0031] For example, it would be conceivable for the trigger criterion to depend on a threshold comparison between a predefined threshold value and the change in the z-position during a rolling time interval. This means that it can be repeatedly checked for different consecutive time intervals whether the temporal change in the z-position during the corresponding time interval is smaller or larger than a corresponding threshold value. The length of the time interval can depend on the ratio of the temporal change in the xy-position to the defocus value during the corresponding time interval. For example, the time interval could be longer (shorter), the larger (smaller) the ratio of the temporal change in the xy-position to the defocus value.

[0032] The threshold can be fixed. It would also be possible for the threshold to be set depending on one or more imaging parameter values ​​of the microscope. This is particularly helpful when the region of interest is not only identified in the microscope images, but is specifically defined or determined based on the microscope images. In such a case, certain imaging settings may result in a higher or lower baseline confidence in determining the region of interest. For example, the predefined threshold could depend on the depth of field of an optical channel of the microscope and / or on the magnification of an optical channel of the microscope.

[0033] The threshold value can, for example, be determined depending on the depth of field of the optical channel of the microscope used to capture the microscope images. For example, it could be taken into account whether the temporal change in the z-position is greater or smaller than the depth of field. In particular, if the temporal change in the z-position is greater than the depth of field, it can be assumed that the influence of a corresponding focusing of the microscope is particularly noticeable. Accordingly, in such a context, a greater confidence level (i.e., a longer time interval) can be required to avoid disturbing the observer. The same applies to the magnification or zoom of the optical channel: at a higher magnification, the change in focus is typically more noticeable than at a lower magnification.

[0034] More generally, the depth of field of the optical channel and / or the magnification of the optical channel can be taken into account in connection with the trigger criterion for focusing.

[0035] In conjunction with the trigger criterion, additional criteria can be considered, either alternatively or in addition. For example, it would be conceivable to consider the sign of the defocus value in conjunction with the trigger criterion. This could, for example, make a distinction between focusing toward a position deeper in the deep channel and focusing toward a position less deep in the deep channel.

[0036] For example, it is conceivable that a lower confidence value is sufficient to consider the autofocus trigger criterion fulfilled when focusing on a deeper focal plane (compared to focusing on a focal plane closer to the observer). Such techniques are based on the realization that, as a rule, when focusing on deeper focal planes, the brightness in the microscope images decreases and the probability of obscured areas in the microscope image increases (for example, because a scene with a deep channel is being viewed). All of this leads to a tendency for (at least on average) reduced confidence values ​​when focusing on deeper focal planes, so it may be useful to also reduce the requirements for the confidence value.

[0037] From the above, it is clear that by taking into account various factors related to the autofocus triggering criterion, a particularly robust focusing behavior of the autofocus assistance functionality can be achieved. This robust autofocus assistance functionality is particularly desirable for certain scenes that depict a deep channel. For other scenes that do not depict a deep channel but rather a surgical procedure region without major depth changes, other autofocus triggering criteria than those described above may be preferred. Accordingly, it would be conceivable for the controller to be further configured to determine whether the microscope images of the sequence of microscope images show a scene with a deep channel. One of the autofocus triggering criteria described herein can then be selectively activated when the microscope images of the sequence of microscope images show the scene with the deep channel.In other words, a different autofocus trigger criterion can be used for situations where a different scene, such as a flat scene, is shown than for a scene with a deep channel. By selecting the autofocus trigger criterion based on the scene, the autofocus trigger criterion can be optimally adapted to the different conditions, thereby improving the user experience.

[0038] It would be conceivable that determining the temporal change in the xy position could involve low-pass filtering. This means that particularly rapid changes in the xy position can be ignored. This is based on the recognition that—depending on the technique used to identify the region of interest—artificial artifacts can sometimes occur that cause a sudden jump in the xy position of the region of interest.

[0039] In the context of controlling a component of the medical visualization system for focusing, for example, a focusing module of the microscope, such as a zoom lens, could be controlled. However, it would also be conceivable to use a lens with a fixed focal length. In such a case, it would be conceivable, for example, to control a robotic microscope carrier of the medical visualization system to perform z-positioning.

[0040] A method for controlling a medical visualization system is disclosed. The medical visualization system includes a microscope. The method comprises obtaining a sequence of microscope images acquired by the microscope. The microscope images depict a region of interest. The method further comprises determining a temporal change in an xy position of the region of interest. Furthermore, the method comprises determining at least one z position of the region of interest. The method also comprises determining a defocus value of the region of interest based on the z position. The method also comprises determining whether an autofocus trigger criterion is met. The autofocus trigger criterion depends on the temporal change in the xy position and the defocus value. The method further comprises controlling a component of the medical visualization system to focus.The component is controlled based on the defocus value depending at least on the autofocus trigger criterion.

[0041] The features set forth above and features described below may be used not only in the corresponding explicitly set forth combinations, but also in further combinations or in isolation, without departing from the scope of the present invention. SHORT DESCRIPTION OF THE CHARACTERS Fig. 1 schematically illustrates a medical visualization system according to various examples. Fig. Figure 2 schematically illustrates a surgical intervention region with a deep canal, depicted as scenes of a microscope image. Fig. 3 schematically illustrates an electronic data processing device according to various examples. Fig. 4 is a flowchart of an exemplary method. Fig. 5 and Fig. 6 schematically illustrate a time dependence of the change in a z-position of a region of interest and a comparison of this change in the z-position with a threshold value. Fig. Figure 7 illustrates the relationship between the temporal change of the xy position of a region of interest and a defocus value associated with the region of interest. Fig. Figure 8 illustrates a data processing pipeline for the autofocus assistance functionality according to various examples. DETAILED DESCRIPTION

[0042] The above-described properties, features and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more clearly understood in connection with the following description of the embodiments, which are explained in more detail in connection with the drawings.

[0043] The present invention is explained in more detail below using preferred embodiments with reference to the drawings. In the figures, identical reference numerals designate identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements shown in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are depicted in such a way that their function and general purpose will be understood by those skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as an indirect connection or coupling. A connection or coupling can be implemented wired or wirelessly. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0044] Various examples of the invention relate to techniques for selectively triggering an assistance functionality of a medical visualization system during the performance of a surgical procedure. Triggering criteria for an assistance functionality are described. Criteria that can be considered in connection with a triggering criterion are described.

[0045] Various types of surgical procedures may benefit from the techniques disclosed herein. Examples include neurosurgical procedures, such as those involving the head or spine. However, other types of surgical procedures are also conceivable, such as those involving the teeth, ears, and neck, to name just a few.

[0046] By executing an assistance function depending on whether one or more trigger criteria are met, unexpected or surprising behavior of the medical visualization system can be avoided. In particular, discrepancies between user expectations and system behavior can be avoided.

[0047] In the following, particular example triggering criteria for an assistance functionality that is controlled depending on a region of interest are described. For example, the assistance functionality could comprise lateral positioning of a microscope of the medical visualization system so that the field of view of an optical channel of the microscope is centered on the region of interest. Another example assistance functionality that is controlled depending on the region of interest relates, for example, to auto-zoom. In this case, a different zoom factor can be selected depending on the position of the region of interest in a field of view of the optical channel of the microscope. Yet another example assistance functionality relates to autofocusing. In this case, the focal plane of the optical channel of the microscope is set to a specific z-position.This can be achieved, for example, by controlling a zoom lens of the optical channel. Alternatively or additionally, it would also be conceivable for a robotic microscope carrier of the medical visualization system to move the microscope in the z-direction until the focal plane reaches the specified z-position.

[0048] In the following, various examples are described, in particular in connection with such an autofocus assistance functionality, whereby the triggering criteria disclosed herein can also be used for other assistance functionalities that are controlled depending on a dynamically determined area of ​​interest.

[0049] An example trigger criterion considers the extent to which the lateral position (xy position) of the region of interest changes as a function of time. For particularly large temporal changes in the xy position, the trigger criterion may not be met; whereas, for relatively small temporal changes in the xy position, the trigger criterion is met.

[0050] Another example trigger criterion considers—for example, alternatively or in addition to the above-mentioned temporal change in the xy position of the region of interest—a defocus value associated with the region of interest. The defocus value describes a difference between the z position of the region of interest and the current focal plane.

[0051] Fig. Figure 1 schematically shows a surgical microscopy system 801 for surgery. The surgical microscopy system 801 implements a medical visualization system. The surgical microscopy system 801 includes a microscope 802, which in the example shown has an eyepiece 803. However, the eyepiece 803 is optional; it could also be a purely digital microscope 802.

[0052] If the eyepiece 803 is present, the surgeon can view magnified images of an object (here the patient 805) located in a field of view 804 of the surgical microscopy system 801 through the eyepiece 803.

[0053] Microscope 802 generally has one or more optical channels. For example, microscope 802 could have two stereoscopic optical channels. For example, two stereoscopic optical channels can be routed to eyepiece 803. Alternatively or additionally, it would be conceivable for a corresponding camera 809 to be provided for each optical channel. An optical channel can be split using a beam splitter to serve both eyepiece 803 and camera 809.

[0054] An operating device 808 is also provided as a human-machine interface, which can be designed, for example, as a handle or foot switch. In the illustrated embodiment of the Fig. 1, it is a handle. The microscope 802, which is attached to crossheads 850, can be moved by the operating device 808. Motors can be provided to automatically perform the movement based on control data, according to a corresponding setting of the surgical microscopy system. The motors could also support the movement initiated by the operating device 808. Thus, a robotic microscope carrier can be provided.

[0055] Furthermore, at least one controller or control device 899 is provided for the surgical microscopy system 801, which controls the operation of the surgical microscopy system 801 in whole or in part.

[0056] The surgical microscopy system 801 can also have one or more additional sensors 860, e.g., lasers for distance measurement. Using such one or more sensors 860, for example, the z-position of a region of interest can be determined.

[0057] Surgical microscopy systems such as the surgical microscopy system 801 described above are used in neurosurgery to visualize a surgical region. Such a surgical region is often characterized by deep structures in narrow cavities or channels. During a surgical procedure in a narrow channel, a region of interest can be determined (e.g., by the control unit 899) that is of particular relevance to the surgeon in connection with the surgical procedure. Such a region of interest is often located at the tip of a surgical instrument; however, other definitions of a region of interest are also possible.

[0058] In Fig. 2 shows an exemplary positioning of the microscope 802 with respect to a situs 53 within the skull 54 of the patient. All this defines a scene 50 of an intervention region (see also Fig. 1) with a deep channel 59. Also shown are surgical instruments 51, 52, which are arranged in the intervention region 50.

[0059] In Fig. 2 also shows two positions 41, 42 where a region of interest can potentially be located. Position 41 corresponds to the tip of the surgical instrument 52, while position 42 corresponds to the upper edge of the deep channel 59. Fig. Figure 2 shows that the x-distance 31 between the two positions 41, 42 is relatively small compared to the z-distance 32 between the two positions 41, 42. This means that—assuming, for example, a situation in which the focal plane of the microscope 802 includes position 41—a slight change in the x-position of the area of ​​interest results in a large defocus value. This behavior is taken into account in the appropriate definition of an autofocus trigger criterion, according to various examples.

[0060] Fig. Figure 3 schematically illustrates a device 90 that can be used for data processing in the various examples described herein. Device 90 could be, for example, a PC or a cloud server. Device 90 could implement a control device for controlling a medical visualization system such as surgical microscope 801. Device 90 could be part of control device 899 of surgical microscope 801.

[0061] The device 90 comprises a processor unit 91 and a non-volatile memory 92. The processor unit 91 can communicate with one or more other elements or communication nodes via a communication interface 93. For example, it would be conceivable for the processor unit 91 to receive a sequence of microscope images from a camera of a surgical microscope (for example, from the camera 809) via the communication interface 93. Fig. 1). Alternatively or additionally, it would also be conceivable for the processor unit 91 to transmit control data to one or more components of a surgical microscope, for example the surgical microscope 801 from Fig. 1, sends. For example, a focusing module could be controlled or a robotic microscope carrier to move a focal plane relative to an object. The processor unit 91 can load and execute program code from the non-volatile memory 92. This causes the processor unit 91 to perform techniques according to the examples described herein. For example, the processor unit 91 could perform techniques as described below in connection with Fig. 4 are explained.

[0062] Fig. 4 is a flowchart of an exemplary method. The method of Fig. 4 is used to control a medical visualization system. The medical visualization system has a microscope with at least one digital optical channel. The method of Fig. 4 can be used, for example, to control the surgical microscope 801. The method from Fig. 4 can be executed, for example, by the device 90 or in particular by the control unit 899. For example, the method from Fig. 4 executed by a processor unit based on program code loaded from a memory.

[0063] The procedure from Fig. 4 relates to an autofocus assistance functionality. In particular, the method relates to triggering the autofocus assistance functionality. The autofocus assistance functionality enables focusing on a region of interest. The autofocus assistance functionality is triggered when it can be ensured with a certain degree of confidence that the region of interest has been correctly determined. To this end, the region of interest is identified in a sequence of microscope images (e.g., based on a specification), and then one or more autofocus trigger criteria are determined that regulate whether or not focusing takes place. In particular, an autofocus trigger criterion is explained below, which depends on the temporal change in the xy position of the region of interest (Δp) and the defocus value (Δf).Alternatively or additionally, other autofocus trigger criteria could also be taken into account, for example whether the user has triggered the shutter by means of a user input (cf. Fig. 1: Control device 808) requests autofocusing or not. Thus, for example, it can be checked whether a number of autofocus trigger criteria are cumulatively met.

[0064] In box 3005, a sequence of images acquired by the microscope is obtained. For example, the sequence of images could cover a predetermined time interval. For example, it would be conceivable that a predetermined number of images be obtained. It would also be conceivable that images from the microscope be obtained at a specific sampling rate, for example, 25 Hz or 50 Hz.

[0065] The images could be two-dimensional. It's also conceivable that depth information is present.

[0066] By obtaining a sequence of microscope images in box 3005, a temporal change in the scene is visualized. This temporal change in the scene serves as the basis for the dynamic determination of a region of interest.

[0067] It is then possible, but not necessary, to execute box 3010. In box 3010, it can optionally be checked whether the microscope images obtained in box 3005 show a scene with a deep channel. A corresponding scene 50 with a deep channel 59 was described above in connection with Fig. 2. Only if a deep channel is shown in the scene can the procedure then continue in box 3015.

[0068] The check in Box 3010 ensures that the autofocus trigger criterion, discussed in detail below, is applied specifically to certain observed and imaged scenes. This can be desirable because the autofocus trigger criterion, discussed in detail below, is optimized for particularly high robustness and confidence in the context of identifying a region of interest. On the other hand, this high degree of robustness and confidence can result in focusing on a dynamically changing region of interest occurring with a comparatively high latency. For other scenes that are not particularly susceptible to an incorrectly determined region of interest, such robustness may be unnecessary; in this case, the use of the special autofocus trigger criterion can be dispensed with to reduce latency.

[0069] Furthermore, the autofocus trigger criterion discussed in detail below is optimized for the geometry of a deep channel, so that scenes with different geometries of the corresponding intervention region benefit less from the techniques described below. In such cases, it may be preferable to consider a different autofocus trigger criterion in conjunction with the autofocus assistance functionality. This is ensured by the use of box 3010. However, it should still be understood that box 3010 is optional. In some examples, for example, it may be desirable for the autofocus trigger criterion discussed in detail below to be permanently activated. In such a case, box 3010 is optional.

[0070] In box 3015, an area of ​​interest is optionally identified in the sequence of microscope images. Box 3015 can be implemented in various ways. For example, it would be conceivable for the area of ​​interest to be specified or determined based on the sequence of microscope images. For this purpose, an image analysis of the microscope images can be carried out. For example, object recognition of certain objects in the area of ​​interest can take place in the microscope images. For example, markers can also be used. Machine-learned algorithms can be used to localize the area of ​​interest. For example, surgical tools or their tips could be localized. For example, it would be conceivable for the temporal change in the xy position of the area of ​​interest to be determined based on tracking the area of ​​interest in the sequence of microscope images.

[0071] However, it would also be conceivable for the region of interest (or its position as a function of time) to be predetermined based on external control data and then simply searched for in the microscope images or mapped onto the microscope images. For this purpose, for example, a camera model can be used which maps 3D coordinates in the object space to 2D coordinates in the image space of the microscope images. For example, control data can be obtained from an external navigation system. For example, the camera model can be obtained through calibration. During calibration, one or more parameter values ​​of the camera model, for example a focal length parameter value, can be adjusted. An intrinsic and / or extrinsic calibration can be carried out. The camera model can therefore be calibrated specifically for the respective microscope. For example, a corresponding camera model can be calibrated for each optical channel.For calibration, for example, a calibration object (2-D or 3-D target) can be used.

[0072] The region of interest can generally refer to a specific point, such as the tip of a surgical tool or the midpoint between two surgical tool tips. However, it is also conceivable that the region of interest refers to an extended region.

[0073] In box 3015, the region of interest can be identified as a function of time in a time interval covered by the sequence of microscope images.

[0074] Accordingly, in box 3020, a temporal change in the xy position in the region of interest is determined during the time interval covered by the microscope images. In other words, this means that a quantity is determined that is larger the more the xy position of the region of interest changes during the time interval. For example, the time derivative of the x-coordinates and the y-coordinates could be determined, and the corresponding magnitudes of the respective maximum values ​​could be averaged. However, other metrics for determining the temporal change in the xy position of the region of interest in box 3020 would also be conceivable.For example, a mean of the time derivative of the x-coordinates of the region of interest could be determined, as well as another mean of the time derivative of the y-coordinates of the region of interest during the time interval, and then the magnitudes of these means could be averaged again. A distribution of the xy-coordinates could also be determined, and a width of the distribution could be considered. For example, a difference between extreme values ​​of the xy-coordinates during the time interval could be determined.

[0075] In box 3025, a z-position of the region of interest during the time interval is determined. For example, a corresponding z-position of the region of interest could be determined for each microscope image. Then, the mean value could be determined.

[0076] The z-position of the region of interest can be determined in different ways. For example, if the microscope has two stereoscopic optical channels, a disparity determination could be performed and then triangulation could be performed to determine the z-position. Other examples include structured illumination or time-of-flight measurements, for example, with pulsed light (see Fig. 1, Sensor 860), laser light, or ultrasonic waves. LIDAR measurements can be performed.

[0077] In box 3030, it is optionally possible to determine a temporal change in the z-position. The temporal change in the z-position is again quantified over the time interval for which microscope images are obtained in box 3005. For example, metrics can be used to quantify the change in the z-position, as already described above in connection with box 3020 for the temporal change in the xy-position.

[0078] In box 3035, it is then possible to determine the defocus value. The defocus value is the deviation between the current focal plane and the z-position of the region of interest according to box 3025. The defocus can take positive or negative values, depending on whether the region of interest is positioned in front of or behind the current focal plane (as seen from the microscope).

[0079] In box 3040, it is then determined whether the autofocus trigger criterion and, if applicable, one or more additional autofocus trigger criteria are met. In particular, the autofocus trigger criterion is taken into account, which depends on the temporal change in the xy position according to box 3020 and the defocus value according to box 3035. For example, the temporal relationship between the change in the xy position and the defocus value could be determined. Based on this relationship, the trigger criterion could be assumed to be met or not met. Some implementations for a specific trigger criterion are described below.

[0080] A first example is in connection with Fig. 5 and Fig. 6 shown. In Fig. 5 and Fig. 6, the calculated z-position 401 is plotted as a function of time. The current focal plane 486 is also shown with a dotted line in Fig. 5 and Fig. 6. The z-position 401 deviates from the current focal plane 486, so focusing might be helpful. This is checked as part of the autofocus trigger criterion.

[0081] In connection with the autofocus trigger criterion, the example shows the Fig. 5 and Fig. 6 takes into account the temporal fluctuation of the z-position 401: The autofocus trigger criterion is met precisely when, for a time interval 412 (e.g., a rolling time interval) with a variable length and which is at least covered by the microscope images in box 3005, the z-position 401 of the region of interest does not vary or fluctuate excessively. This means that the temporal change 402 (referred to as Δz) of the z-position 401 must be smaller than a threshold value 411 during the entire time interval 412.

[0082] In such a scenario, the strength of the change in the focal plane 486, i.e., the defocus value 702 (Δf), is not limited; rather, the temporal fluctuation 402 of the z-position 401 or the defocus value 702 is taken into account (the temporal fluctuation 402 of the z-position 401 is equal to the temporal fluctuation of the defocus value 702). Fig. 5 and Fig. 6 shows the defocus value 702 Δf, which is calculated, for example, from the mean value of the z-position 401 during the eighth time interval 412 (this mean value is indicated by the dotted-dashed line in Fig. 5 and Fig. 6) and the current focal plane 486.

[0083] The threshold value 411 for the fluctuation 402 of the z-position 401 results, for example, from imaging parameter values ​​of the microscope, such as the parameters set on the microscope for gamma (i.e., magnification of the afocal zoom system) and focus. From these values, for example, the value for the depth of field (e.g., calculated using the Berek depth of field formula, mentioned in WO 2010 017 944 A1; or using a suitable camera model) can be determined. In a specific case, the autofocus triggering criterion can be that the difference between the maximum and minimum z-position (Δz) during the time interval must be below the depth of field value.

[0084] An optional variable length t wFor the time interval, for example, Δf is determined from the defocus value 702 and the change in the xy position Δp (e.g., quantified in image pixels or mm). This means that the ratio of the change in the xy position to the defocus value is taken into account. A possible implementation of this relationship would be: tw=0.5+ΔfΔp

[0085] This dependency enables robust focusing. At the same time, the surgeon is not restricted in his or her actions. He or she can still focus into or out of deep channels. To do so, either the instrument must be moved slightly away from the edge (thus increasing Δp) or the instrument or the area of ​​interest must be briefly paused (waiting for t).

[0086] Variations are conceivable. For example, a scenario was described above in which the defocus value 702 is determined relatively indirectly by the length of the tw of the time interval 412 is taken into account. On the other hand, in the example the Fig. 5 and Fig. 6, the interval band defined by the threshold value 411 is not specified with respect to an absolute z-position, in particular not with respect to the current focal plane 486. This means that the temporal change of the z-position 401 around a certain mean value can occur at any position in the entire observed z-range. In other variants, however, it would be conceivable to specify an interval band that is centered, for example, at the current focal plane 486. This limits the maximum defocus value 702 more directly than via the length of the time interval 412. Asymmetric upper and lower threshold values ​​with respect to the current focal plane 486 would also be conceivable.

[0087] Fig. 5 shows a scenario in which the temporal change 402 of the z-position 401 in the time interval 412 is greater than the threshold value 411 determined based on the depth of field. Fig. 6 shows a scenario in which the temporal change 402 of the z-position 401 in the time interval 412 is smaller than the threshold value 411 determined based on the depth of field. In the scenario of Fig. 5, the autofocus triggering criterion is not met; while in the scenario of Fig. 6 the autofocus trigger criterion is met.

[0088] Another example of a possible implementation of the autofocus trigger criterion is in connection with Fig. 7. Depending on the temporal change 701 of the xy position Δp of the area of ​​interest, a maximum permissible change in focus Δf is defined, i.e., a maximum defocus value 702. In area 711, the autofocus trigger criterion is met; in area 712, the autofocus trigger criterion is not met. This in turn means that the relationship between the temporal change 701 of the xy position and the defocus value 702 is determined. Here, the current confidence in determining the area of ​​interest results from the ratio of Δp and Δf. The confidence limit or a corresponding threshold value 703 can be fixed. The threshold value could be defined depending on the depth of field.

[0089] This example from Fig. 7 is based on the realization that for small lateral movements of the region of interest, no strong changes in focus are allowed. Focusing is deactivated in the edge regions. This implementation enables robust focusing behavior without introducing any latency (unlike in the example of Fig. 5 and Fig. 6). However, focusing on the edge is no longer possible by holding the instrument still. Focusing out of or into the deep channel is only possible by moving the instrument away from the edge region.

[0090] Techniques were described above in which the autofocus trigger criterion depends on the temporal change of the xy position of the area of ​​interest, the defocus value and optionally the temporal change of the z position of the area of ​​interest ( Fig. 5 and Fig. 6). However, this is only an example, and variations for the autofocus trigger criterion are conceivable.

[0091] For example, due to poor lighting conditions in deep channels and the increased occurrence of occlusions, it is often easier to determine both the depth and the position of the region of interest outside a deep channel than within a channel (e.g., when object detection is performed based on microscope images). One way to counteract this disadvantage of deep channels when determining the region of interest is to take the sign of the defocus value into account in the decision algorithm. If, for example, the calculated depth value (z-position of the region of interest) lies above the set focus (z-position of the focal plane), an attempt is made to focus into the depth. In this case, a corresponding threshold value can be increased, for example.

[0092] Fig. Figure 8 illustrates a data processing pipeline for the autofocus assistance functionality according to various examples. The autofocus assistance functionality is provided for a medical visualization system with a microscope. For example, a controller of the medical visualization system could perform data processing according to the pipeline from Fig. 8 execute.

[0093] The autofocus assistance functionality is selectively triggered (module 322 and path 332) depending on an autofocus trigger criterion checked in module 300. This is explained in detail below.

[0094] In module 321, an autofocus request is first received. This can occur, for example, based on a specific operating mode. A user input could request autofocus. However, a continuous autofocus mode could also be activated, in which corresponding autofocus triggers are received periodically at a specific refresh rate.

[0095] Such an autofocus requirement could be referred to as an additional, upstream autofocus trigger criterion, beyond the autofocus trigger criterion described below in module 300.

[0096] In module 300, it is checked whether the autofocus trigger criterion is met. If it is determined in module 300 that the autofocus trigger criterion is met, one or more components of the medical visualization system are controlled in module 322 to set the focus. Module 322 therefore corresponds to box 3045. Module 300 corresponds to boxes 3005 to 3040 of the method from Fig. 4.

[0097] In module 300, a threshold comparison takes place in submodule 317. A threshold is determined in submodule 316, which depends on one or more device parameters, for example, in particular, on imaging parameters of the microscope of the medical visualization system. For example, the device parameters 313 can be indicative of the depth of field with which the microscope images the scene. For example, the threshold 411 could be determined according to the example of Fig. 5 or the example of Fig. 6 can be determined.

[0098] In submodule 315, the current confidence in connection with the determination of the region of interest is determined. This is done based on a ratio of the defocus value (data input 311) and the temporal change of the xy position (data input 312). For example, as previously described in connection with Fig. 5 and Fig. 6, the length of a time interval is determined based on the temporal change in the xy position and the defocus value, and then it is checked whether the temporal change in the z position within the time interval is greater or smaller than the threshold value or the confidence limit from submodule 316. Depending on the result of the threshold comparison in submodule 316, path 332 is then tracked (the autofocus is set) or path 331 is tracked (no focusing).

[0099] In summary, the above techniques for autofocus assistance functionality based on an automatically determined region of interest in and around deep channels or at "edges" in the situs were disclosed. In comparison to previously known techniques, an adaptive, confidence-based decision algorithm is described, which is adapted - optionally depending on the device parameters of the microscope such as gamma (magnification of the afocal zoom system) and focus - and the temporal change of the xy position of the region of interest in order to maximize the robustness of the autofocus while keeping the response time to a minimum. The decision algorithm (i.e., the trigger criterion) depends on the temporal change of the xy position as well as the defocus value. The decision algorithm can optionally consider a rolling time interval (sometimes referred to as a "sliding window").The decision algorithm can also take into account the depth of field of the microscope.

[0100] Of course, the features of the previously described embodiments and aspects of the invention can be combined with one another. In particular, the features can be used not only in the described combinations, but also in other combinations or on their own, without departing from the scope of the invention.

[0101] For example, various examples were disclosed above in connection with a surgical microscopy system implementing a medical visualization system. However, other implementations for the medical visualization system would also be conceivable, for example, an endoscopy system.

Claims

[1] Controller (90, 899) for a medical visualization system (801) with a microscope (802), wherein the controller (90, 899) is configured to perform the following steps: - Obtain (3005) a sequence of microscope images captured by the microscope depicting an area of ​​interest, - Determining (3020) a temporal change (701) of an xy position of the area of ​​interest, - Determine (3025) at least one z-position (401) of the area of ​​interest, - based on the z-position (401) of the area of ​​interest, determining (3035) a defocus value (702) of the area of ​​interest, - Determine (3040) whether an autofocus trigger criterion is met, which depends on the temporal change (701) of the xy position and the defocus value (702), wherein the autofocus trigger criterion depends on a ratio of the temporal change (701) of the xy position to the defocus value (702), and - optionally controlling a component (802, 805) of the medical visualization system (801) for focusing based on the defocus value (702) depending at least on the autofocus trigger criterion. [2] Control according to claim 1, where the autofocus trigger criterion depends on a confidence value in determining the area of ​​interest, where the confidence value depends on the ratio of the temporal change (701) of the xy position to the defocus value (702). [3] Control according to claim 1 or 2, where the control (90, 899) is still set up to perform the following step: - Determining (3030) a temporal change (402) of the z-position of the area of ​​interest, where the trigger criterion still depends on the temporal change in the z-position of the area of ​​interest. [4] Control according to claim 3, wherein the trigger criterion depends on a threshold comparison between a given threshold (411) and the temporal change (402) of the z-position of the area of ​​interest during a rolling time interval (412), where the length of the rolling time interval (412) depends on the ratio of the temporal change (701) of the xy position to the defocus value (702). [5] Control according to claim 4, wherein the predetermined threshold (411) depends on at least one depth of field range of an optical channel of the microscope or magnification of an optical channel of the microscope. [6] Control according to one of the preceding claims, wherein the autofocus trigger criterion further depends on at least one depth of field range of an optical channel of the microscope (802) or magnification of an optical channel of the microscope (802). [7] Control according to one of the preceding claims, wherein the autofocus trigger criterion further depends on a sign of the defocus value. [8] Control according to one of the preceding claims, where the control (90, 899) is still set up to perform the following step: - Determine (3010) whether the microscope images of the sequence of microscope images show a scene (50) with a deep channel (59), wherein the autofocus trigger criterion is selectively checked when the microscope images of the sequence of microscope images show the scene (50) with the deep channel (59). [9] Control according to one of the preceding claims, wherein determining the change (701) of the xy position includes low-pass filtering. [10] Control according to one of the preceding claims, wherein the control is further configured to determine the area of ​​interest based on an image analysis of the multiple microscope images. [11] Control according to one of the preceding claims, where the component includes a varifocal lens of the microscope, or the component includes a robotic microscope carrier of the medical visualization system. [12] Method for controlling a medical visualization system with a microscope, the method comprising: - Obtain (3005) a sequence of microscope images captured by the microscope depicting an area of ​​interest, - Determining (3020) a temporal change (701) of an xy position of the area of ​​interest, - Determine (3025) at least one z-position (401) of the area of ​​interest, - based on the z-position (401) of the area of ​​interest, determining (3035) a defocus value (702) of the area of ​​interest, - Determine (3040) whether an autofocus trigger criterion is met, which depends on the temporal change (701) of the xy position and the defocus value (702), wherein the autofocus trigger criterion depends on a ratio of the temporal change (701) of the xy position to the defocus value (702), and - optionally controlling a component (802, 805) of the medical visualization system (801) for focusing based on the defocus value (702) depending at least on the autofocus trigger criterion. [13] Method according to claim 12, wherein the method is carried out by the control according to any one of claims 1 to 11.

Images