Robust autofocus function for scenes containing deep channels

Abstract

The invention relates to techniques for controlling a medical visualization system, in particular for assisting autofocus functionalities during the process of imaging regions of interest in and around deep channels or at "edges" in situ. The invention describes techniques for an autofocus assistance functionality based on an automatically determined region of interest. An adaptive confidence-based decision algorithm is used which is optionally adapted on the basis of the device parameters of a microscope of the medical visualization system and the change in the xy position of the region of interest over time in order to maximize the robustness of the autofocus function and at the same time keep the reaction time to a minimum. The decision algorithm is based on the change in the xy position over time and on the defocus value.

Description

[0001] DESCRIPTION

[0002] ROBUST AUTOFOCUS FOR DEEP CHANNEL SCENES

[0003] TECHNICAL AREA

[0004] Several examples of the present invention relate to an autofocus assistance functionality of a medical visualization system with a microscope. Several examples of the invention relate in particular to a trigger criterion for focusing on an area of ​​interest.

[0005] BACKGROUND

[0006] Medical visualization systems, such as surgical microscopy systems, are used in various fields. One example is neurosurgery. In neurosurgery, the aim is often to access a deep-lying tumor or aneurysm through a narrow and deep channel. The tumor or aneurysm is to be surgically removed while minimizing damage to the surrounding tissue.

[0007] In order to appropriately display surgical intervention regions in the deep canal using a medical visualization system, it is necessary to control one or more components of the medical visualization system for focusing.

[0008] For example, techniques for implementing autofocus based on a height map are known from DE 102015 117 824 A1. The user selects a pixel or image area to focus on. Such an area of ​​interest could be, for example, the tip of an instrument. This technique allows the user to precisely define the area of ​​interest. However, there are sometimes scenarios where the area of ​​interest is automatically identified—for example, to avoid interrupting the workflow during a surgical procedure.

[0009] One technique for automatically identifying the area of ​​interest is the detection of relevant objects in the microscope images, such as the tip of an instrument. For example, the optical flow between two successive images could be considered, and the area of ​​interest could then be determined based on this optical flow information. Alternatively, external navigation systems can be used to track the position of, for example, a marker. The area of ​​interest can then be defined relative to the marker.

[0010] However, it has been found that such techniques, where the area of ​​interest is automatically identified, can have limited accuracy, especially in scenes with deep channels, or can lead to undesirable system behavior. For example, it has been observed that the focus can suddenly jump back and forth. Even with nearly identical scenes, very different focus values ​​are set. Particularly with continuous focusing, this leads to the undesirable behavior of large z-adjustments being implemented by the autofocus assist function. The focus value suddenly "jumps" back and forth. The problem is exacerbated by poor image quality, such as limited brightness or obstruction of part of the scene, for example, by an instrument.

[0011] SUMMARY

[0012] Therefore, there is a need for techniques to control a medical visualization system that address or mitigate at least some of the aforementioned disadvantages and limitations. In particular, there is a need for techniques to determine a defocus value for scenes with a deep channel, such as in neurosurgery. There is also a need for improved autofocus assistance functionality that can provide focus on automatically determined areas of interest.

[0013] This problem is solved by the features of independent patent claims.

[0014] The features of the dependent claims define embodiments. Techniques for providing autofocus assistance functionality for a medical visualization system are disclosed below. These techniques provide, in particular, especially robust focusing on an automatically determined area of ​​interest. Specifically, they enable robust focusing in scenes with a deep canal, such as in neurosurgery.

[0015] To achieve this, the techniques disclosed herein take into account several factors related to an autofocus trigger criterion. These factors describe the confidence with which the area of ​​interest has been correctly determined (and, for example, corresponds to a user request). Thus, a confidence-based, adaptive decision algorithm is provided for the autofocus functionality. When a corresponding autofocus trigger criterion is met, focusing occurs based on a defocus value determined in relation to the area of ​​interest.

[0016] A controller for a medical visualization system is disclosed. The medical visualization system includes a microscope. The controller is configured to acquire a sequence of microscope images. The microscope images are acquired by the microscope. The controller is also configured to identify an area of ​​interest within the sequence of microscope images. Furthermore, the controller is configured to determine a change in the xy-position of the area of ​​interest. This can, in particular, be a change over time, i.e., describe the magnitude of the change in the xy-position as a function of time. The controller is also configured to determine at least one z-position of the area of ​​interest. Finally, the controller is configured to determine a defocus value of the area of ​​interest based on the z-position of the area of ​​interest.The control system is further configured to determine whether an autofocus trigger criterion is met. This autofocus trigger criterion depends, for example, on the temporal change in the xy-position and the defocus value. The control system is also configured to selectively control a component of the medical visualization system for focusing based on the defocus value, with this selective control occurring at least as a function of the autofocus trigger criterion. In addition to the autofocus trigger criterion discussed above, which depends on the temporal change in the xy-position and the defocus value, it would be conceivable to consider further autofocus trigger criteria. For example, it could be taken into account whether a user has requested a focus trigger via user input.

[0017] For example, it would be conceivable to determine the area of ​​interest by means of object localization based on the microscope images. Alternatively, it would be conceivable to determine the area of ​​interest based on instrument tracking by an external navigation system that communicates with the medical visualization system as a separate component; in this case, the previously determined or defined area of ​​interest can simply be located in the microscope images or mapped onto them.

[0018] The x-direction and y-direction denote directions perpendicular to the main ray of a microscope's optical channel. The xy-position can also be referred to as the lateral position. The x-direction and y-direction, or changes in the xy-position, can thus define, in particular, a change in the location of the area of ​​interest within the image plane of the microscope image. The z-direction is oriented parallel to the main ray. The z-position can also be referred to as the depth position.

[0019] The temporal change of the xy-position (also referred to here as Ap) can thus be defined as the rate of change of the xy-position as a function of time. A numerical measure for the temporal change of the xy-position can be determined, which is greater the more lateral movement is observed in the area of ​​interest. In principle, the metric used in determining the temporal change of the xy-position is variable. For example, extreme positions in the x-direction and / or in the y-direction could be considered during a predefined time interval, and then the difference between these extreme positions could be determined. Alternatively, a time derivative could be calculated, for example, for a specific point in time. The time derivative could also be determined for a given time interval, and then the values ​​of the time derivative could be averaged.

[0020] The defocus value (also denoted as Af) corresponds to the difference between the current focal plane of the microscope's optical channel and the z-position of the area of ​​interest. The defocus value is therefore a spatial change in the z-position.

[0021] The use of the autofocus trigger criterion, which depends on both the temporal change in the xy-position and the defocus value, is based on the understanding that relatively small temporal changes in the xy-position of the area of ​​interest, combined with a significant change in the z-position relative to the current value (i.e., a large defocus value), can lead to unwanted focusing behavior. In such a case (Ap less than the threshold and Af greater than the threshold), a particularly small or even barely perceptible lateral shift of the area of ​​interest would nevertheless result in a significant change in the focus position of the microscope's optical channel. Thus, to an observer, the focus appears to jump, while the area of ​​interest itself remains unchanged or shifts only slightly.Such a situation can occur particularly in certain scenes depicting a deep channel: namely, when the area of ​​interest shifts along the walls of the deep channel. Using an autofocus trigger criterion that considers both the temporal change in the xy-position and the defocus value allows for better control of this system behavior.

[0022] For example, the trigger criterion could depend on the ratio of the temporal change in the xy-position to the defocus value. It would therefore be conceivable that a larger temporal change in the xy-position would also result in a larger defocus value than is acceptable for fulfilling the autofocus trigger criterion. Conversely, if the temporal change in the xy-position is particularly small (relative to a specific defocus value), it can be assumed that the autofocus trigger criterion is not met. In other words, a threshold comparison could be performed between the ratio of the temporal change in the xy-position to the defocus value and a predefined threshold. Such a threshold could be fixed. However, it would also be conceivable for the threshold to be determined variably, for example, depending on one or more imaging parameters of the microscope.

[0023] By considering the relationship between the temporal change of the xy-position and the defocus value, it is possible to flexibly respond to different scenarios related to a time-varying area of ​​interest. In particular, this eliminates the need for, for example, two fixed thresholds: one for the temporal change of the xy-position and the other for the defocus value. Instead, the relationship between the temporal change of the xy-position and the defocus value allows for dynamic adjustments to different scenarios related to the time-varying area of ​​interest.

[0024] The relationship between the temporal change of the xy-position and the defocus value can be indicative of confidence in determining the area of ​​interest. This is based on the understanding that in typical surgical procedures, such as in neurosurgery, it is rather unusual for the area of ​​interest relevant to the surgeon to shift back and forth between two points, for example, at the bottom of the deep canal and another point, for example, on the wall of the deep canal. Such a scenario, however, corresponds to a relatively small temporal change in the xy-position with respect to a relatively large defocus value.

[0025] In other words, it would generally be conceivable that the trigger criterion depends on a confidence value when determining the area of ​​interest.

[0026] The confidence value can depend, for example, on the ratio of the change in the xy-position over time to the defocus value. In other words, a higher (lower) confidence in determining the region of interest can be assumed if the ratio of the change in the xy-position over time to the defocus value is large (small).

[0027] The techniques described above involve determining the defocus value and considering it within the autofocus trigger criterion. Alternatively or additionally, a temporal change in the z-position (also referred to as Az) can also be considered within the autofocus trigger criterion. In other words, the camera control could be configured to determine the temporal change in the z-position. This allows determining how much the z-position of the area of ​​interest varies over time. Various metrics are conceivable for determining the temporal change in the z-position. For example, it would be possible to determine a minimum z-position and a maximum z-position during a given time interval. The difference between the minimum and maximum z-positions could then be used to calculate the temporal change in the z-position.It is also possible to determine the difference between exactly two z-positions. The time derivative of the z-position over a time interval can be determined and then averaged.

[0028] For example, if the z-position of the area of ​​interest varies greatly as a function of time (such as during a certain time interval), a lower confidence in determining the area of ​​interest can be assumed, and it can be assumed that the autofocus trigger criterion is not met.

[0029] For example, it would be conceivable to perform two threshold comparisons. A first threshold comparison could be performed between the ratio of the change in the xy-position over time to the defocus value with a first predefined threshold; a second threshold comparison could be performed between the change in the z-position over time and a corresponding second predefined threshold. Only if both threshold comparisons yield a positive result can the autofocus trigger criterion be considered fulfilled.

[0030] However, other combinations of the temporal change of the xy position, the defocus value, and the temporal change of the z position in connection with the autofocus trigger criterion would also be conceivable.

[0031] For example, the trigger criterion could depend on a threshold comparison between a predefined threshold value and the change in the z-position during a rolling time interval. This means that it's possible to repeatedly check, for different successive time intervals, whether the change in the z-position during that interval is less than or greater than a given threshold value. The length of the time interval could depend on the ratio of the change in the xy-position to the defocus value during that interval. For instance, the time interval could be longer (shorter) the greater (smaller) the ratio of the change in the xy-position to the defocus value. The threshold value could be fixed. Alternatively, the threshold value could be set based on one or more imaging parameter values ​​of the microscope.This is particularly helpful when the area of ​​interest is not only identified in the microscope images, but specifically defined or determined based on them. In such a case, certain imaging settings may result in a higher or lower basic confidence level in determining the area of ​​interest. For example, the specified threshold might depend on the depth of field of an optical channel of the microscope and / or on the magnification of an optical channel.

[0032] The threshold can be determined, for example, depending on the depth of field of the microscope's optical channel used to capture the images. For instance, it could be considered whether the change in the z-position over time is greater or less than the depth of field. In particular, if the change in the z-position over time is greater than the depth of field, the effect of a corresponding refocusing of the microscope can be assumed to be particularly noticeable. Accordingly, a higher confidence level (i.e., a longer time interval) can be required in such a context to avoid disturbing the observer. The same applies to the magnification or zoom of the optical channel: at higher magnifications, changes in focus are typically more noticeable than at lower magnifications.

[0033] More generally, the depth of field of the optical channel and / or the magnification of the optical channel can be taken into account in connection with the trigger criterion for focusing.

[0034] In connection with the trigger criterion, alternative or additional criteria can also be considered. For example, it would be conceivable to take the sign of the defocus value into account in connection with the trigger criterion. This would allow a distinction to be made, for instance, between focusing on a position deeper in the deep channel and focusing on a position less deep in the deep channel. For example, it could be stipulated that a lower confidence level is sufficient to consider the autofocus trigger criterion fulfilled when focusing on a deeper focal plane (compared to focusing on a focal plane closer to the viewer).Such techniques are based on the understanding that, as a rule, focusing on deeper focal planes reduces the brightness in microscope images and increases the probability of obscured areas in the image (for example, when viewing a scene with a deep channel). All of this leads to a tendency towards (at least on average) reduced confidence levels when focusing on deeper focal planes, so it can be advantageous to also reduce the confidence level requirements.

[0035] It is evident from the above that by considering various factors related to the autofocus trigger criterion, particularly robust focusing behavior of the autofocus assist functionality can be achieved. This robust autofocus assist functionality is especially desirable for certain scenes depicting a deep canal. For other scenes that do not represent a deep canal, but rather a surgical intervention region without significant depth changes, different autofocus trigger criteria than those described above may be preferred. Accordingly, it would be conceivable for the control to be further configured to determine whether the microscope images in the sequence depict a scene with a deep canal. One of the autofocus trigger criteria described herein can then be selectively activated when the microscope images in the sequence depict the scene with the deep canal.In other words, situations where a different scene is displayed, for example, a flat scene, may require a different autofocus trigger criterion than for a scene with a deep channel. By selecting a scene-adaptive autofocus trigger criterion, the autofocus trigger can be optimally adapted to different conditions, thus improving the user experience.

[0036] It is conceivable that determining the temporal change of the xy-position involves low-pass filtering. This means that particularly rapid changes in the xy-position can be disregarded. This is based on the understanding that—depending on the technique used to identify the region of interest—artificial artifacts can sometimes occur that cause a sudden jump in the xy-position of the region of interest.

[0037] In connection with controlling a component of the medical visualization system for focusing, a focusing module of the microscope could be controlled, for example, a varifocal lens. However, it would also be conceivable to use a fixed focal length lens. In such a case, it would be possible, for example, to control a robotic microscope carrier of the medical visualization system to perform z-positioning.

[0038] A method for controlling a medical visualization system is disclosed. The medical visualization system includes a microscope. The method comprises obtaining a sequence of microscope images captured by the microscope. The microscope images depict an area of ​​interest. The method also includes determining a change in the xy-position of the area of ​​interest over time. Furthermore, the method includes determining at least one z-position of the area of ​​interest. The method also includes determining a defocus value of the area of ​​interest based on the z-position. The method also includes determining whether an autofocus trigger criterion is met. The autofocus trigger criterion depends on the change in the xy-position over time and the defocus value. The method further includes controlling a component of the medical visualization system for focusing.The component is controlled based on the defocus value, depending at least on the autofocus trigger criterion.

[0039] The features set out above and those described below can be used not only in the corresponding explicitly set out combinations, but also in further combinations or in isolation, without leaving the scope of protection of the present invention.

[0040] BRIEF DESCRIPTION OF THE FIGURES FIG. 1 schematically illustrates a medical visualization system according to various examples.

[0041] FIG. 2 schematically illustrates a surgical intervention region with a deep canal, depicted as scenes from a microscope image.

[0042] FIG. 3 schematically illustrates an electronic data processing device according to various examples.

[0043] FIG. 4 is a flowchart of an exemplary process.

[0044] FIG. 5 and FIG. 6 schematically illustrate a time dependency of the change in the z-position of an area of ​​interest and a comparison of this change in the z-position with a threshold value.

[0045] FIG. 7 illustrates the relationship between the temporal change of the xy position of an area of ​​interest and a defocus value associated with the area of ​​interest.

[0046] FIG. 8 illustrates a data processing pipeline for the autofocus assistance functionality according to various examples.

[0047] DETAILED DESCRIPTION

[0048] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings.

[0049] The present invention is explained in more detail below with reference to preferred embodiments and the drawings. In the figures, identical reference numerals denote identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily shown to scale. Rather, the various elements depicted in the figures are represented in such a way that their function and general purpose are understandable to a person skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings. A connection or coupling can be implemented as a wired or wireless connection. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0050] Several examples of the invention relate to techniques for selectively triggering an assistance function of a medical visualization system during a surgical procedure. Trigger criteria for an assistance function are described. Criteria that can be considered in connection with a trigger criterion are also described.

[0051] Various types of surgical procedures can benefit from the techniques revealed herein. Examples include neurosurgical procedures, such as those involving the head or spine. Other types of surgical interventions are also conceivable, for example, in the dental, ear, or neck area, to name just a few.

[0052] By executing an assistance function depending on whether one or more trigger criteria are met, unexpected or surprising behavior of the medical visualization system can be avoided. In particular, discrepancies between user expectations and system behavior can be prevented.

[0053] The following section describes exemplary trigger criteria for an assistance function that is controlled based on an area of ​​interest. For example, the assistance function could involve laterally positioning a microscope within the medical visualization system so that the field of view of one optical channel of the microscope is centered on the area of ​​interest. Another example of an assistance function controlled based on the area of ​​interest is an auto-zoom. Here, a different zoom factor can be selected depending on the position of the area of ​​interest within the field of view of the microscope's optical channel. Yet another example of an assistance function involves autofocus. This sets the focal plane of the microscope's optical channel to a specific z-position.This can be achieved, for example, by controlling a varifocal lens in the optical channel. Alternatively or additionally, it would also be conceivable for a robotic microscope carrier of the medical visualization system to move the microscope in the z-direction until the focal plane reaches the specific z-position.

[0054] The following describes various examples, particularly in connection with such an autofocus assistance functionality, but the trigger criteria disclosed herein can also be used for other assistance functionalities that are controlled depending on a dynamically determined area of ​​interest.

[0055] One example of a trigger criterion considers how the lateral position (xy-position) of the area of ​​interest changes as a function of time. The trigger criterion may not be met if the xy-position changes over time in a particularly large way; whereas it is met if the xy-position changes over time in a relatively small way.

[0056] Another exemplary trigger criterion considers – for example, as an alternative or in addition to the aforementioned temporal change in the xy-position of the area of ​​interest – a defocus value associated with the area of ​​interest. The defocus value describes a difference between the z-position of the area of ​​interest and the current focal plane.

[0057] FIG. 1 schematically shows a surgical microscopy system 801. The surgical microscopy system 801 incorporates a medical visualization system. The surgical microscopy system 801 includes a microscope 802, which in the example shown has an eyepiece 803. However, the eyepiece 803 is optional; it could also be a purely digital microscope 802.

[0058] If the eyepiece 803 is present, the surgeon can view magnified images of an object (here the patient 805) located in a field of view 804 of the surgical microscopy system 801 through the eyepiece 803.

[0059] The microscope 802 generally has one or more optical channels. For example, the microscope 802 could have two stereoscopic optical channels. For example, two stereoscopic optical channels could be routed to the eyepiece 803. Alternatively or additionally, it would be conceivable that a corresponding camera 809 is provided for each optical channel. An optical channel can be divided by means of a beam splitter to operate both the eyepiece 803 and the camera 809.

[0060] An operating device 808 is also provided as a human-machine interface, which can be designed, for example, as a handle or foot switch. In the embodiment shown in FIG. 1, it is a handle. The operating device 808 allows the microscope 802, which is attached to traverses 850, to be moved. Motors can be provided to perform the movement automatically based on control data, according to a corresponding setting of the surgical microscopy system. The motors could also assist the movement initiated by the operating device 808. Thus, a robotic microscope carrier can be provided.

[0061] Furthermore, at least one control unit or control device 899 is provided for the operating room microscopy system 801, which controls the operation of the operating room microscopy system 801 completely or partially.

[0062] The 801 surgical microscopy system can also be equipped with one or more additional 860 sensors, e.g., lasers for distance measurement. Using such one or more 860 sensors, the z-position of an area of ​​interest can be determined, for example.

[0063] Surgical microscopy systems, such as the Surgical Microscopy System 801 described above, are used in neurosurgery to visualize a surgical site. Such a surgical site is often characterized by deep-lying structures in narrow cavities or channels. During a surgical procedure in a narrow channel, an area of ​​interest can be identified (e.g., by the Control Unit 899) that is of particular relevance to the surgeon in the context of the surgical procedure. Often, such an area of ​​interest is located at the tip of a surgical instrument; however, other definitions of an area of ​​interest are also possible.

[0064] Figure 2 shows an exemplary positioning of the microscope 802 in relation to a situs 53 within the patient's skull 54. This defines a scene 50 of a surgical region (see also Figure 1) with a deep canal 59. Surgical instruments 51, 52 are also shown, positioned within the surgical region 50. Figure 2 also shows two positions 41, 42 where a potential area of ​​interest could be located. Position 41 corresponds to the tip of the surgical instrument 52, while position 42 corresponds to the upper edge of the deep canal 59. Figure 2 shows that the x-distance 31 between the two positions 41, 42 is relatively small compared to the z-distance 32 between the two positions 41, 42.This means that – assuming, for example, that the focal plane of microscope 802 includes position 41 – a slight change in the x-position of the area of ​​interest results in a large defocus value. This behavior is taken into account, as illustrated by various examples, when appropriately defining an autofocus trigger criterion.

[0065] FIG. 3 schematically illustrates a device 90 that can be used for data processing in the various examples described herein. The device 90 could, for example, be a PC or a cloud server. The device 90 could implement a control device for controlling a medical visualization system such as the operating microscope 801. The device 90 could be part of the control unit 899 of the operating microscope 801.

[0066] The device 90 comprises a processor unit 91 and non-volatile memory 92. The processor unit 91 can communicate with one or more other elements or communication nodes via a communication interface 93. For example, the processor unit 91 could receive a sequence of microscope images from a camera of a surgical microscope (for example, camera 809 from FIG. 1) via the communication interface 93. Alternatively or additionally, the processor unit 91 could send control data via the communication interface 93 to one or more components of a surgical microscope, for example, the surgical microscope 801 from FIG. 1. For example, a focusing module could be controlled, or a robotic microscope carrier could be moved to shift a focal plane relative to an object. The processor unit 91 can load and execute program code from the non-volatile memory 92.This causes the processor unit 91 to execute techniques according to the examples described herein. For example, the processor unit 91 could execute techniques as explained below in connection with FIG. 4. FIG. 4 is a flowchart of an exemplary procedure. The procedure from FIG. 4 is used to control a medical visualization system. The medical visualization system has a microscope with at least one digital optical channel. The procedure from FIG. 4 can be executed, for example, to control the operating microscope 801. The procedure from FIG. 4 can be executed, for example, by the device 90 or, in particular, by the control unit 899. For example, the procedure from FIG. 4 can be executed by a processor unit based on program code loaded from memory.

[0067] The method shown in FIG. 4 relates to an autofocus assistance function. Specifically, the method relates to triggering the autofocus assistance function. The autofocus assistance function enables focusing on an area of ​​interest. The autofocus assistance function is triggered when it can be ensured with a certain degree of confidence that the area of ​​interest has been correctly identified. For this purpose, the area of ​​interest is identified in a sequence of microscope images (for example, based on a predefined set of parameters), and then one or more autofocus trigger criteria are determined that regulate whether or not focusing takes place. In particular, one autofocus trigger criterion is explained below, which depends on the temporal change in the xy-position of the area of ​​interest (Ap) and on the defocus value (A).Alternatively or additionally, further autofocus trigger criteria could be considered, for example, whether the user requests autofocus via a user input (compare FIG. 1: Control Unit 808). It is therefore possible, for instance, to check whether a large number of autofocus trigger criteria are cumulatively fulfilled.

[0068] Box 3005 contains a sequence of images captured by the microscope. For example, the sequence of images could cover a predetermined time interval. It would also be conceivable to obtain a predetermined number of images. Furthermore, it would be conceivable to obtain images from the microscope at a specific sampling rate, such as 25 Hz or 50 Hz.

[0069] The images can be two-dimensional. Depth information is also conceivable. By obtaining a sequence of microscope images in Box 3005, a temporal change of the scene is visualized. This temporal change of the scene serves as the basis for the dynamic determination of an area of ​​interest.

[0070] It is then possible, but not necessary, to execute Box 3010. In Box 3010, it can optionally be checked whether the microscope images obtained in Box 3005 show a scene with a deep channel. A corresponding scene 50 with a deep channel 59 has been shown above in connection with FIG. 2. Only if a deep channel is shown in the scene can the procedure subsequently be continued in Box 3015.

[0071] The check in Box 3010 ensures that the autofocus trigger criterion discussed in detail below is applied specifically to certain observed and imaged scenes. This can be desirable because the autofocus trigger criterion discussed in detail below is optimized for particularly high robustness and confidence in identifying an area of ​​interest. On the other hand, this high degree of robustness and confidence can result in focusing on a dynamically changing area of ​​interest with a comparatively long latency. For other scenes that are not particularly prone to an incorrectly identified area of ​​interest, such robustness may be unnecessary; in these cases, the use of the specific autofocus trigger criterion can be avoided to reduce latency.

[0072] Furthermore, the autofocus trigger criterion discussed in detail below is optimized for the geometry of a deep channel, meaning that scenes with different geometries of the corresponding intervention region benefit less from the techniques described below. In such cases, it may be preferable to consider a different autofocus trigger criterion in conjunction with the autofocus assist functionality. This is ensured by using Box 3010. However, it should be understood that Box 3010 is optional. In some examples, for instance, it may be desirable to have the autofocus trigger criterion discussed in detail below permanently enabled. In such a case, Box 3010 is particularly optional.

[0073] Box 3015 optionally contains an area of ​​interest in the sequence of

[0074] The area of ​​interest is identified in microscope images. Box 3015 can be implemented in various ways. For example, it would be conceivable to define or determine the area of ​​interest based on the sequence of microscope images. This can be achieved through image analysis of the microscope images. For instance, object recognition of specific objects within the area of ​​interest can take place in the microscope images. Markers can also be used. Machine-learned algorithms can be employed to locate the area of ​​interest. For example, surgical tools or their tips could be located. Alternatively, the temporal change in the xy-position of the area of ​​interest could be determined based on tracking the area of ​​interest within the sequence of microscope images.

[0075] It would also be conceivable that the area of ​​interest (or its position as a function of time) is already predetermined based on external control data and is then simply located in or mapped onto the microscope images. For this purpose, a camera model can be used, for example, which maps 3D coordinates in object space to 2D coordinates in the image space of the microscope images. Control data can be obtained from an external navigation system, for instance. The camera model can be obtained through calibration. During calibration, one or more parameter values ​​of the camera model, such as a focal length parameter, can be set. Intrinsic and / or extrinsic calibration can be performed. The camera model can therefore be calibrated specifically for the respective microscope. For example, a corresponding camera model can be calibrated for each optical channel.For calibration, a calibration object (2-D or 3-D target) can be used, for example.

[0076] The area of ​​interest can generally refer to a specific point, for example, the tip of an operating tool or the midpoint between two operating tool tips. However, it is also conceivable that the area of ​​interest refers to an extended region.

[0077] In Box 3015, the area of ​​interest can be identified as a function of time within a time interval covered by the sequence of microscope images.

[0078] Accordingly, in Box 3020, the temporal change of the xy-position in the region of interest during the time interval covered by the microscope images is determined. In other words, a quantity is determined that is larger the more the xy-position of the region of interest changes during the time interval. For example, the time derivatives of the x-coordinates and the y-coordinates could be determined, and the corresponding magnitudes of the respective maximum values ​​averaged. However, other metrics for determining the temporal change of the xy-position of the region of interest in Box 3020 would also be conceivable. For example, a mean value of the time derivative of the x-coordinates of the region of interest, as well as another mean value of the time derivative of the y-coordinates of the region of interest during the time interval, could be determined, and then the magnitudes of these means averaged again.It would also be possible to determine the distribution of the xy-coordinates and consider the width of the distribution. For example, the difference between extreme values ​​of the xy-coordinates during the time interval could be determined.

[0079] In Box 3025, a z-position of the region of interest is determined during the time interval. For example, a corresponding z-position of the region of interest could be determined for each microscope image. Then the mean value could be calculated.

[0080] The z-position of the area of ​​interest can be determined in various ways. For example, if the microscope has two stereoscopic optical channels, the disparity could be determined, followed by triangulation to determine the z-position. Other examples include structured illumination or time-of-flight measurements, such as with pulsed light (see FIG. 1, Sensor 860), laser light, or ultrasound waves. LiDAR measurements can also be performed.

[0081] In Box 3030, it is optionally possible to determine a change in the z-position over time. This change is then quantified over the time interval for which microscope images are obtained in Box 3005. For example, metrics for quantifying the change in the z-position can be used, as previously described in connection with Box 3020 for the change in the xy-position over time. In Box 3035, it is then possible to determine the defocus value. The defocus value is the deviation between the current focal plane and the z-position of the area of ​​interest according to Box 3025. The defocus can take on positive or negative values, depending on whether the area of ​​interest is positioned in front of or behind the current focal plane (as seen from the microscope).

[0082] Box 3040 then determines whether the autofocus trigger criterion, and if applicable, one or more other autofocus trigger criteria, are met. In particular, it considers the autofocus trigger criterion that depends on the temporal change of the xy-position according to Box 3020 and the defocus value according to Box 3035. For example, the temporal relationship between the change in the xy-position and the defocus value could be determined. Based on this relationship, the trigger criterion could be considered met or not met. Some implementations for a specific trigger criterion are described below.

[0083] A first example is shown in connection with FIGS. 5 and 6. In FIGS. 5 and 6, the calculated z-position 401 is plotted as a function of time. The current focal plane 486 is also shown with a dotted line in FIGS. 5 and 6. The z-position 401 differs from the current focal plane 486, so refocusing might be helpful. This is checked as part of the autofocus trigger criterion.

[0084] In connection with the autofocus trigger criterion, the temporal fluctuation of the z-position 401 is taken into account in the example of FIGS. 5 and FIGS. 6: The autofocus trigger criterion is fulfilled if and only if, for a time interval 412 (e.g., a rolling time interval) of variable length, which is covered by the microscope images in Box 3005, the z-position 401 of the area of ​​interest does not vary, fluctuate, or oscillate too much. This means that the temporal change 402 (denoted as Az) of the z-position 401 during the entire time interval 412 must be less than a threshold value 411.

[0085] In such a scenario, the magnitude of the change in the focal plane 486, that is, the defocus value 702 (A), is not limited; rather, the temporal variation 402 of the z-position 401 or the defocus value 702 is taken into account (the temporal variation 402 of the z-position 401 is equal to the temporal variation of the defocus value 702). In FIG. 5 and FIG. 6, the defocus value 702 is shown in each case.

[0086] A shown, which results, for example, from the mean of the z-position 401 during the cht time interval 412 (this mean is shown by the dotted-dashed line in FIG. 5 and FIG. 6) and the current focal plane 486.

[0087] The threshold value 411 for the fluctuation 402 of the z-position 401, for example, results from imaging parameter values ​​of the microscope, such as the parameters set on the microscope for gamma (i.e., magnification of the afocal zoom system) and focus. From these values, the value for the depth of field can be determined (e.g., calculated using the formula for depth of field according to Berek, mentioned in WO 2010 017 944 A1; or by means of a suitable camera model). In this specific case, the autofocus trigger criterion can be that the difference between the maximum and minimum z-position (Az) during the time interval must be less than the value for the depth of field.

[0088] An optional variable length t wThe time interval is calculated, for example, from the defocus value 702 Af and the change in the xy-position Ap (e.g., quantified in image pixels or mm). This means that the ratio of the change in the xy-position to the defocus value is taken into account. One possible implementation of this relationship would be:

[0089] This dependency enables robust focusing. At the same time, the surgeon's movements are not restricted. They can still focus into and out of deep channels. This requires either moving the instrument slightly away from the edge (thus increasing Ap) or briefly pausing the instrument or the area of ​​interest (waiting t).

[0090] Variations are conceivable. For example, a scenario was described above in which the defocus value 702 is determined relatively indirectly by the length of the t. wthe time interval 412 is taken into account. On the other hand, in the example of FIG. 5 and FIG. 6, the interval band defined by the threshold value 411 is not fixed with respect to an absolute z-position, in particular not with respect to the current focal plane 486. This means that the temporal change of the z-position 401 by a certain mean value can occur at any position within the entire observed z-range. In other variations, however, it would be conceivable to define an interval band that is centered, for example, on the current focal plane 486. This limits the maximum defocus value 702 more directly than by the length of the time interval 412. Asymmetric upper and lower thresholds with respect to the current focal plane 486 would also be conceivable.

[0091] FIG. 5 shows a scenario where the temporal change 402 of the z-position 401 in the time interval 412 is greater than the threshold 411 determined based on the depth of field range. FIG. 6 shows a scenario where the temporal change 402 of the z-position 401 in the time interval 412 is less than the threshold 411 determined based on the depth of field range. In the scenario of FIG. 5, the autofocus trigger criterion is not met; whereas in the scenario of FIG. 6, the autofocus trigger criterion is met.

[0092] Another example of a possible implementation of the autofocus trigger criterion is shown in conjunction with FIG. 7. Depending on the temporal change 701 of the xy-position Ap of the area of ​​interest, a maximum permissible change in focus Af is defined, i.e., a maximum defocus value 702. In the area 711, the autofocus trigger criterion is met; in the area 712, the autofocus trigger criterion is not met. This means, in turn, that the ratio between the temporal change 701 of the xy-position and the defocus value 702 is determined. Here, the current confidence in determining the area of ​​interest is derived from the ratio of Ap and Af. The confidence limit, or a corresponding threshold 703, can be fixed. The threshold could be defined as a function of the depth of field.

[0093] This example from FIG. 7 also relies on the understanding that small lateral movements of the area of ​​interest do not permit significant changes in focus. Focusing is deactivated in the edge regions. This design enables robust focusing behavior completely without introducing latency (unlike the examples in FIG. 5 and FIG. 6). However, focusing on the edge by pausing the instrument is no longer possible. Focusing in and out of the deep canal is only possible by moving the instrument away from the edge region. Techniques have been described above in which the autofocus trigger criterion depends on the temporal change in the xy-position of the area of ​​interest, the defocus value, and optionally, the temporal change in the z-position of the area of ​​interest (FIG. 5 and FIG. 6).However, this is just one example, and variations for the autofocus trigger criterion are conceivable.

[0094] For example, due to poorer lighting conditions in deep channels and the increased occurrence of occlusions, it is often easier to determine both the depth and the position of the area of ​​interest outside a deep channel compared to inside one (for example, when object recognition is performed based on microscope images). One way to counteract this disadvantage of deep channels in determining the area of ​​interest is to consider the sign of the defocus value in the decision algorithm. For example, if the calculated depth value (z-position of the area of ​​interest) is above the set focus (z-position of the focal plane), the algorithm attempts to focus into the depth. In this case, a corresponding threshold value can be increased.

[0095] Figure 8 illustrates a data processing pipeline for autofocus assistance functionality according to various examples. The autofocus assistance functionality is provided for a medical visualization system with a microscope. For example, a controller of the medical visualization system could perform data processing according to the pipeline shown in Figure 8.

[0096] The autofocus assist functionality is selectively triggered depending on an autofocus trigger criterion that is checked in module 300 (module 322 and path 332). This is explained in detail below.

[0097] Module 321 first receives an autofocus request. This can occur, for example, based on a specific operating mode. A user input could request autofocus. Alternatively, a continuous autofocus mode could be activated, in which corresponding autofocus triggers are received periodically at a specific repetition rate. Such an autofocus request could be described as an additional, upstream autofocus trigger criterion, beyond the autofocus trigger criterion described subsequently in Module 300.

[0098] Module 300 checks whether the autofocus trigger criterion is met. If Module 300 determines that the autofocus trigger criterion is met, then one or more components of the medical visualization system are activated in Module 322 to set the focus. Module 322 therefore corresponds to Box 3045. Module 300 corresponds to Boxes 3005 to 3040 of the method shown in FIG. 4.

[0099] In module 300, a threshold comparison takes place in submodule 317. A threshold value is determined in submodule 316, which depends on one or more device parameters, for example, imaging parameters of the microscope in the medical visualization system. For instance, device parameter 313 can be indicative of the depth of field range with which the microscope images the scene.

[0100] For example, the threshold value 411 could be determined according to the example in FIG. 5 or the example in FIG. 6.

[0101] In submodule 315, the current confidence level is determined in connection with the selection of the area of ​​interest. This is based on the ratio of the defocus value (data input 311) to the temporal change of the xy-position (data input 312). For example, as described above in connection with FIG. 5 and FIG. 6, the length of a time interval could be determined based on the temporal change of the xy-position and the defocus value. Then, it could be checked whether the temporal change of the z-position within that time interval is greater or less than the threshold or confidence limit from submodule 316. Depending on the result of the threshold comparison in submodule 316, path 332 is then followed (autofocus is set), or path 331 is followed (no focusing).

[0102] In summary, techniques for autofocus assistance based on an automatically determined area of ​​interest in and around deep channels or at "edges" in the situs have been disclosed above. In contrast to previously known techniques, an adaptive, confidence-based decision algorithm is described that is adjusted—optionally depending on the microscope's instrument parameters, such as gamma (magnification of the afocal zoom system) and focus—and the temporal change in the xy-position of the area of ​​interest, in order to maximize the robustness of the autofocus while minimizing the response time. The decision algorithm (i.e., the trigger criterion) is dependent on the temporal change in the xy-position as well as the defocus value. The decision algorithm can optionally incorporate a rolling time interval (sometimes referred to as a "sliding window").The decision algorithm can also take into account the depth of field of the microscope.

[0103] In summary, the following EXAMPLES were described in particular:

[0104] EXAMPLE 1. Controller (90, 899) for a medical visualization system (801) with a microscope (802), wherein the controller (90, 899) is configured to perform the following steps:

[0105] - Obtain (3005) a sequence of microscope images captured by the microscope depicting an area of ​​interest,

[0106] - Determining (3020) a temporal change (701) of an xy position of the area of ​​interest,

[0107] - Determine (3025) at least one z-position (401) of the area of ​​interest,

[0108] - based on the z-position (401) of the area of ​​interest, determining (3035) a defocus value (702) of the area of ​​interest,

[0109] - Determine (3040) whether an autofocus trigger criterion is met, which depends on the temporal change (701) of the xy position and the defocus value (702), and

[0110] - optionally controlling a component (802, 805) of the medical visualization system (801) for focusing based on the defocus value (702) depending at least on the autofocus trigger criterion.

[0111] EXAMPLE 2. Control according to EXAMPLE 1, wherein the autofocus trigger criterion depends on a ratio of the temporal change (701) of the xy position to the defocus value (702).

[0112] EXAMPLE 3. Control according to EXAMPLE 2, wherein the autofocus trigger criterion depends on a confidence value in determining the area of ​​interest, wherein the confidence value depends on the ratio of the temporal change (701) of the xy position to the defocus value (702).

[0113] EXAMPLE 4. Control according to one of the preceding EXAMPLES, wherein the control (90, 899) is still set up to perform the following step:

[0114] - Determining (3030) a temporal change (402) of the z-position of the area of ​​interest, wherein the trigger criterion still depends on the temporal change of the z-position of the area of ​​interest.

[0115] EXAMPLE 5. Control according to EXAMPLE 2 or 3, as well as according to EXAMPLE 4, wherein the trigger criterion depends on a threshold comparison between a given threshold (411) and the temporal change (402) of the z-position of the area of ​​interest during a rolling time interval (412), wherein the length of the rolling time interval (412) depends on the ratio of the temporal change (701) of the xy-position to the defocus value (702).

[0116] EXAMPLE 6. Control according to EXAMPLE 5, wherein the specified threshold (411) depends on at least one depth of field range of an optical channel of the microscope or magnification of an optical channel of the microscope.

[0117] EXAMPLE 7. Control according to one of the preceding EXAMPLES, wherein the autofocus trigger criterion still depends on at least one depth of field range of an optical channel of the microscope (802) or magnification of an optical channel of the microscope (802).

[0118] EXAMPLE 8. Control according to one of the preceding EXAMPLES, where the autofocus trigger criterion continues to depend on a sign of the

[0119] Defocus value depends. EXAMPLE 9. Control according to one of the preceding EXAMPLES, with the control (90, 899) still set up to perform the following

[0120] Step to be performed:

[0121] - Determine (3010) whether the microscope images of the sequence of microscope images show a scene (50) with a deep channel (59), selectively checking the autofocus trigger criterion when the microscope images of the sequence of microscope images show the scene (50) with the deep channel (59).

[0122] EXAMPLE 10. Control according to one of the preceding EXAMPLES, wherein determining the change (701) of the xy position includes low-pass filtering.

[0123] EXAMPLE 11. Control according to one of the preceding EXAMPLES, wherein the control is still set up to determine the area of ​​interest based on an image analysis of the multiple microscope images.

[0124] EXAMPLE 12. Control according to one of the preceding EXAMPLES, wherein the component includes a varifocal lens of the microscope, or wherein the component includes a robotic microscope carrier of the medical visualization system.

[0125] EXAMPLE 13. Method for controlling a medical visualization system with a microscope, wherein the method comprises:

[0126] - Obtain (3005) a sequence of microscope images captured by the microscope depicting an area of ​​interest,

[0127] - Determining (3020) a temporal change (701) of an xy position of the area of ​​interest,

[0128] - Determine (3025) at least one z-position (401) of the area of ​​interest,

[0129] - based on the z-position (401) of the area of ​​interest, determining (3035) a defocus value (702) of the area of ​​interest, - determining (3040) whether an autofocus trigger criterion is met, which depends on the temporal change (701) of the xy-position as well as the defocus value (702), and

[0130] - optionally controlling a component (802, 805) of the medical visualization system (801) for focusing based on the defocus value (702) depending at least on the autofocus trigger criterion.

[0131] EXAMPLE 14. Method according to EXAMPLE 13, wherein the method is executed by the controller according to one of EXAMPLES 1 to 12.

[0132] Naturally, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or individually, without leaving the scope of the invention.

[0133] For example, several examples related to an operating room microscopy system implementing a medical visualization system have been presented above. However, other implementations for the medical visualization system would also be conceivable, such as an endoscopy system.

Claims

REQUIREMENTS 1. Controller (90, 899) for a medical visualization system (801) with a microscope (802), wherein the controller (90, 899) is configured to perform the following steps: - Obtain (3005) a sequence of microscope images captured by the microscope depicting an area of ​​interest, - Determining (3020) a temporal change (701) of an xy position of the area of ​​interest, - Determine (3025) at least one z-position (401) of the area of ​​interest, - based on the z-position (401) of the area of ​​interest, determining (3035) a defocus value (702) of the area of ​​interest, - Determine (3040) whether an autofocus trigger criterion is met, which depends on the temporal change (701) of the xy position and the defocus value (702), wherein the autofocus trigger criterion depends on a ratio of the temporal change (701) of the xy position to the defocus value (702), and - optionally controlling a component (802, 805) of the medical visualization system (801) for focusing based on the defocus value (702) depending at least on the autofocus trigger criterion.

2. Control according to claim 1, wherein the autofocus trigger criterion depends on a confidence value in determining the area of ​​interest, wherein the confidence value depends on the ratio of the temporal change (701) of the xy position to the defocus value (702).

3. Control according to claim 1 or 2, wherein the control (90, 899) is further configured to perform the following step: - Determining (3030) a temporal change (402) of the z-position of the area of ​​interest, where the trigger criterion still depends on the temporal change in the z-position of the area of ​​interest.

4. Control according to claim 3, wherein the trigger criterion depends on a threshold comparison between a predetermined threshold value (411) and the temporal change (402) of the z-position of the area of ​​interest during a rolling time interval (412), wherein the length of the rolling time interval (412) depends on the ratio of the temporal change (701) of the xy-position to the defocus value (702).

5. Control according to claim 4, wherein the predetermined threshold (411) depends on at least one depth of field range of an optical channel of the microscope or magnification of an optical channel of the microscope.

6. Control according to one of the preceding claims, wherein the autofocus trigger criterion further depends on at least one depth of field range of an optical channel of the microscope (802) or a magnification of an optical channel of the microscope (802).

7. Control according to one of the preceding claims, wherein the autofocus trigger criterion further depends on a sign of the defocus value.

8. Control according to one of the preceding claims, wherein the control (90, 899) is further configured to perform the following step: - Determine (3010) whether the microscope images of the sequence of microscope images show a scene (50) with a deep channel (59), selectively checking the autofocus trigger criterion when the microscope images of the sequence of microscope images show the scene (50) with the deep channel (59).

9. Control according to one of the preceding claims, wherein determining the change (701) of the xy-position is a Low-pass filtering is included.

10. Control according to one of the preceding claims, wherein the control is further configured to determine the area of ​​interest based on an image analysis of the multiple microscope images.

11. Control according to one of the preceding claims, wherein the component comprises a varifocal lens of the microscope, or wherein the component comprises a robotic microscope carrier of the medical visualization system.

12. Method for controlling a medical visualization system with a microscope, the method comprising: - Obtain (3005) a sequence of microscope images captured by the microscope depicting an area of ​​interest, - Determining (3020) a temporal change (701) of an xy position of the area of ​​interest, - Determine (3025) at least one z-position (401) of the area of ​​interest, - based on the z-position (401) of the area of ​​interest, determining (3035) a defocus value (702) of the area of ​​interest, - Determine (3040) whether an autofocus trigger criterion is met, which depends on the temporal change (701) of the xy position and the defocus value (702), wherein the autofocus trigger criterion depends on a ratio of the temporal change (701) of the xy position to the defocus value (702), and - optionally controlling a component (802, 805) of the medical visualization system (801) for focusing based on the defocus value (702) depending at least on the autofocus trigger criterion.

13. Method according to claim 12, wherein the method is executed by the control according to any one of claims 1 to 11.

Images