Disparity-based autofocus techniques for medical visualisation system

Abstract

The present disclosure relates to aspects of controlling a medical visualisation system with a stereoscopic microscope having a first and a second optical channel. The disclosure describes a controller configured to use different disparity calculation methods at different autofocus stages, for example. Special disparity calculation methods are disclosed. As a result, it is possible to determine a defocus value which is used for focusing. The use of different or specific disparity calculation methods makes it possible to improve the robustness and accuracy of the focusing.

Description

[0001] DESCRIPTION

[0002] DISPARITIES-BASED AUTOFOCUS TECHNIQUES FOR MEDICAL

[0003] VISUALIZATION SYSTEM

[0004] TECHNICAL AREA

[0005] Several examples of the disclosure relate to autofocus techniques. In particular, disparity-based autofocus techniques for a stereoscopic microscope are disclosed, which can use one or more autofocus stages.

[0006] BACKGROUND

[0007] In neurosurgical procedures, surgeons use an operating microscope or other medical visualization system. This typically involves using images with high optical magnification (up to 40x). Due to optical effects, these high magnifications result in a shallow depth of field, often only a few millimeters, causing a perceived blurriness in the image. To ensure the sharpest possible image at all times, autofocus techniques are used.

[0008] Known autofocus techniques in the art use, for example, two laser points in the surgical field. Triangulation is used to iteratively determine the physical distance of the operating microscope to the working plane in the surgical field. Thus, a defocus value is iteratively determined and minimized. However, a disadvantage of laser-based solutions is the requirement for laser hardware.

[0009] Image-based autofocus techniques are also known in the prior art. These do not require additional laser hardware, as they operate based on images captured by one or more cameras of the surgical microscope. Disparity-based autofocus techniques are a subset of image-based autofocus techniques. These operate based on stereoscopic image pairs. For example, US 2022 / 0311989 A1 describes an image-based autofocus technique that uses disparity calculation in a stereoscopic surgical microscope. A disparity value is determined as the numerical pixel shift between an identifiable region in the left image and the same identifiable region in the right image. The disparity value can be calculated by taking the identifiable region of the left image and performing a template match over a second (larger) area in the right image.A range of the working distance is sampled in a selected direction, and the disparity value is calculated at each step. As the image becomes increasingly sharp, the disparity value tends linearly towards zero from the extremes of the valid disparity range. When the disparity value is zero, the left and right images overlap and are not offset, resulting in a more focused image. Within a valid disparity range, the disparity value assumes valid / legitimate values, while in an initial outer range (at large and small defocus values), the disparity value may exhibit artifacts or nonsensical values.

[0010] Such an iterative procedure, as described in US 2022 / 0311989 A1 (or also in DE 201 710 125 453), is relatively time-consuming. The relevant components of the operating microscope must be mechanically adjusted several times to repeatedly set the working distance. This multiple adjustment of the working distance to scan a given area is necessary because the disparity value determined based on the template alignment assumes nonsensical values ​​in the outer regions (at large defocus values). Therefore, a wide range of working distances must be searched to enable a robust autofocus technique.

[0011] SUMMARY

[0012] There is therefore a need for improved autofocus techniques that address at least some of the aforementioned limitations and disadvantages. In particular, there is a need for fast and robust image-based autofocus techniques. This need is met by the features of the independent patent claims.

[0013] The features of the dependent patent claims define embodiments.

[0014] A controller for a medical visualization system is disclosed. The medical visualization system comprises a stereoscopic microscope. The stereoscopic microscope has a first optical channel and a second optical channel. The controller is configured to perform the following steps at each of one or more autofocus stages: based on a respective image pair of the first optical channel and the second optical channel, determine a respective disparity value based on one or more disparity calculation methods; based on the respective disparity value, determine a respective defocus value using a camera model of the first optical channel and the second optical channel; and control one or more components of the medical visualization system based on the respective defocus value.In at least one of the one or more autofocus stages, an area-based disparity calculation method is used.

[0015] A controller for a medical visualization system with a stereoscopic microscope is disclosed. The stereoscopic microscope has a first optical channel and a second optical channel. The controller is configured to perform the following steps in each of several autofocus stages: based on a corresponding image pair of the first and second optical channels, determining a disparity value based on a respective disparity calculation method; and based on the respective disparity value, determining a respective defocus value using a camera model of the first and second optical channels. Different disparity calculation methods are used in at least two of the several autofocus stages.

[0016] A method for controlling a medical visualization system with a stereoscopic microscope is disclosed, wherein the stereoscopic microscope has a first optical channel and a second optical channel. The method includes performing several autofocus stages, wherein an area-based disparity calculation method is used in at least one of the one or more autofocus stages.The procedure includes, in each of the one or more autofocus stages: determining a respective disparity value based on one or more disparity calculation methods based on a respective image pair of the first optical channel and the second optical channel; determining a respective defocus value based on the respective disparity value using a camera model of the first optical channel and the second optical channel; and controlling one or more components of the medical visualization system based on the respective defocus value.

[0017] A method for controlling a medical visualization system using a stereoscopic microscope is disclosed, wherein the stereoscopic microscope has a first optical channel and a second optical channel. The method includes performing multiple autofocus stages, employing different disparity calculation methods in two different autofocus stages. In each of the one or more autofocus stages, the method includes: determining a disparity value based on a corresponding image pair of the first and second optical channels using a respective disparity calculation method; and determining a respective defocus value based on the respective disparity value using a camera model of the first and second optical channels.

[0018] The features set out above and those described below can be used not only in the corresponding explicitly set out combinations, but also in further combinations or in isolation, without leaving the scope of protection of the present invention.

[0019] BRIEF DESCRIPTION OF THE FIGURES

[0020] FIG. 1 is a flowchart of an exemplary process.

[0021] FIG. 2 schematically illustrates an area-based measurement system.

[0022] Disparity calculation method according to various examples. FIG. 3 schematically illustrates an object-based disparity calculation method according to various examples.

[0023] FIG. 4 schematically illustrates an autofocus workflow according to various examples.

[0024] FIG. 5 schematically illustrates an autofocus workflow according to various examples.

[0025] FIG. 6 schematically illustrates a system according to various examples.

[0026] DETAILED DESCRIPTION

[0027] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings.

[0028] The present invention is explained in more detail below with reference to preferred embodiments and the drawings. In the figures, identical reference numerals denote identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily shown to scale. Rather, the various elements depicted in the figures are represented in such a way that their function and general purpose are understandable to a person skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings. A connection or coupling can be implemented as a wired or wireless connection. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0029] The following describes autofocus techniques for focusing a medical visualization system, such as a surgical microscope or an endoscope. The medical visualization system includes a microscope. Robust and fast autofocus techniques are described in particular.

[0030] The autofocus techniques described herein can be particularly beneficial for microsurgical procedures, such as neurosurgical interventions. This is because such microsurgical procedures often employ very high optical magnification (e.g., 30x or more), resulting in a particularly shallow depth of field. Furthermore, the vertical variations of the surgical site can be especially pronounced in these procedures, making a correctly positioned focal plane of the microscope with respect to the surgical site particularly important.

[0031] The techniques described herein do not involve an iterative search for optimal focus through repeated adjustments of the working distance or shifting of the focal plane. Instead, a disparity value is determined and then, using a camera model, this disparity value is "translated" into a defocus value; that is, the defocus value is determined based on the disparity value. This defocus value can be specified, for example, in a unit of length such as millimeters and quantifies the actual distance between the current focal plane of the microscope and the desired focal plane. For example, an area of ​​interest might be located at the desired focal plane. In this way, during a single adjustment process, for example, an autofocus module of a varifocal lens in a medical visualization system microscope or a robotic stand (for example, in microscopes with a fixed focal length objective) can shift the focal plane to the desired focal plane.The search for a minimum disparity value by iteratively adjusting the focal plane, as described, for example, in the prior art US 2022 / 0311989 A1, is unnecessary. This makes the techniques described herein particularly fast.

[0032] In particular, autofocus techniques that work on an image-based basis are described. This means, for example, that a laser for projecting laser points into the working area is unnecessary. Accordingly, the required hardware is simple and compact.

[0033] Techniques are described that can be used for a medical visualization system with a stereoscopic microscope. The stereoscopic microscope has a first and a second optical channel. The beam paths of these two optical channels define a stereo angle.

[0034] The optical channels are at least partially digital; this means that each optical channel is assigned a corresponding camera, which can be used to capture an image for that specific optical channel. An eyepiece may also be optionally included in a surgical microscope.

[0035] The autofocus techniques described herein are based on determining the disparity value using a pair of images from the two optical channels. The disparity value characterizes the different image positions of an object point in the image space of the two optical channels. If the object point lies in the focal plane, the image positions in the image space are identical; if the object point is out of focus, i.e., in front of or behind the focal plane, the image positions in the image space are different.

[0036] To determine the disparity value, one or more disparity calculation methods are used. Several techniques disclosed herein are based on the understanding that different disparity calculation methods each have their own advantages and disadvantages. Two possible disparity calculation methods that can be used in the various examples disclosed herein are described below in Table 1.

[0037] Table 1: Two examples of possible disparity calculation methods that can be used individually or in combination according to the techniques described herein. For example, different disparity calculation methods can be used at different autofocus levels.

[0038] Area-based disparity calculation methods – see Table 1 – use, for example, a reference sub-region in the first image of a stereoscopic image pair. Several values ​​for an area similarity measure are then calculated between the reference sub-region and various candidate sub-regions in the second image of the stereoscopic image pair. The second candidate sub-regions can be offset from each other and optionally cover the entire second image. An extremum of these comparison values, calculated using the area similarity measure, is then determined. This extremum corresponds to a particularly high degree of similarity between the reference sub-region and the candidate sub-regions. The disparity value (e.g., in units of image pixels) can then be determined based on this extremum, namely as the distance between the reference sub-region and the candidate sub-region associated with the extremum (e.g., minimum or maximum).Each sub-area has an extent of several pixels. The area similarity measure is based on the pixel values ​​in the respective sub-areas. Specifically, the area similarity measure does not operate on an object-based basis. This means that—when using the area-based disparity calculation method—no specific objects are located within the different sub-areas, for example, by their edges or corners. An example area similarity measure could determine the brightness distribution of the pixels in the images (possibly after preprocessing) in each sub-area (for example, for each color channel). For instance, a mean value of the distribution, a variance, and a covariance could be determined. Then, the brightness distribution or corresponding parameters such as mean, variance, etc., could be compared (different distance measures are conceivable). A correlation would also be possible.In such calculations, information about the spatial relationships between different pixels is discarded. Therefore, no individual objects or localized structures are considered. With the area-based disparity calculation method, the spatial relationship between individual pixels within the respective sub-area can be discarded or at least not taken into account. The area similarity measure eliminates the need to search for edges, textures, or other structures in individual pixels in highly blurred images (as with the object-based disparity method). No characteristic points (keypoints) need to be found, nor do the correspondences between these points need to be established. This makes the area similarity measure particularly robust against blurring. The "snapping area"—i.e.,The range of permissible defocus values ​​for which a meaningful disparity value can be calculated using the area-based disparity calculation method is relatively large.

[0039] In contrast, an object-based disparity calculation method uses the localization of individual object structures—such as edges, textures, patterns, or shapes—within the images or in sub-areas of the images. To locate such object structures, the spatial relationship between different pixels in the images (possibly after preprocessing) is relevant, because it is precisely this spatial relationship that enables the recognition of extended object structures. For example, an object recognition algorithm—perhaps machine-learned or template-based—could search for specific structures in the two images of an image pair, so that their distance can then be determined as a disparity value. It would also be conceivable to search for a specific reference structure in the first image of the image pair and then use template matching to search for this reference structure in the other image of the image pair.Examples of object-based disparity calculation methods include SURF, SIFT, and ORB. SURF (Speeded-Up Robust Features) is an algorithm for extracting and describing local image features, offering faster computation than SIFT. It utilizes integral images to accelerate the calculation of image intensity sums, making it more efficient in terms of computation time. SURF is particularly robust against object rotations and scaling and is frequently used in real-time applications. SIFT (Scale Invariant Feature Transform) is used for detecting and describing local image features and is especially known for its robustness against scaling, rotation, and lighting changes. The algorithm identifies characteristic keypoints in the image by comparing differences in Gaussian filtering at various scales (scale-space). These keypoints are then described by gradient histograms.ORB (Oriented FAST and Rotated BRIEF) combines modified versions of the FAST keypoint detector and the BRIEF descriptor to provide an efficient and robust feature-matching solution. ORB improves rotational invariance by calculating the orientation of the keypoints using intensity centroid moments and rotating the BRIEF descriptors accordingly. The algorithm is particularly well-suited for real-time applications because it is less computationally intensive than SIFT or SURF.

[0040] Several examples are based on the understanding that it can be desirable to switch between different disparity calculation methods, such as those listed in Table 1, depending on various criteria and / or dynamically and / or situationally. Several other examples are based on the understanding that it can be desirable to perform a sequence of several autofocus stages, using at least partially different disparity calculation methods in different autofocus stages.

[0041] For example, it would be conceivable to use a particularly robust disparity calculation method in an initial autofocus stage. This would ensure that even with a particularly large defocus value—for example, greater than the depth of field—the defocus value can still be reliably determined. In a subsequent autofocus stage, a particularly precise disparity calculation method (which might be less robust) could then be used to allow for fine-tuning of the focal plane. The disparity calculation methods used in the different autofocus stages can therefore exhibit complementary performance characteristics in terms of robustness and accuracy.Several examples are based in particular on the finding that an area-based disparity calculation method is especially suitable for reliably and robustly determining the defocus value in an initial autofocus stage—where, in principle, a large defocus may be present, for example, a defocus value greater than the depth of field of the microscope. On the other hand, an object-based disparity calculation method may be suitable if it can be assumed that there is no particularly strong defocus (i.e., the defocus value is smaller than the depth of field of the microscope). Such and other autofocus techniques are explained below with reference to FIG. 1.

[0042] Figure 1 is a flowchart of an exemplary procedure. The procedure shown in Figure 1 can be executed, for example, by a controller for a medical visualization system. For instance, a processor in the controller could load and execute program code from memory. When the processor executes the program code, this causes the various steps shown in Figure 1 to be carried out.

[0043] The medical visualization system could be a surgical microscope. The medical visualization system could be an endoscope. The medical visualization system features a stereoscopic microscope, which has a first optical channel and a second optical channel. The stereoscopic microscope has a field of view.

[0044] The procedure shown in FIG. 1 concerns autofocus techniques. In the procedure shown in FIG. 1, one or more autofocus stages 3007 are executed sequentially (running index / , indicating the respective iteration 3041). In each autofocus stage 3007, a disparity value is calculated (Box 3020, Box 3025) using one or more disparity calculation methods. This disparity value is then translated into a defocus value (Box 3030), and subsequently, one or more components of the medical visualization system are controlled based on the defocus value (Box 3035) to shift the focal plane. This is described in more detail below. Box 3005 checks whether an autofocus trigger is present. For example, there could be a user request for focusing. For example, a voice command could request autofocus.Other autofocus triggers are also conceivable, for example, adjusting or tilting the stereoscopic microscope. A periodic autofocus trigger (continuous autofocus) is also possible. An image-based autofocus trigger could also be implemented. In particular, with this image-based approach, the images used to trigger the autofocus can also be used in subsequent steps. An autofocus trigger could also be set by changing the area of ​​interest.

[0045] If box 3005 detects an autofocus trigger, an autofocus stage 3007 is subsequently executed. In principle, it is conceivable that several autofocus stages 3007 are executed sequentially (iterations 3041). The running index / indicates the autofocus stages 3007 and is incremented from iteration 3041 to iteration 3041; for example, the first autofocus stage 3007 after an autofocus trigger has the running index / =1; the second autofocus stage has the running index / =2; and so on.

[0046] In Box 3010, it is optionally possible to configure the current autofocus level 3007 according to the current iteration 3041. For example, it would be possible to define one or more settings for the autofocus level. It would also be possible to select or parameterize the one or more disparity calculation methods to be used in Box 3020. Different disparity calculation methods can be selected for different autofocus levels. Such a selection can be made depending on one or more performance characteristics of the disparity calculation methods. Examples of performance characteristics include robustness, speed, accuracy, capture area size, etc.

[0047] Alternatively or additionally, it would be conceivable, for example, to define a specific preprocessing of images in Box 3015. For instance, depending on the disparity calculation method to be used, different preprocessing of the images in Box 3015 could be configured. For example, the configuration could depend on / . For instance, at the first autofocus stage 3007 ( / =1), one or more disparity calculation methods could be used that are at least partially different from the one or more disparity calculation methods used at the subsequent, second autofocus stage 3007 ( / =2). Thus, at / =1, a particularly robust (but possibly relatively inaccurate) disparity calculation method could be used; and then at / =2, a less robust but particularly accurate disparity calculation method.For example, an area-based disparity calculation method could be used for / =1 and an object-based disparity calculation method for / =2 (see TABLE 1).

[0048] It would be conceivable (for / >1) that one or more settings of the current autofocus stage 3007 are determined depending on one or more settings of a preceding autofocus stage 3007 and / or depending on a result of the preceding autofocus stage 3007, that is, in the preceding iteration 3041 at / -1. For example, the disparity calculation method to be used in box 3020 in the current autofocus stage 3007 could be selected depending on the disparity calculation method used in the preceding autofocus stage 3007. For example, the disparity calculation method to be used in box 3020 in the current autofocus stage 3007 could be determined depending on the disparity value or the defocus value determined in the preceding autofocus stage 3007.For example, if the defocus value in the preceding autofocus stage 3007 was greater than a threshold (the threshold could be determined, for example, depending on the depth of field range), a particularly robust disparity calculation method could be selected; if, on the other hand, the defocus value was smaller than the threshold, a particularly accurate disparity calculation method could be selected.

[0049] Box 3015 contains a pair of images from the stereoscopic microscope. This image pair comprises two images depicting a surgical site. The two images offer different perspectives of the surgical site, with the corresponding stereo angle being known. The images can have, for example, an optical magnification of at least 20x, optionally at least 30x, and further optionally at least 40x.

[0050] Box 3015 can, for example, control the microscope's cameras to capture images. The images can also be loaded from memory.

[0051] Optionally, Box 3015 can include image preprocessing. For example, the images could be rescaled. Specifically, the image resolution could be reduced, meaning the images can be downscaled. Reducing the resolution reduces blurring; however, it also reduces spatial resolution—and thus the achievable accuracy for determining the defocus value. Several techniques described here are based on the understanding that, for example, when using an area-based disparity calculation method, a reduced resolution is less critical than with an object-based disparity calculation method. Such an area-based disparity calculation method, for instance, does not rely on the visibility of highly localized object features: at a reduced resolution, highly localized structures, for example, may disappear.On the other hand, the globally reduced uncertainty can also be advantageous for the accuracy of the area-based disparity calculation method. For example, a scaling factor for such a scaling could be set in Box 3010. For example, the scaling factor could be set depending on the disparity calculation method used. For example, if an area-based disparity calculation method is subsequently used, a stronger rescaling could occur than if an object-based disparity calculation method is subsequently used. For example, no rescaling could occur if an object-based disparity calculation method is subsequently used.

[0052] Box 3016 can optionally retrieve a specific area of ​​interest. This area of ​​interest lies within the microscope's field of view. For example, control data can be retrieved that specifies the XY position of the area of ​​interest and, if applicable, its extent within the microscope's field of view or in images acquired via the optical channels. Such positioning information for the area of ​​interest could, for instance, be determined using a camera model or triangulation based on the known stereo angle.

[0053] In particular, the various autofocus techniques described herein can utilize a specific region of interest (ROI) as the basis for determining the disparity value. Such techniques are based on the understanding that the surgical intervention region often exhibits height variations within the microscope's field of view. For example, in neurosurgical procedures, the intervention region might include a deep canal. In such cases, the autofocus technique can take into account whether the focus should be on the floor or the upper edge of the deep canal. This is specified by the region of interest. Several techniques exist in the prior art for determining the region of interest. Examples include tracking eye movements or detecting surgical tools or machine-readable markers.The area of ​​interest could also be marked on the image by the user. For example, an optical flow between successive images could be determined, and a center of activity identified. The specific technique used to determine the area of ​​interest is not crucial; various methods related to the autofocus techniques described herein can be employed.

[0054] In Box 3020, one or more candidate disparity values ​​are then determined. For each disparity calculation method active in the respective autofocus stage 3007, a corresponding candidate disparity value is determined. Different disparity calculation methods can yield different candidate disparity values ​​because they are associated with different errors.

[0055] For example, one or more of the disparity calculation methods described in Table 1 can be used in Box 3020 to determine a corresponding candidate disparity value. For example, a first candidate disparity value could be determined using the area-based disparity calculation method; a second candidate disparity value could also be determined using the object-based disparity calculation method.

[0056] If multiple disparity values ​​are determined in Box 3020 using different disparity calculation methods, then consolidation can take place in Box 3025.

[0057] In box 3025, a (final) disparity value is determined.

[0058] If only a single candidate disparity value was previously determined in Box 3020, then the final disparity value in Box 3025 corresponds to the candidate disparity value.

[0059] However, if several candidate disparity values ​​have been determined in Box 3020, the final disparity value can be determined in Box 3025, taking these values ​​into account. For example, a mean could be calculated. Alternatively, a comparison between the different candidate disparity values ​​could be made. If, for instance, a deviation between the candidate disparity values ​​exceeds a certain threshold, the disparity value determined using a particularly robust disparity calculation method could be selected (because the capture range is typically larger, so better results can be expected even in situations with high uncertainty). A significant deviation is indicative of a low confidence level.For example, it would be conceivable to determine the difference between the candidate disparity values ​​from Box 3020 and then perform a threshold comparison of this difference with a predefined threshold. Certain candidate disparity values ​​could then be discarded depending on the result of the threshold comparison. For instance, candidate disparity values ​​obtained using an object-based disparity calculation method could be discarded if the difference is greater than the predefined threshold. In Box 3030, a defocus value is then determined based on the disparity value from Box 3025. A camera model can be used for this purpose. For example, a pinhole camera model could be used. In this way, an XY shift in image space translates into a Z-distance in object space.

[0060] Focusing then takes place in Box 3035. For this purpose, one or more components of the medical visualization system can be controlled based on the defocus value from Box 3030. For example, a varifocal lens of the microscope in the medical visualization system could be controlled. It would also be conceivable to control a robotic stand to move the microscope in the z-direction, thus adjusting the focal plane relative to the surgical area.

[0061] Box 3040 then checks whether a further iteration 3041 should be performed, that is, whether another autofocus stage 3007 should be executed. This means that it can be checked whether a subsequent autofocus stage is optionally triggered. One or more different decision criteria can be considered in Box 3040. If no further autofocus stage 3007 is to be executed, the corresponding autofocus workflow is terminated, and the system waits for a new autofocus trigger (Box 3005). Otherwise, the value is incremented, and Box 3010 is executed again in the next iteration 3041.

[0062] An example of a decision criterion for the decision in Box 3040 is the magnitude of the current defocus value from the current iteration 3041 of Box 3030. For example, for relatively small (large) defocus values—for instance, in relation to the depth of field or another predefined threshold—it might be decided that a further autofocus stage 3007 is unnecessary (necessary). With a large defocus value, subsequent fine-tuning in a later autofocus stage 3007 can be helpful. With a small defocus value, it can be assumed that the relative error is already quite small.

[0063] Another exemplary decision criterion for the decision in Box 3040 concerns the disparity calculation method used in the preceding autofocus stage 3007. For example, if a less precise disparity calculation method is used in autofocus stage 3007 of the current iteration 3041 to calculate the disparity value (Box 3030), it could be decided that the subsequent autofocus stage 3007 is necessary and that a particularly precise disparity calculation method should be used there for fine-tuning. For example, if the defocus value in Box 3030 of the current iteration 3041 was calculated based on a disparity value determined using the area-based disparity calculation method, it could be decided that a further autofocus stage 3007 is necessary.

[0064] Another exemplary decision criterion for the decision in Box 3040 concerns the difference between different candidate disparity values, already discussed above in connection with Boxes 3020 and 3025. For example, if a particularly small difference is determined, a subsequent autofocus stage 3007 may be unnecessary. With a relatively large difference, a subsequent autofocus stage may be desirable. More generally, it would therefore be conceivable to consider a deviation between different results of several disparity calculation methods (see also FIG. 5, where a corresponding technique is described).

[0065] Another exemplary decision criterion for the decision in Box 3040 concerns the running index / . For example, in some scenarios it would be conceivable that a certain number of autofocus steps 3007 are always performed. For example, exactly two autofocus steps could be performed. With exactly two autofocus steps, a coarse adjustment can first be made; in which a well-defined defocus value can already be achieved, enabling a subsequent fine adjustment; the error remaining after this fine adjustment is then below a predetermined value.

[0066] The tolerance threshold is low, making further autofocus stages unnecessary. Using only two autofocus stages allows for very fast focusing, especially within a guaranteed execution time. Endless or lengthy iterative focusing is avoided.

[0067] Therefore, if i > 2, aborting can occur in Box 3040 and waiting for a new autofocus trigger (Box 3005). FIG. 2 illustrates aspects relating to an area-measure-based disparity calculation method. A “left” image 170 (from the left stereoscopic optical channel) and a “right” image 180 (from the right stereoscopic optical channel) are shown. In the left image 170, a reference subregion 171 is marked, centered on an area of ​​interest 190. In the right image 180, several candidate subregions 181–186 are shown, all shifted relative to each other and approximately aligned with the area of ​​interest 190. Comparisons can then be made between a measure determined for the reference sub-area 171 and measures determined for each of the candidate sub-areas 181-186 in order to determine an area similarity measure in each case.Such measures can be calculated, for example, based on the pixel values ​​of images 170 and 180 (possibly after preprocessing). In this calculation of the area similarity measure, spatial relationships between the pixel values ​​in the various sub-areas 171 and 181-186 can be disregarded. For example, pixel value histograms could be determined and compared. An average brightness could be calculated. These are just a few examples. The candidate sub-area 181-186 that exhibits the greatest area similarity to the reference sub-area 171 can serve as the basis for determining the disparity value. For example, if the candidate sub-area 186 has a measure that is closest to the measure of the reference sub-area 171, the disparity value can be determined as the distance between the center of the reference sub-area 171 and the center of the candidate sub-area 186.

[0068] FIG. 3 illustrates aspects relating to an object-based disparity calculation method. A specific object 271 is located at or near the area of ​​interest 190 in a right-hand image 270 of a stereoscopic image pair. Then, the same object 271 is found in the left-hand image 280 by template matching. The displacement 275 can be used as the disparity value.

[0069] FIG. 4 illustrates a schematic autofocus workflow according to various examples. For instance, the autofocus workflow from FIG. 4 could implement the procedure from FIG. 1. First, two images 470 and 480 are acquired in Box 3100 (compare FIG. 1: Box 3015). Then, a so-called "capture algorithm" is executed in Box 3105; that is, a disparity calculation method is used, which is particularly robust. In particular, an area-based disparity calculation method is used to calculate a disparity value (see FIG. 1: Boxes 3020 and 3025; and FIG. 2). Subsequently, the disparity value is converted into a defocus value, and, for example, a zoom lens is controlled accordingly (Box 3110; compare FIG. 1: Boxes 3030 and 3035). This completes a first autofocus stage ( / =1). A second autofocus stage ( / =2) is then performed for fine-tuning.In Box 3115, two images of a pair are retrieved, and then an object-based disparity calculation method is applied in Box 3120 to calculate a new disparity value. Based on this, the focus is then set in Box 3125. Instead of using an object-based disparity calculation method in Box 3120 (here, in particular, template matching as an example technique for locating identical objects in two images), it would also be conceivable to use an area-based disparity calculation method again.

[0070] FIG. 5 illustrates a schematic autofocus workflow according to various examples. For instance, the autofocus workflow in FIG. 5 could implement the procedure from FIG. 1. FIG. 5 illustrates a variant of the autofocus workflow from FIG. 4.

[0071] In FIG. 5, in the first autofocus stage ( / =1), based on images 470 and 480 obtained from 3200 images, two different disparity calculation methods are applied (3205). Specifically, an area-based disparity calculation method and an object-based disparity calculation method are used. Thus, two candidate disparity values ​​are obtained (compare FIG. 1: Box 3020). The difference between these candidate disparity values ​​is then calculated and compared to a threshold value (see 3210). Depending on the result of the threshold comparison, either Box 3250 or Box 3215 is performed. If the difference is less than the specified threshold value, Box 3250 is used. There, the candidate disparity value from the object-based disparity calculation method is used as the final disparity value. Based on the final disparity value, the defocus value is then calculated.Focusing then occurs in Box 3250, and the autofocus workflow ends (i.e., no further autofocus stage is performed; see FIG. 1: Box 3040; this corresponds to a "one-shot autofocus"). If, on the other hand, the difference between the candidate disparity values ​​is greater than the predefined threshold, the candidate disparity value from the area-based disparity calculation method is used as the final disparity value, and the defocus value is calculated in Box 3215 based on this final disparity value. Then, another autofocus stage (≠ 2) is performed, in which an object-based disparity calculation method is used in Box 3220 on another image pair to determine a corresponding disparity value. Based on this new disparity value, the defocus value is then determined in Box 3225, and the microscope's focal plane is shifted accordingly.

[0072] The variant of the autofocus workflow shown in FIG. 5 is based on the finding that the area-based disparity calculation method yields a less accurate disparity value than the object-based disparity calculation method. However, in the range where the object-based disparity calculation method also provides a reliable disparity value, the disparity values ​​of both methods should be similar. If they are similar (difference less than the threshold), a high level of confidence in the object-based disparity calculation method can be assumed; an additional autofocus stage is not necessary (Box 3250).If the disparity values ​​of the two disparity calculation methods differ significantly, a very blurry image (large defocus value) can be assumed; so that a coarse adjustment based on the area-based disparity calculation method ( / =1 ) can be followed by a fine adjustment based on the object-based disparity calculation method ( / =2).

[0073] FIG. 6 schematically illustrates a system 60 comprising a medical visualization system 69 with a stereoscopic microscope 61. In the example shown in FIG. 6, the medical visualization system 69 also includes a robotic stand, which is optional. The microscope 61 may, for example, have a varifocal lens. The medical visualization system 69 also includes one or more cameras 63, which can be used to capture microscope images for the two optical channels of the microscope 61. Furthermore, the system 60 includes a controller 64. The controller 64 may, for example, include a processor and memory. The processor can load and execute program code from memory. When the processor executes the program code, this causes autofocus techniques, as described therein, to be carried out. For example, the processor could then execute the procedure from FIG. 1 or one of the autofocus workflows from FIG. 4 or FIG. 5.The controller 64 can, for example, control the microscope 61 and / or the stand 62. The controller 64 can, for example, receive images from the cameras 63.

[0074] In summary, the preceding section described autofocus techniques that can perform one or more autofocus stages. In one method, defocus values ​​are determined based on both an area comparison measure and object matching. Thus, two different disparity calculation methods are used. Focusing on the area of ​​interest occurs based on the defocus value determined using the object-based disparity calculation method if the two defocus values ​​differ only slightly. In this case, the autofocus workflow is complete, and the system can wait for another autofocus trigger. Otherwise, focusing on the area of ​​interest occurs based on the defocus value determined using the area-based disparity calculation method.A further autofocus stage is then performed for fine-tuning; for this, the defocus value is determined using the object-based disparity calculation method, and refocusing is then performed based on this defocus value.

[0075] Naturally, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or individually, without leaving the scope of the invention.

Claims

REQUIREMENTS 1. Controller (64) for a medical visualization system (69) with a stereoscopic microscope (61), wherein the stereoscopic microscope (61) has a first optical channel and a second optical channel, wherein the controller (64) is configured to perform the following steps in each of one or more autofocus stages (3007): - based on a respective image pair of the first optical channel and the second optical channel, determining (3020, 3025) a respective disparity value based on one or more disparity calculation methods, - based on the respective disparity value, determining (3030) a respective defocus value using a camera model of the first optical channel and the second optical channel, - Controlling (3035) one or more components of the medical visualization system (69) based on the respective defocus value, wherein an area-based disparity calculation method is used in at least one of the one or more autofocus stages.

2. Control (64) according to claim 1, wherein the control (64) is further configured to selectively trigger a subsequent autofocus stage (3007) of one or more autofocus stages (3007) depending on one or more decision criteria (3040).

3. Control (64) according to claim 2, wherein the one or more decision criteria comprise a magnitude of the defocus value in the respective preceding autofocus stage.

4. Control (64) according to claim 2 or 3, wherein the one or more decision criteria comprise a disparity calculation method used in the preceding autofocus stage (3007).

5. Control (64) according to one of claims 2 to 4, wherein the one or more decision criteria comprise a deviation between the results of several disparity calculation procedures determined in the respective preceding autofocus stage (3007).

6. Control (64) according to one of the preceding claims, wherein in at least one of the one or more autofocus stages (3007) at least one of the one or more disparity calculation methods is an object-based disparity calculation method.

7. Control (64) according to one of the preceding claims, comprising the area-based disparity calculation method: - Selecting a reference sub-area (171) in a first image (170) of the respective image pair, - Calculating a respective area similarity measure between the reference sub-area (171) and each of several second candidate sub-areas (181, 182, 183, 184, 185, 186) in a second image (180) of the respective image pair, and - Determining the disparity value based on an extremum of the area similarity measures.

8. Control (64) according to one of the preceding claims, wherein the control (64) is configured to determine one or more settings of a subsequent autofocus stage (3007) of the one or more autofocus stages (3007) depending on one or more settings of a preceding autofocus stage (3007) and / or a result of the preceding autofocus stage (3007).

9. Control (64) according to one of the preceding claims, wherein the one or more autofocus levels (3007) comprise exactly two autofocus levels (3007).

10. Control (64) according to one of the preceding claims, wherein the control (64) is configured to use the area-based disparity calculation method in a first autofocus stage (3007) of the one or more autofocus stages (3007), wherein the control (64) is further configured to use an object-based disparity calculation method in a second autofocus stage (3007) of the one or more autofocus stages (3007).

11. Control (64) according to one of the preceding claims, wherein the control (64) is configured to use several different disparity calculation methods in at least one of the one or more autofocus stages (3007) to determine several candidate disparity values ​​(3020), wherein the control (64) is further configured to determine the disparity value based on the several candidate disparity values ​​(3025).

12. Control (64) according to claim 11, wherein the control (64) is configured to determine the disparity value based on a comparison of the multiple candidate disparity values.

13. Control (64) according to claim 12, wherein the control (64) is configured to determine a difference of the candidate disparity values ​​and to perform a threshold comparison of this difference with a predetermined threshold, wherein the control (64) is further configured to reject one or more of the candidate disparity values ​​depending on a result of the threshold comparison.

14. Control (64) according to claim 13, wherein the control (64) is configured to reject, if the difference is greater than the specified threshold, such candidate disparity values ​​obtained by an object-based disparity calculation method.

15. Controller (64) for a medical visualization system (69) with a stereoscopic microscope (61), wherein the stereoscopic microscope (61) has a first optical channel and a second optical channel, wherein the controller (64) is configured to perform the following steps in each of several autofocus stages (3007): - based on a corresponding image pair of the first optical channel and the second optical channel, determining (3020, 3025) a disparity value based on a respective disparity calculation method, and - based on the respective disparity value, determining (3030) a respective defocus value using a camera model of the first optical channel and the second optical channel, using different disparity calculation methods in two different autofocus levels (3007) of the multiple autofocus levels (3007).

16. Control (64) according to claim 15, wherein the different disparity calculation methods have complementary performance characteristics with respect to robustness and accuracy.

17. Method for controlling a medical visualization system using a stereoscopic microscope (61) wherein the stereoscopic microscope (61) has a first optical channel and a second optical channel, the method comprising: - Performing multiple autofocus stages, wherein at least one of the one or more autofocus stages uses an area-measure-based disparity calculation method, the method comprising in each of the one or more autofocus stages: - based on a respective image pair of the first optical channel and the second optical channel, determining (3020, 3025) a respective disparity value based on one or more disparity calculation methods, - based on the respective disparity value, determining (3030) a respective defocus value using a camera model of the first optical channel and the second optical channel, and - Controlling (3035) one or more components of the medical visualization system (69) based on the respective defocus value.

18. Method according to claim 17, wherein the method is executed by the control according to any one of claims 1 to 14.

19. Method for controlling a medical visualization system (69) using a stereoscopic microscope (61), wherein the stereoscopic microscope (61) has a first optical channel and a second optical channel, the method comprising: - Performing multiple autofocus stages, wherein different disparity calculation methods are used in two different autofocus stages (3007) of the multiple autofocus stages (3007), wherein the method in each of the one or more autofocus stages comprises: - based on a corresponding image pair of the first optical channel and the second optical channel, determining (3020, 3025) a disparity value based on a respective disparity calculation method, and - based on the respective disparity value, determining (3030) a respective defocus value using a camera model of the first optical channel and the second optical channel, using different disparity calculation methods in two different autofocus levels (3007) of the multiple autofocus levels (3007).

20. Method according to claim 19, wherein the method is executed by the control according to claim 15 or 16.

Images