DISPARITIES-BASED AUTOFOCUS TECHNIQUES FOR MEDICAL VISUALIZATION SYSTEMS

Abstract

This disclosure relates to aspects of the control of a medical visualization system with a stereoscopic microscope having a first and a second optical channel. The disclosure describes a control system configured to use different disparity calculation methods at various autofocus levels. Specific disparity calculation methods are disclosed. This allows a defocus value to be determined, which is used for focusing. The use of different or specific disparity calculation methods enables an improvement in the robustness and accuracy of the focusing.

Description

TECHNICAL AREA

[0001] Several examples of the disclosure relate to autofocus techniques. In particular, disparity-based autofocus techniques for a stereoscopic microscope are disclosed, which can use one or more autofocus stages. BACKGROUND

[0002] In neurosurgical procedures, surgeons use an operating microscope or other medical visualization system. This typically involves using images with high optical magnification (up to 40x). Due to optical effects, these high magnifications result in a shallow depth of field, often only a few millimeters, causing a perceived blurriness in the image. To ensure the sharpest possible image at all times, autofocus techniques are used.

[0003] Known autofocus techniques in the art use, for example, two laser points in the surgical field. Triangulation is used to iteratively determine the physical distance of the operating microscope to the working plane in the surgical field. Thus, a defocus value is iteratively determined and minimized. However, a disadvantage of laser-based solutions is the requirement for laser hardware.

[0004] Image-based autofocus techniques are also known in the prior art. These do not require additional laser hardware, as they operate based on images captured by one or more cameras of the surgical microscope. Disparity-based autofocus techniques are a subset of image-based autofocus techniques. These operate based on stereoscopic image pairs. For example, US 2022 / 0311989 A1 describes an image-based autofocus technique that uses disparity calculation in a stereoscopic surgical microscope. A disparity value is determined as the numerical pixel shift between an identifiable region in the left image and the same identifiable region in the right image. The disparity value can be calculated by taking the identifiable region of the left image and performing a template match over a second (larger) area in the right image.A range of the working distance is sampled in a selected direction, and the disparity value is calculated at each step. As the image becomes increasingly sharp, the disparity value tends linearly towards zero from the extremes of the valid disparity range. When the disparity value is zero, the left and right images overlap and are not offset, resulting in a more focused image. Within a valid disparity range, the disparity value assumes valid / legitimate values, while in an initial outer range (at large and small defocus values), the disparity value may exhibit artifacts or nonsensical values.

[0005] Such an iterative procedure, as described in US 2022 / 0311989 A1 (or also in DE 10 2017 125 453 B3), is relatively time-consuming. The relevant components of the operating microscope must be mechanically adjusted several times to repeatedly set the working distance. This multiple adjustment of the working distance to scan a given area is necessary because the disparity value determined based on the template alignment assumes nonsensical values ​​in the outer regions (at large defocus values). Therefore, a wide range of working distances must be searched to enable a robust autofocus technique.

[0006] WO 2020 / 121457 A1 describes a microscope comprising: an optical system that causes a right-side imaging element to produce an image of right-side observation object light based on the right-side observation object light received from an observation object, and that causes a left-side imaging element to produce an image of left-side observation object light based on the left-side observation object light showing the observation object; an adjustment device for adjusting the focus position of the optical system relative to the observation object based on an input instruction;and a control unit for displaying suggestion information on a display unit, which suggests the instruction content required for a focus position adjustment by the adjusting device in order to align the focus position with a specific area of ​​the observed object.

[0007] US 2017 / 0017136A1 describes an autofocus method and an image acquisition device comprising a distance sensor and an image sensor with a lens and a sensor array. The method includes the following steps: First, distance sensor data and phase detection data are obtained from the distance sensor and the sensor array, respectively. The lens is controlled to perform a coarse search within a focus area by switching between a distance sensor autofocus mode and a phase detection autofocus mode based on the distance sensor and phase detection data to determine a fine search area. The lens is further controlled to perform a fine search within the fine search area in a contrast detection autofocus mode to determine an optimal focusing lens position. Subsequently, the lens is controlled to move to the optimal focusing lens position.

[0008] US 2023 / 0140956 A1 describes a method and system for automatically optimizing stereoscopic 3D perception, as well as a medium. The method comprises the following steps, performed sequentially: Step 1: Starting with the current left and right images, a stereo disparity is calculated to generate a disparity map; Step 2: A depth value is calculated for each individual pixel using the calculated disparity; Step 3: A depth distance of an object to be observed is calculated; Step 4: Using the depth distance calculated in Step 3, corresponding displacement values ​​are acquired for the left and right images; and Step 5: The acquired image displacement values ​​are applied to a 3D display.The advantageous effect of the present invention is that the method for automatically optimizing stereoscopic 3D perception according to the present invention eliminates the fatigue and dizziness that easily occur when using a 3D endoscope.

[0009] EP 3 496 398 A1 describes a safe stereo camera comprising at least a first image sensor for capturing a first raw image and a second image sensor for capturing a second raw image of a monitored area from a displaced perspective, a stereoscopic unit for generating a depth map from the two raw images, and a test unit for verifying the functionality of the image sensors. The test unit is designed to detect errors in the image sensors by comparing first image information from a first image window of the first raw image with second image information from a second image window of the second raw image and by selecting the image windows based on information from the stereoscopic unit when generating the depth map.

[0010] EP 4 006 615 A1 describes an imaging system for a microscope, comprising a processor configured to receive first, second, and third images. The processor determines a disparity map from the second and third images. A fourth image is determined from the first image and the disparity map. A surgical microscope comprising the imaging system is also disclosed. Furthermore, a method for determining an image is disclosed, comprising receiving a first, second, and third image, determining a disparity map, and determining a fourth image from the first image based on the disparity map.

[0011] DE 10 2009 045 107 A1 discloses the use of several different focusing functions, which enables significantly improved focusing, since it can be assumed that not every focusing function will produce incorrect focus change directions, even if at least two shots are taken in the far focus range of the respective focusing function. Because all focus change directions are taken into account when determining the principal change direction, a better basis for deciding the correct principal change direction is provided.

[0012] DE 10 2007 003 059 A1 discloses a method for determining an optimized focus criterion, with the aim of implementing operator-independent passive focusing in optical length measurement technology. Instead of just one, two, or three focus criteria, a focus criterion combined from a plurality (≥ 5) of individual focus criteria is used, wherein the individual focus criteria differ only in their operating principle.

[0013] DE 42 26 523 A1 discloses a method for automatically focusing on objects during image processing of microscopic specimens. Signals are processed from the partial image, based on which one or more focus criteria are calculated.

[0014] US 9,207,444 B2 discloses a focusing device that determines disparities between multiple images. In this device, an imaging area is divided into a plurality of layers, the number of layers being based on the first depth of field of the lens; a separate focusing position is determined for each layer. SUMMARY

[0015] Therefore, there is a need for improved autofocus techniques that address at least some of the limitations and disadvantages mentioned above. In particular, there is a need for fast and robust image-based autofocus techniques.

[0016] This task is solved by the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0017] A control system for a medical visualization system according to claim 1 is disclosed.

[0018] A method for controlling a medical visualization system with a stereoscopic microscope according to claim 10 is disclosed.

[0019] The features set out above and those described below can be used not only in the corresponding explicitly set out combinations, but also in further combinations or in isolation, without leaving the scope of protection of the present invention. BRIEF DESCRIPTION OF THE FIGURES Fig. Figure 1 is a flowchart of an example procedure. Fig. Figure 2 schematically illustrates an area-based disparity calculation method according to various examples. Fig. Figure 3 schematically illustrates an object-based disparity calculation method according to various examples. Fig. Figure 4 schematically illustrates an autofocus workflow according to various examples. Fig.Figure 5 schematically illustrates an autofocus workflow according to various examples. Fig. Figure 6 schematically illustrates a system according to various examples. DETAILED DESCRIPTION

[0020] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings.

[0021] The present invention is explained in more detail below with reference to preferred embodiments and the drawings. In the figures, identical reference numerals denote identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily shown to scale. Rather, the various elements depicted in the figures are represented in such a way that their function and general purpose are understandable to a person skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings. A connection or coupling can be implemented as a wired or wireless connection. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0022] The following describes autofocus techniques for focusing a medical visualization system, such as a surgical microscope or an endoscope. The medical visualization system includes a microscope. Robust and fast autofocus techniques are described in particular.

[0023] The autofocus techniques described herein can be particularly beneficial for microsurgical procedures, such as neurosurgical interventions. This is because such microsurgical procedures often employ very high optical magnification (e.g., 30x or more), resulting in a particularly shallow depth of field. Furthermore, the vertical variations of the surgical site can be especially pronounced in these procedures, making a correctly positioned focal plane of the microscope with respect to the surgical site particularly important.

[0024] The techniques described herein do not involve an iterative search for optimal focus through repeated adjustments of the working distance or shifting of the focal plane. Instead, a disparity value is determined and then, using a camera model, this disparity value is "translated" into a defocus value; that is, the defocus value is determined based on the disparity value. This defocus value can be specified, for example, in a unit of length such as millimeters and quantifies the actual distance between the current focal plane of the microscope and the desired focal plane. For example, an area of ​​interest might be located at the desired focal plane. In this way, during a single adjustment process, for example, an autofocus module of a varifocal lens in a medical visualization system microscope or a robotic stand (for example, in microscopes with a fixed focal length objective) can shift the focal plane to the desired focal plane.The search for a minimum disparity value by iteratively adjusting the focal plane, as described, for example, in the prior art US 2022 / 0311989 A1, is unnecessary. This makes the techniques described herein particularly fast.

[0025] In particular, autofocus techniques that work on an image-based basis are described. This means, for example, that a laser for projecting laser points into the working area is unnecessary. Accordingly, the required hardware is simple and compact.

[0026] Techniques are described that can be used for a medical visualization system with a stereoscopic microscope. The stereoscopic microscope has a first and a second optical channel. The beam paths of these two optical channels define a stereo angle.

[0027] The optical channels are at least partially digital; this means that each optical channel is assigned a corresponding camera, which can be used to capture an image for that specific optical channel. An eyepiece may also be optionally included in a surgical microscope.

[0028] The autofocus techniques described herein are based on determining the disparity value using a pair of images from the two optical channels. The disparity value characterizes the different image positions of an object point in the image space of the two optical channels. If the object point lies in the focal plane, the image positions in the image space are identical; if the object point is out of focus, i.e., in front of or behind the focal plane, the image positions in the image space are different.

[0029] To determine the disparity value, one or more disparity calculation methods are used. Several techniques disclosed herein are based on the understanding that different disparity calculation methods each have their own advantages and disadvantages. Two possible disparity calculation methods that can be used in the various examples disclosed herein are described below in Table 1. DESIGNATION ADVANTAGES AND DISADVANTAGES 1 Area-based disparity calculation method ▪ Robust, i.e., also applicable to blurred structures ▪ Partly inaccurate because averaged over a larger area 2 Object-based disparity calculation method ▪ Not very robust, because it is unreliable with blurred structures ▪ If structures can be located, very precisely

[0030] Table 1: Two examples of possible disparity calculation methods that can be used individually or in combination according to the techniques described herein. For example, different disparity calculation methods can be used at different autofocus levels.

[0031] Area-based disparity calculation methods – see Table 1 – use, for example, a reference sub-region in the first image of a stereoscopic image pair. Several values ​​for an area similarity measure are then calculated between the reference sub-region and various candidate sub-regions in the second image of the stereoscopic image pair. The second candidate sub-regions can be offset from each other and optionally cover the entire second image. An extremum of these comparison values, calculated using the area similarity measure, is then determined. This extremum corresponds to a particularly high degree of similarity between the reference sub-region and the candidate sub-regions. The disparity value (e.g., in units of image pixels) can then be determined based on this extremum, namely as the distance between the reference sub-region and the candidate sub-region corresponding to the extremum (e.g., minimum or maximum).Each sub-area has an extent of several pixels. The area similarity measure is based on the pixel values ​​in the respective sub-areas. Specifically, the area similarity measure does not operate on an object-based basis. This means that—when using the area-based disparity calculation method—no specific objects are located within the different sub-areas, for example, by their edges or corners. An example area similarity measure could determine the brightness distribution of the pixels in the images (possibly after preprocessing) in each sub-area (for example, for each color channel). For instance, a mean value of the distribution, a variance, and a covariance could be determined. Then, the brightness distribution or corresponding parameters such as mean, variance, etc., could be compared (different distance measures are conceivable). A correlation would also be possible.In such calculations, information about the spatial relationships between different pixels is discarded. Therefore, no individual objects or localized structures are considered. With the area-based disparity calculation method, the spatial relationship between individual pixels within the respective sub-area can be discarded or at least not taken into account. The area similarity measure eliminates the need to search for edges, textures, or other structures in individual pixels in highly blurred images (as with the object-based disparity method). No characteristic points (keypoints) need to be found, nor do the correspondences between these points need to be established. This makes the area similarity measure particularly robust against blurring. The "snapping area"—i.e.,The range of permissible defocus values ​​for which a meaningful disparity value can be calculated using the area-based disparity calculation method is relatively large.

[0032] In contrast, an object-based disparity calculation method uses the localization of individual object structures—such as edges, textures, patterns, or shapes—within the images or in sub-areas of the images. To locate such object structures, the spatial relationship between different pixels in the images (possibly after preprocessing) is relevant, because it is precisely this spatial relationship that enables the recognition of extended object structures. For example, an object recognition algorithm—perhaps machine-learned or template-based—could search for specific structures in the two images of an image pair, so that their distance can then be determined as a disparity value. It would also be conceivable to search for a specific reference structure in the first image of the image pair and then use template matching to search for this reference structure in the other image of the image pair.Examples of object-based disparity calculation methods include SURF, SIFT, and ORB. SURF (Speeded-Up Robust Features) is an algorithm for extracting and describing local image features, offering faster computation than SIFT. It utilizes integral images to accelerate the calculation of image intensity sums, making it more efficient in terms of computation time. SURF is particularly robust against object rotations and scaling and is frequently used in real-time applications. SIFT (Scale Invariant Feature Transform) is used for detecting and describing local image features and is especially known for its robustness against scaling, rotation, and lighting changes. The algorithm identifies characteristic keypoints in the image by comparing differences in Gaussian filtering at various scales (scale-space). These keypoints are then described by gradient histograms.ORB (Oriented FAST and Rotated BRIEF) combines modified versions of the FAST keypoint detector and BRIEF descriptor to provide an efficient and robust feature-matching solution. ORB improves rotational invariance by calculating the orientation of the keypoints using intensity centroid moments and rotating the BRIEF descriptors accordingly. The algorithm is particularly well-suited for real-time applications because it is less computationally intensive than SIFT or SURF.

[0033] Several examples are based on the understanding that it can be desirable to switch between different disparity calculation methods, such as those listed in Table 1, depending on various criteria and / or dynamically and / or situationally. Several other examples are based on the understanding that it can be desirable to perform a sequence of several autofocus stages, using at least partially different disparity calculation methods in different autofocus stages. For example, it would be conceivable to use a particularly robust disparity calculation method in an initial autofocus stage. This would ensure that even with a particularly large defocus value—for example, greater than the depth of field—the defocus value can still be reliably determined.In a subsequent autofocus stage, a particularly precise disparity calculation method (which may be less robust) can then be used to fine-tune the focal plane. The disparity calculation methods used in the different autofocus stages can therefore exhibit complementary performance characteristics with regard to robustness and accuracy. Several examples are based, in particular, on the finding that an area-based disparity calculation method is especially suitable for reliably and robustly determining the defocus value in an initial autofocus stage—where, in principle, a large defocus may be present, for example, a defocus value greater than the depth of field of the microscope.On the other hand, an object-based disparity calculation method may be suitable if it can be assumed that there is no particularly strong defocus (i.e., the defocus value is smaller than the depth of field of the microscope). Such and other autofocus techniques are described below using [examples of techniques]. Fig. 1 explained.

[0034] Fig. Figure 1 is a flowchart of an example procedure. The procedure is from Fig. 1. This can be executed, for example, by a controller for a medical visualization system. For instance, a processor in the controller could load and execute program code from memory. When the processor executes the program code, this causes the various steps from Fig. 1 will be executed.

[0035] The medical visualization system could be a surgical microscope. The medical visualization system could be an endoscope. The medical visualization system features a stereoscopic microscope, which has a first optical channel and a second optical channel. The stereoscopic microscope has a field of view.

[0036] The procedure from Fig. 1 concerns autofocus techniques. In the process of Fig.1. One or more autofocus stages 3007 are executed sequentially (running index i, which indicates the respective iteration 3041). In each autofocus stage 3007, a disparity value is calculated (Box 3020, Box 3025) using one or more disparity calculation methods. This disparity value is then translated into a defocus value (Box 3030), and subsequently, one or more components of the medical visualization system are controlled based on the defocus value (Box 3035) to shift the focal plane. This is described in more detail below.

[0037] Box 3005 checks for an autofocus trigger. For example, a user request for focus might be received. A voice command could also request autofocus. Other autofocus triggers are conceivable, such as adjusting or tilting the stereoscopic microscope. A periodic autofocus trigger (continuous autofocus) is also possible. An image-based autofocus trigger is also an option. In this image-based approach, the images used to trigger the autofocus can be reused in subsequent steps. A change in the area of ​​interest could also trigger an autofocus.

[0038] If box 3005 detects an autofocus trigger, an autofocus stage 3007 is subsequently executed. In principle, it is conceivable that several autofocus stages 3007 are executed sequentially (iterations 3041). The running index i indicates the autofocus stages 3007 and is incremented from iteration 3041 to iteration 3041; for example, the first autofocus stage 3007 after an autofocus trigger has the running index i=1; the second autofocus stage has the running index i=2; and so on.

[0039] In Box 3010, it is optionally possible to configure the current autofocus level 3007 according to the current iteration 3041. For example, it would be possible to define one or more settings for the autofocus level. It would also be possible to select or parameterize the one or more disparity calculation methods to be used in Box 3020. Different disparity calculation methods can be selected for different autofocus levels. Such a selection can be made depending on one or more performance characteristics of the disparity calculation methods. Examples of performance characteristics include robustness, speed, accuracy, capture area size, etc.

[0040] Alternatively or additionally, it would be conceivable, for example, to define a specific preprocessing of images in Box 3015. For instance, depending on the disparity calculation method to be used, different preprocessing of the images in Box 3015 could be configured.

[0041] For example, the configuration could depend on i. For instance, at the first autofocus stage 3007 (i=1), one or more disparity calculation methods could be used that are at least partially different from the one or more disparity calculation methods used at the subsequent, second autofocus stage 3007 (i=2). Thus, at i=1, a particularly robust (but possibly relatively inaccurate) disparity calculation method could be used; and then at i=2, a less robust but particularly accurate disparity calculation method. For example, an area-based disparity calculation method could be used at i=1, and an object-based disparity calculation method at i=2 (see Table 1).

[0042] It would be conceivable (for i>1) that one or more settings of the current autofocus stage 3007 are determined depending on one or more settings of a preceding autofocus stage 3007 and / or depending on a result of the preceding autofocus stage 3007, that is, in the preceding iteration 3041 at i-1. For example, the disparity calculation method to be used in box 3020 in the current autofocus stage 3007 could be selected depending on the disparity calculation method used in the preceding autofocus stage 3007. For example, the disparity calculation method to be used in box 3020 in the current autofocus stage 3007 could be determined depending on the disparity value or the defocus value determined in the preceding autofocus stage 3007.For example, if the defocus value in the preceding autofocus stage 3007 was greater than a threshold (the threshold could be determined, for example, depending on the depth of field range), a particularly robust disparity calculation method could be selected; if, on the other hand, the defocus value was smaller than the threshold, a particularly accurate disparity calculation method could be selected.

[0043] Box 3015 contains a pair of images obtained from the stereoscopic microscope. This image pair comprises two images depicting a surgical site. The two images offer different perspectives of the surgical site, and the corresponding stereo angle is known.

[0044] The images can, for example, have an optical magnification of at least 20x, optionally at least 30x, and further optionally at least 40x.

[0045] Box 3015 can, for example, control the microscope's cameras to capture images. The images can also be loaded from memory.

[0046] Optionally, Box 3015 can include image preprocessing. For example, the images could be rescaled. Specifically, the image resolution could be reduced, meaning the images can be downscaled. Reducing the resolution reduces blurring; however, it also reduces spatial resolution—and thus the achievable accuracy for determining the defocus value. Several techniques described here are based on the understanding that, for example, when using an area-based disparity calculation method, a reduced resolution is less critical than with an object-based disparity calculation method. Such an area-based disparity calculation method, for instance, does not rely on the visibility of highly localized object features: at a reduced resolution, highly localized structures, for example, may disappear.On the other hand, the globally reduced uncertainty can also be advantageous for the accuracy of the area-based disparity calculation method. For example, a scaling factor for such a scaling could be set in Box 3010. For example, the scaling factor could be set depending on the disparity calculation method used. For example, if an area-based disparity calculation method is subsequently used, a stronger rescaling could occur than if an object-based disparity calculation method is subsequently used. For example, no rescaling could occur if an object-based disparity calculation method is subsequently used.

[0047] Box 3016 can optionally retrieve a specific area of ​​interest. This area of ​​interest lies within the microscope's field of view. For example, control data can be retrieved that specifies the XY position of the area of ​​interest and, if applicable, its extent within the microscope's field of view or in images acquired via the optical channels. Such positioning information for the area of ​​interest could, for instance, be determined using a camera model or triangulation based on the known stereo angle.

[0048] In particular, the various autofocus techniques described herein can utilize a specific region of interest (ROI) as the basis for determining the disparity value. Such techniques are based on the understanding that the surgical intervention region often exhibits height variations within the microscope's field of view. For example, in neurosurgical procedures, the intervention region might include a deep canal. In such cases, the autofocus technique can take into account whether the focus should be on the floor or the upper edge of the deep canal. This is specified by the region of interest. Several techniques exist in the prior art for determining the region of interest. Examples include tracking eye movements or detecting surgical tools or machine-readable markers.The area of ​​interest could also be marked on the image by the user. For example, an optical flow between successive images could be determined, and a center of activity identified. The specific technique used to determine the area of ​​interest is not crucial; various methods related to the autofocus techniques described herein can be employed.

[0049] In Box 3020, several candidate disparity values ​​are then determined. For each disparity calculation method active in the respective autofocus stage 3007, a corresponding candidate disparity value is determined. Different disparity calculation methods can yield different candidate disparity values ​​because they are associated with different errors.

[0050] For example, one or more of the disparity calculation methods described in Table 1 can be used in Box 3020 to determine an associated candidate disparity value.

[0051] For example, a first candidate disparity value could be determined using the area-based disparity calculation method; a second candidate disparity value could also be determined using the object-based disparity calculation method.

[0052] If multiple disparity values ​​are determined in Box 3020 using different disparity calculation methods, then consolidation can take place in Box 3025.

[0053] In box 3025, a (final) disparity value is determined.

[0054] If multiple candidate disparity values ​​have been determined in Box 3020, the final disparity value can be determined in Box 3025, taking these values ​​into account. For example, a mean value could be calculated. A comparison between the different candidate disparity values ​​is also performed. If, for example, a deviation between the candidate disparity values ​​exceeds a certain threshold, the disparity value determined using a particularly robust disparity calculation method is selected (because the capture range is typically larger, so better results can be expected even in situations with high uncertainty). A significant deviation is indicative of a low confidence level.The difference between the candidate disparity values ​​from box 3020 is determined, and then a threshold comparison of this difference with a predefined threshold is performed. Certain candidate disparity values ​​are then discarded depending on the result of the threshold comparison. For example, candidate disparity values ​​obtained using an object-based disparity calculation method could be discarded if the difference is greater than the predefined threshold.

[0055] In box 3030, a defocus value is then determined based on the disparity value from box 3025. A camera model can be used for this purpose. For example, a pinhole camera model could be used. Thus, an XY shift in image space is translated into a Z-distance in object space.

[0056] Focusing then takes place in Box 3035. For this purpose, one or more components of the medical visualization system can be controlled based on the defocus value from Box 3030. For example, a varifocal lens of the microscope in the medical visualization system could be controlled. It would also be conceivable to control a robotic stand to move the microscope in the z-direction, thus adjusting the focal plane relative to the surgical site.

[0057] Box 3040 then checks whether a further iteration 3041 should be performed, that is, whether another autofocus stage 3007 should be executed. This means that it can be checked whether a subsequent autofocus stage is optionally triggered. One or more different decision criteria can be considered in Box 3040. If no further autofocus stage 3007 is to be executed, the corresponding autofocus workflow is terminated, and the system waits for a new autofocus trigger (Box 3005). Otherwise, i is incremented, and Box 3010 is executed again, in the next iteration 3041.

[0058] An example of a decision criterion for the decision in Box 3040 is the magnitude of the current defocus value from the current iteration 3041 of Box 3030. For example, for relatively small (large) defocus values—for instance, in relation to the depth of field or another predefined threshold—it might be decided that a further autofocus stage 3007 is unnecessary (necessary). With a large defocus value, subsequent fine-tuning in a later autofocus stage 3007 can be helpful. With a small defocus value, it can be assumed that the relative error is already quite small.

[0059] Another exemplary decision criterion for the decision in Box 3040 concerns the disparity calculation method used in the preceding autofocus stage 3007. For example, if a less precise disparity calculation method is used in autofocus stage 3007 of the current iteration 3041 to calculate the disparity value (Box 3030), it could be decided that the subsequent autofocus stage 3007 is necessary and that a particularly precise disparity calculation method should be used there for fine-tuning. For example, if the defocus value in Box 3030 of the current iteration 3041 was calculated based on a disparity value determined using the area-based disparity calculation method, it could be decided that a further autofocus stage 3007 is necessary.

[0060] Another exemplary decision criterion for the decision in Box 3040 concerns the difference between different candidate disparity values, already discussed above in connection with Boxes 3020 and 3025. For example, if a particularly small difference is determined, a subsequent autofocus stage 3007 may be unnecessary. With a relatively large difference, a subsequent autofocus stage may be desirable. More generally, it would therefore be conceivable to consider a deviation between different results of several disparity calculation methods (see also...). Fig. 5, where a corresponding technique is described).

[0061] Another exemplary decision criterion for the decision in Box 3040 concerns the running index i. For example, in some scenarios it would be conceivable that a certain number of autofocus steps 3007 are always performed. For instance, exactly two autofocus steps could be performed. With exactly two autofocus steps, a coarse adjustment can first be made; this can already achieve a well-defined defocus value, enabling subsequent fine adjustment; the error remaining after this fine adjustment is then below a predetermined tolerance threshold, so that further autofocus steps are unnecessary. By using only two autofocus steps, focusing can be achieved very quickly, especially within a guaranteed execution time. "Endless" or lengthy iterative focusing is avoided.

[0062] Therefore, if i ≥ 2, an abort can occur in box 3040 and wait for a new autofocus trigger (box 3005).

[0063] Fig.Figure 2 illustrates aspects of an area-based disparity calculation method. A "left" image 170 (from the left stereoscopic optical channel) and a "right" image 180 (from the right stereoscopic optical channel) are shown. In the left image 170, a reference subregion 171 is marked, centered on an area of ​​interest 190. In the right image 180, several candidate subregions 181-186 are shown, all shifted relative to each other and approximately aligned with the area of ​​interest 190. Comparisons can then be made between a measure determined for the reference subregion 171 and measures determined for each of the candidate subregions 181-186 to determine an area similarity measure for each. Such measures can be calculated, for example, based on pixel values ​​of images 170 and 180 (possibly after preprocessing).In this calculation of the area similarity measure, spatial relationships between the pixel values ​​in the various sub-areas 171, 181-186 can be disregarded. For example, pixel value histograms could be determined and compared. An average brightness could be determined. These are just a few examples. The candidate sub-area 181-186 that exhibits the greatest area similarity to the reference sub-area 171 can serve as the basis for determining the disparity value. For example, if the candidate sub-area 186 has a measure that is closest to the measure of the reference sub-area 171, the disparity value can be determined as the distance between the center of the reference sub-area 171 and the center of the candidate sub-area 186.

[0064] Fig.Figure 3 illustrates aspects relating to an object-based disparity calculation method. A specific object 271 is located at or near the area of ​​interest 190 in a right-hand image 270 of a stereoscopic image pair. Then, the same object 271 is located in the left-hand image 280 using template matching. The resulting displacement 275 can be used as the disparity value.

[0065] Fig. Figure 4 illustrates a schematic autofocus workflow according to various examples. For instance, the autofocus workflow could consist of Fig. 4 the procedure from Fig. 1. Implement. First, two images 470 and 480 are obtained in box 3100, compare. Fig.1: Box 3015. Then, in Box 3105, a so-called "capture algorithm" is executed; that is, a disparity calculation method is used which is particularly robust. In particular, an area-based disparity calculation method is used to calculate a disparity value. See Fig. 1: Boxes 3020, 3025; and Fig. 2. The disparity value is then converted into a defocus value, and, for example, a zoom lens is controlled accordingly, Box 3110 (compare Fig.1 Box 3030; Box 3035). This completes a first autofocus stage (i=1); a second autofocus stage (i=2) is then executed for fine-tuning. In Box 3115, two images of an image pair are again obtained, and then an object-based disparity calculation method is applied in Box 3120 to calculate a new disparity value. Based on this, the focus is then set in Box 3125. Instead of using an object-based disparity calculation method in Box 3120 (here, in particular, template matching as an exemplary technique for locating identical objects in two images), it would also be conceivable to use an area-based disparity calculation method.

[0066] Fig. Figure 5 illustrates a schematic autofocus workflow according to various examples. For instance, the autofocus workflow could consist of Fig. 5 the procedure from Fig. 1. Implement. Fig.Figure 5 illustrates one variant of the autofocus workflow from Fig. 4.

[0067] In Fig. In the first autofocus stage (i=1), based on 3200 images obtained (470, 480), two different disparity calculation methods are applied (3205). Specifically, an area-based disparity calculation method and an object-based disparity calculation method are used. Thus, two candidate disparity values ​​are obtained (compare Fig.1: Box 3020). Then, the difference between these candidate disparity values ​​is calculated and compared to a threshold value (see 3210). Depending on the result of the threshold comparison, either Box 3250 or Box 3215 is executed. If the difference is less than the specified threshold, Box 3250 is used. There, the candidate disparity value from the object-based disparity calculation method is used as the final disparity value. Based on this final disparity value, the defocus value is then calculated. Focusing then occurs in Box 3250, and the autofocus workflow ends (i.e., no further autofocus step is performed; see [reference]). Fig.1: Box 3040; corresponds to a "one-shot autofocus"). If, on the other hand, the difference between the candidate disparity values ​​is greater than the specified threshold, the candidate disparity value from the area-based disparity calculation method is used as the final disparity value, and the defocus value in Box 3215 is calculated based on this final disparity value. Then, another autofocus stage (i=2) is performed, in which an object-based disparity calculation method in Box 3220 is used on another image pair to determine a corresponding disparity value. Based on this new disparity value, the defocus value is then determined in Box 3225, and the focal plane of the microscope is shifted accordingly.

[0068] The variant of the autofocus workflow according to Fig.5 is based on the finding that the area-based disparity calculation method yields a less accurate disparity value than the object-based disparity calculation method. However, in the range where the object-based disparity calculation method also provides a reliable disparity value, the disparity values ​​of both methods should be similar. If they are similar (difference less than the threshold), a high level of confidence in the object-based disparity calculation method can be inferred; a further autofocus stage is not necessary (Box 3250).If the disparity values ​​of the two disparity calculation methods differ significantly, a very blurry image (large defocus value) can be assumed; so that a coarse adjustment based on the area-based disparity calculation method (i=1) can be followed by a fine adjustment based on the object-based disparity calculation method (i=2).

[0069] Fig. Figure 6 schematically illustrates a system 60, which comprises a medical visualization system 69 with a stereoscopic microscope 61. The medical visualization system 69, in this example, features... Fig.6 also includes a robotic stand, which is optional. The microscope 61 can, for example, have a varifocal lens. The medical visualization system 69 also includes one or more cameras 63, by means of which microscope images can be captured for the two optical channels of the microscope 61. In addition, the system 60 includes a controller 64. The controller 64 can, for example, include a processor and memory. The processor can load and execute program code from memory. When the processor executes the program code, this causes autofocus techniques, as described therein, to be carried out. For example, the processor could then perform the procedure from Fig. 1 or one of the autofocus workflows from Fig. 4 of the Fig. 5. The controller 64 can, for example, control the microscope 61 and / or the stand 62. The controller 64 can, for example, receive images from the cameras 63.

[0070] In summary, the preceding section described autofocus techniques that can perform one or more autofocus stages. In one method, defocus values ​​are determined based on both an area comparison measure and object matching. Thus, two different disparity calculation methods are used. Focusing on the area of ​​interest occurs based on the defocus value determined using the object-based disparity calculation method if the two defocus values ​​differ only slightly. In this case, the autofocus workflow is complete, and the system can wait for another autofocus trigger. Otherwise, focusing on the area of ​​interest occurs based on the defocus value determined using the area-based disparity calculation method.A further autofocus stage is then performed for fine-tuning; for this, the defocus value is determined using the object-based disparity calculation method, and refocusing is then carried out based on this defocus value.

[0071] Naturally, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or individually, without leaving the scope of the invention.

Claims

[1] Control (64) for a medical visualization system (69) with a stereoscopic microscope (61), wherein the stereoscopic microscope (61) has a first optical channel and a second optical channel, wherein the control (64) is set up to perform the following steps in each of one or more autofocus levels (3007): - based on a respective image pair of the first optical channel and the second optical channel, determining (3020, 3025) a respective disparity value based on one or more disparity calculation methods, - based on the respective disparity value, determining (3030) a respective defocus value using a camera model of the first optical channel and the second optical channel, - Controlling (3035) one or more components of the medical visualization system (69) based on the respective defocus value, wherein the control (64) is set up to use a first disparity calculation procedure in at least one of the one or more autofocus stages (3007) to determine a first candidate disparity value (3020), wherein the control (64) is further configured to use a second disparity calculation method in at least one of the one or more autofocus stages (3007) to determine a second candidate disparity value (3020), wherein the second disparity calculation method is less robust to blur in the respective image pair than the first disparity calculation method, wherein the control (64) is still set up to determine the disparity value based on the first and second candidate disparity values ​​(3025), wherein the control (64) is set up to determine the disparity value based on a comparison of the first and second candidate disparity values, wherein the control (64) is set up to determine a difference between the first and second candidate disparity values ​​and to perform a threshold comparison of this difference with a predetermined threshold, wherein the control (64) is still set up to reject one or more of the candidate disparity values ​​depending on a result of the threshold comparison, wherein the control (64) is set up to reject, if the difference is greater than the specified threshold, such candidate disparity values ​​obtained by the second disparity calculation procedure. [2] Control (64) according to claim 1, wherein the control (64) is further configured to selectively trigger a subsequent autofocus stage (3007) of one or more autofocus stages (3007) depending on one or more decision criteria (3040). [3] Control (64) according to claim 2, wherein the one or more decision criteria comprise a magnitude of the defocus value in the preceding autofocus stage. [4] Control (64) according to claim 2 or 3, wherein the one or more decision criteria comprise a disparity calculation method used in the preceding autofocus stage (3007). [5] Control (64) according to one of claims 2 to 4, wherein the one or more decision criteria comprise a deviation between the results of several disparity calculation methods determined in the respective preceding autofocus stage (3007). [6] Control (64) according to one of the preceding claims, wherein in at least one of the one or more autofocus stages (3007) at least one of the one or more disparity calculation methods is an object-based disparity calculation method. [7] Control (64) according to any of the preceding claims, where the first disparity calculation method is an area-based disparity calculation method, the area-based disparity calculation method includes: - Selecting a reference sub-area (171) in a first image (170) of the respective image pair, - Calculating a respective area similarity measure between the reference sub-area (171) and each of several second candidate sub-areas (181, 182, 183, 184, 185, 186) in a second image (180) of the respective image pair, and - Determining the disparity value based on an extremum of the area similarity measures. [8] Control (64) according to one of the preceding claims, wherein the control (64) is configured to determine one or more settings of a subsequent autofocus stage (3007) of the one or more autofocus stages (3007) depending on one or more settings of a preceding autofocus stage (3007) and / or a result of the preceding autofocus stage (3007). [9] Control (64) according to one of the preceding claims, wherein the one or more autofocus levels (3007) comprise exactly two autofocus levels (3007). [10] Method for controlling a medical visualization system with a stereoscopic microscope (61), wherein the stereoscopic microscope (61) has a first optical channel and a second optical channel, the procedure includes: - Performing multiple autofocus stages, the procedure includes in each of the one or more autofocus stages: - based on a respective image pair of the first optical channel and the second optical channel, determining (3020, 3025) a respective disparity value based on one or more disparity calculation methods, - based on the respective disparity value, determining (3030) a respective defocus value using a camera model of the first optical channel and the second optical channel, and - Controlling (3035) one or more components of the medical visualization system (69) based on the respective defocus value, wherein in at least one of the one or more autofocus stages (3007) a first disparity calculation method is used to determine a first candidate disparity value (3020), wherein in at least one of the one or more autofocus stages (3007) a second disparity calculation method is used to determine a second candidate disparity value (3020), wherein the second disparity calculation method is less robust to blurring in the respective image pair than the first disparity calculation method, where the disparity value is determined based on the first and second candidate disparity values ​​(3025), where the disparity value is determined based on a comparison of the first and second candidate disparity values, where a difference between the first and second candidate disparity values ​​is determined and a threshold comparison of this difference with a predetermined threshold is performed, where one or more of the candidate disparity values ​​are discarded depending on a result of the threshold comparison, where, if the difference is greater than the specified threshold, such candidate disparity values ​​obtained by the second disparity calculation method are discarded.

Images