Method for calibrating stereoscopic medical microscope and medical microscope assembly
By combining the imaging of three-dimensional and two-dimensional calibration objects, the calibration data is generated and applied, and the problem of insufficient image error and information transmission in stereomedical microscope calibration is solved, thereby improving the imaging quality during surgery and surgeon work efficiency.
Patent Information
- Application Number
- CN202380018157.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-25
- Filing Date
- 2023-01-24
- Publication Date
- 2024-11-19
AI Technical Summary
The existing stereomedical microscope calibration methods are difficult to achieve the optimal three-dimensional image and information superposition, resulting in insufficient image error and information transmission during surgery, increasing the work burden and fatigue of surgeons.
By combining the stereo camera system to acquire imaging of three-dimensional and two-dimensional calibration objects, and generate and store calibration data, it is used to correct the camera's distortion error and the illumination geometry of the light source to achieve overall calibration.
It improves imaging quality during surgery, reduces image errors and interference in information transmission, and improves surgeons' work efficiency and safety during surgery.
Smart Images

Figure CN118984924A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for calibrating a stereoscopic medical microscope and a medical microscope assembly. Background Art
[0002] The goal of digital visualization in surgery and microsurgery is to show the surgeon an optimal three-dimensional image of the surgical area on the monitor of the medical microscope during surgery and, if necessary, an optimal three-dimensional overlay of additional generated information onto the field of view. To achieve this, the acquired camera images are processed by digital image processing using defined generated calibration data.
[0003] Methods for combining extrinsic and intrinsic calibration cameras to generate distortion errors are known, for example, from Reg G. Willson, Modeling and Calibration of Automated Zoom Lenses, PhD-Dissertation, The Robotics Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, 1994. Furthermore, a method for calibrating multiple working point settings (zoom and focus) with the goal of "stereo-augmented reality" is known from AP King et al., StereoAugmented Reality in the Surgical Microscope, Presence, Vol. 9, No. 4, August 2000, 360-368, Massachusetts Institute of Technology.
[0004] A three-dimensional calibration object is known from DE 10 2019 131 646 A1. The three-dimensional calibration object has, for example, a transparent body and calibration markings embedded in the volume of the transparent body. Summary of the invention
[0005] The invention is based on the object of improving a method for calibrating a stereo medical microscope and a medical microscope assembly.
[0006] According to the invention, this object is achieved by a method having the features described below and a medical microscope assembly having the features described below. Advantageous embodiments of the invention are described below.
[0007] The present invention provides a method for calibrating a stereoscopic medical microscope, comprising:
[0008] a) acquiring an image of at least one three-dimensional calibration object by means of a camera of a stereo camera system of the medical microscope;
[0009] b) generating calibration data based on the acquired imaging, wherein the generated calibration data is stored for use in correction;
[0010] c) acquiring a further imaging of at least one two-dimensional calibration object by means of a camera of the stereo camera system;
[0011] d) Generating further calibration data based on the acquired further imaging, wherein the further calibration data is stored for use in corrections.
[0012] In one embodiment, step a) and / or step c) are performed at different operating points of the medical microscope, wherein the calibration data are generated in step b) and / or the further calibration data are generated in step d) for each of the different operating points.
[0013] In one embodiment, the calibration data generated in step b) and / or the further calibration data generated in step d) are at least partially applied to the acquired imaging and / or the acquired further imaging for correction, wherein step b) is performed and / or repeated for the corrected imaging and / or step d) is performed and / or repeated for the corrected further imaging.
[0014] In one embodiment, the calibration and the steps c) and d) are repeated until at least one predetermined optimization criterion is met.
[0015] In one embodiment, the generated calibration data are checked by at least partially applying to the acquired imaging and / or the acquired further imaging and evaluating the result of the application, wherein if the evaluation result does not meet at least one predetermined criterion, steps a) to d) are at least partially repeated.
[0016] In one embodiment, step b) and / or step d) comprises one or more of the following measures: determining external calibration data, determining internal calibration data, determining distortion correction field data, determining edge attenuation correction data, determining color shift field correction data.
[0017] In one embodiment, step b) comprises determining respective focus positions of cameras of the stereo camera system and setting respective focus positions.
[0018] In one embodiment, step d) comprises determining an offset between the acquired further images of the cameras of the stereo camera system and determining an offset correction data and / or determining a rotation between the acquired further images of the cameras of the stereo camera system and determining a rotation correction data.
[0019] In one embodiment, within the scope of step a) or step c), different operating points of the movable lens system of the stereo medical microscope are selected when acquiring at least a portion of the imaging, wherein within the scope of step b) or step d), offset values are determined for individual operating points of the operating points of the movable lens system, wherein the calibration data and / or the further calibration data are generated taking into account the correspondingly determined offset values.
[0020] In one embodiment, at least one additional image of a uniformly reflecting and / or transmitting calibration object is also acquired with the aid of the stereo camera system, wherein illumination correction data are generated based on the at least one acquired additional image to correct the illumination geometry of at least one light source of the medical microscope, and the illumination correction data are taken into account during the generation of the calibration data and / or the further calibration data.
[0021] The present invention also provides a medical microscope assembly, comprising:
[0022] A stereoscopic medical microscope and a data processing device, wherein the stereoscopic medical microscope has a stereoscopic camera system including a camera and storable calibration data,
[0023] The data processing device is configured to:
[0024] i) generating calibration data based on images of at least one three-dimensional calibration object acquired by means of cameras of the stereo camera system, and storing the calibration data for correction; and
[0025] ii) generating further calibration data based on further images of at least one two-dimensional calibration object acquired by means of the cameras of the stereo camera system, and storing the further calibration data for correction purposes.
[0026] The basic idea of the present invention is to combine a three-dimensional calibration with a two-dimensional calibration, so that an improved, in particular optimal calibration result is achieved overall. This is done by acquiring an image of at least one three-dimensional calibration object by means of a camera of a stereo camera system of a medical microscope. In particular, each camera of the camera of the stereo camera system acquires at least one image. The image is acquired in particular in such a way that the images of the two cameras of the stereo camera system are linked to each other for the same working point of the medical microscope, that is, they are in particular associated with each other or can be associated with each other. Calibration data are generated based on the acquired images, and these calibration data are stored for correction. In addition, other images of at least one two-dimensional calibration object are acquired by means of the camera of the stereo camera system. These other images are also acquired in such a way that the other images of the two cameras of the stereo camera system are linked to each other for the same working point, that is, they are in particular associated with each other or can be associated with each other. In this case, the image and the other image are acquired in particular at the same working point. Based on the acquired other images, other calibration data are generated, and these other calibration data are stored for correction. Storing the calibration data here particularly includes storing the calibration data in a memory provided for this purpose so that they can be retrieved and applied from the memory when necessary. The stored calibration data and the stored further calibration data can be applied to subsequent images (and imaging sequences, in particular videos) acquired, in particular during surgery, so that these images can be corrected by the calibration data and the further calibration data and can be subsequently acquired by the surgeon and / or other personnel.
[0027] In particular, a method for calibrating a stereoscopic medical microscope is provided, the method comprising:
[0028] a) acquiring an image of at least one three-dimensional calibration object by means of a camera of a stereo camera system of a medical microscope;
[0029] b) generating calibration data based on the acquired imaging, wherein the generated calibration data is stored for use in correction;
[0030] c) acquiring a further imaging of the at least one two-dimensional calibration object by means of a camera of the stereo camera system;
[0031] d) Generating further calibration data based on the acquired further imaging, wherein the further calibration data is stored for use in correction.
[0032] In addition, a medical microscope assembly is particularly provided, which includes: a stereoscopic medical microscope and a data processing device, wherein the stereoscopic medical microscope has a stereoscopic camera system including a camera and storable calibration data, wherein the data processing device is configured to: generate calibration data based on images of at least one three-dimensional calibration object acquired by means of the camera of the stereoscopic camera system, and store these calibration data for correction; and generate additional calibration data based on additional images of at least one two-dimensional calibration object acquired by means of the camera of the stereoscopic camera system, and store these additional calibration data for correction.
[0033] The method and the medical microscope assembly have the advantage that, by combining three-dimensional and two-dimensional calibration with associated two-dimensional and three-dimensional calibration objects, an overall calibration of a stereoscopic medical microscope can be achieved. After the overall calibration data have been generated, corrected images can be provided to the surgeon and / or other personnel with the aid of these calibration data. The stereoscopic superposition of the respective corrected images is improved compared to uncorrected images, so that the information transfer during surgery is also improved thereby, and interferences caused by insufficient superposition and / or image errors and the resulting impairment and fatigue during the work process during surgery can be reduced.
[0034] The generation of calibration data and other calibration data is realized in particular by evaluating the acquired imaging and the acquired other imaging. Here, in particular, the features on the corresponding calibration object are identified and evaluated. Since the calibration object has known characteristics, the calibration data and other calibration data can be generated and determined in particular based on the acquired imaging and other imaging after the known characteristics and evaluation. In particular, the corresponding features and their characteristics, in particular the orientation (position and orientation), geometric arrangement, shape, color, brightness, etc. are known, and can therefore be identified in the imaging and other imaging. By comparing the known theoretical characteristics of the corresponding features in the imaging and other imaging with the actual characteristics, the calibration data and other calibration data can be determined in a known manner with the help of a known calibration method. The evaluation is carried out in particular with the help of a data processing device. In particular, computer vision, pattern recognition and / or machine learning methods known per se can be used.
[0035] The medical microscope is in particular a surgical microscope. However, the medical microscope can also be a microscope used for medical research and / or diagnostic purposes (for example in the field of ophthalmology or other fields). The medical microscope assembly is in particular a surgical microscope assembly.
[0036] The calibration data and the further calibration data can in principle be of the same or different types and relate to different properties and / or effects and / or disturbances and / or devices of the medical microscope. The distinction between the calibration data and the further calibration data has been selected in particular for the purpose of language differentiation.
[0037] The two-dimensional calibration object can have a shape and pattern known per se, such as a two-dimensional chessboard pattern, line markings, crosshairs, etc. The three-dimensional calibration object has a three-dimensional structure in particular. In particular, the three-dimensional calibration object can have structures that are arranged in multiple planes perpendicular to the optical axis of the imaging system of the medical microscope and can be acquired, for example, using the illumination and / or defined intrinsic illumination of the medical microscope or with the aid of a camera. This can be, for example, a mark with high contrast that is arranged in two planes perpendicular to each other in order to provide depth information relative to the optical axis (see, for example, FIG. 2 of King et al.). For example, a three-dimensional calibration object is also described in DE 10 2019 131 646 A1. Therefore, the three-dimensional calibration object has, for example, a transparent body and a calibration mark embedded in the volume of the transparent body. The three-dimensional calibration object in one of the embodiments described in DE 10 2019131 646 A1 is generally a 3D calibration body in the shape of a cube constructed from transparent layers. The 3D calibration body is constructed from stacked light guides. The light guide is composed of alternating transparent layers, wherein the layers have a refractive index greater than the layers arranged between them. With the aid of a selectively switchable light source, light is coupled into the layers in such a way that the light is totally reflected at the interface between the layers. Since total reflection occurs at the interface between the layers, light propagation (i.e. propagation of electromagnetic waves of light) only occurs within the corresponding light guide. In order to be able to present calibration marks in the light guide of the 3D calibration body, thin films are applied to the layers at specific intervals. The refractive index of these films is selected here so that total reflection is suppressed at these positions, so that the light propagates into the layers with a lower refractive index. Now, when light is coupled into one of the light guides, the light is coupled out of the light guide at the position where the film is applied thereon, thereby forming a luminous position serving as a calibration mark in the volume of the transparent body constructed from the layers. Depending on the light source that is switched on, luminous positions are formed at different depths of the transparent body.
[0038] It can also be provided that at least one two-dimensional calibration object is combined with at least one three-dimensional calibration object to form a combined calibration object. The method sequence is then essentially the same.
[0039] The components of the medical microscope assembly, in particular the data processing device, can be designed individually or in an integrated manner as a combination of hardware and software, for example as a program code implemented on a microcontroller or microprocessor. The data processing device can, for example, include a computing device in the form of a microprocessor or microcontroller, and a memory. However, it can also be proposed to design the components individually or in an integrated manner as an application-specific integrated circuit (ASIC) and / or a field programmable gate array (FPGA). The data processing device can also be part of the medical microscope. In principle, however, the data processing device can also be formed separately from the medical microscope, for example, as a desktop computer, a laptop or a tablet computer or as a cloud-based solution.
[0040] In one embodiment, it is proposed that step a) and / or step c) are performed at different working points of the medical microscope, wherein calibration data are generated in step b) and / or further calibration data are generated in step d) for each of the different working points. As a result, calibration data can be generated over a larger, in particular, entire working range and / or operating range of the medical microscope. The working point here particularly includes one or more of the following parameters: the position (position and / or orientation) of (multiple) cameras and / or the magnification (zoom) and / or the working distance (position of the focus or focusing lens) and / or the position of the aperture and / or the value of the aperture opening (aperture aperture) and / or the presence of a cover lens (Drapelinse) on the microscope (yes / no). It can be proposed here that for working points where imaging and / or further imaging has not yet been acquired, the calibration data and further calibration data are generated based on the calibration data and / or further calibration data of adjacent working points, in particular by interpolation and / or extrapolation. It can also be provided that the calibration data for such an operating point are estimated by means of a function adapted to the calibration data generated on the basis of the measured operating point.
[0041] In one embodiment, it is proposed that the calibration data generated in step b) and / or the further calibration data generated in step d) are at least partially applied to the acquired imaging and / or the acquired further imaging for correction, wherein step b) is performed and / or repeated for the corrected imaging, and / or step d) is performed and / or repeated for the corrected further imaging). As a result, the calibration data and / or the further calibration data can be gradually optimized, because the calibration data and / or the further calibration data that have been generated are applied to the acquired imaging and / or the acquired further imaging. As a result, in particular, a variety of different effects that may cause interference between the imaging and / or the further imaging and may influence each other can be taken into account. Thus, a distortion error in the imaging acquired by means of the left camera of the stereo camera system may, for example, affect the offset between the imaging acquired by the left and right cameras of the stereo camera system, and vice versa. If the distortion error is corrected before determining the offset, the effect of the distortion error can be minimized or even eliminated when determining the offset.
[0042] In particular, it is proposed that the calibration data generated in step b) are first at least partially applied to the acquired further images for correction before step d) is carried out. Thus, step d) can be carried out with the acquired further images that have already been corrected by the calibration data. In this way, interfering effects that influence each other can be reduced.
[0043] In an improved embodiment, it is proposed that the calibration and step c) and step d) are repeated until at least one predetermined optimization criterion is met. In this way, the further calibration data can be improved step by step. The at least one predetermined optimization criterion is, for example, a predetermined maximum value of the (x / y) offset between the further images acquired by the two cameras of the stereo camera system (e.g. + / -1 pixel) and / or a predetermined maximum value of the rotation between the further images acquired by the two cameras of the stereo camera system (e.g. + / -0.001° relative to the image center point).
[0044] If the method steps are carried out in the order c), d), a), b), the image acquired in step a) can also be corrected with the aid of the further calibration data generated in step d) and then step b) can be carried out. In this case, the process is basically similar to the one described above.
[0045] In one embodiment, it is proposed that the generated calibration data is checked by at least partially applying it to the acquired imaging and / or the acquired further imaging and evaluating the result of the application, wherein if the evaluation result does not meet at least one predetermined criterion, steps a) to d) are at least partially repeated. The evaluation result may, for example, include the value of the deviation determined for the selected parameter. The selected parameter may, for example, include one or more of the following parameters: offset, rotation, brightness difference, distortion, etc. The predetermined criterion may, for example, include a predetermined limit value of the corresponding evaluation result, that is, in particular, the maximum permissible value of the corresponding deviation. It may also be proposed that the individual evaluation results of the selected parameters are aggregated into a single evaluation result and compared with the at least one criterion (for example, with the maximum value of the aggregated deviation).
[0046] In one embodiment, step b) and / or step d) includes one or more of the following measures: determining external calibration data, determining internal calibration data, determining distortion correction field data, determining edge attenuation correction data, and determining color shift field correction data.
[0047] For the extrinsic calibration data and / or the intrinsic calibration data, the camera projection matrix is determined, in particular by means of methods known per se. Methods for extrinsic and intrinsic calibration of cameras are known, for example, from the document mentioned at the outset (Reg G. Willson, 1994).
[0048] In order to determine the distortion correction field data, the distortion of the acquired image is determined in particular with the aid of a suitable calibration object, in the simplest case, for example with the aid of a calibration object having a plurality of straight lines perpendicular to one another in a plane perpendicular to the optical axis of the medical microscope, or with the aid of a checkerboard pattern, and the distortion field is determined with the aid of the determined distortion, from which the distortion correction field data are subsequently derived. In order to improve the determination of the distortion correction field data, in particular the three-dimensional structure of the three-dimensional calibration object is acquired and evaluated. This is described, for example, by King et al.
[0049] In order to determine the edge attenuation correction data, in particular (camera) vignetting in the acquired image is determined, which is corrected by means of the edge attenuation correction data in such a way that the brightness values in the acquired image do not decay in the direction of the acquired image edge. For this purpose, in particular images of a homogeneously self-luminous three-dimensional calibration object are acquired for different working points (working distance, zoom, etc.). Within the scope of the evaluation carried out in step b), the brightness curve in the acquired image is determined and the edge attenuation correction data are determined based on the determined brightness curve in such a way that the acquired image of the homogeneously self-luminous three-dimensional calibration object, which is corrected by means of the edge attenuation correction data, has a constant, i.e., homogeneous brightness over the entire calibration object.
[0050] With the aid of color shift field correction data, in particular lateral chromatic aberrations should be corrected, i.e. in particular color gamuts that are visible at the edges should be corrected. In order to determine the color shift field correction data, in particular the acquired imaging of the three-dimensional calibration object is evaluated in relation to the color channels. In particular, an offset (e.g. an x / y offset) is determined in relation to the color channel and an offset field of the acquired imaging is determined for each color channel. For example, due to chromatic aberrations, the edges with the color gamuts in each color channel are located at slightly different positions in the acquired imaging. The correction field generated accordingly from the offset field then indicates how the acquired imaging must be shifted in the corresponding color channel so that the imaging of all color channels is superimposed on one another in a pixel-accurate manner, so that the color gamuts disappear as a result. The correction fields of all color channels then result in color shift field correction data.
[0051] In one embodiment, it is proposed that: step b) includes determining the corresponding focal position of the cameras of the stereo camera system and setting the corresponding focal position. The setting of the focal position is especially based on the focal position determined for the cameras of the stereo camera system based on the imaging. The purpose of the determination and setting is that the camera always has the same (actual) focal position at each working point. For example, the focal position is set by a technician, and the value of the focal position determined by means of the acquired imaging is displayed to the technician on the display device of the medical microscope. The technician then adjusts the focal position in such a way that the (actual) focal position is always the same for the two cameras of the stereo camera system at different working points. However, in principle, automatic adjustment can also be performed, for example, by means of an actuator configured for this purpose of the medical microscope.
[0052] In one embodiment, it is proposed that step d) comprises: determining an offset between the acquired further images of the cameras of the stereo camera system and determining offset correction data, and / or determining a rotation between the acquired further images of the cameras of the stereo camera system and determining rotation correction data. In particular, the offset (e.g. x / y offset) and / or the rotation between the acquired further images of the cameras is determined with reference to the respective image centers of the acquired further images.
[0053] In one embodiment, it is proposed that: within the scope of step a) or step c), different working points of the movable lens system of the stereo medical microscope are selected when acquiring at least a portion of the imaging, wherein within the scope of step b) or step d), offset values are determined for each of the working points of the movable lens system, wherein calibration data and / or further calibration data are generated taking into account the correspondingly determined offset values. Thus, the offset caused by the movable lens system can also be corrected at different working points of the movable lens system. The offset value refers in particular to the (x / y) offset value of the pixel in the imaging acquired by the two cameras. By selecting different working points, the position of the lens in the lens system changes in particular. If the optical axis of the lens system changes here (for example, due to tolerances and inaccuracies in the mechanical device and the actuator used), imaging errors may occur, which may cause an offset between the acquired imaging and / or the acquired further imaging of the two cameras of the stereo camera system. Then, such an offset can be corrected at each working point with the aid of the generated calibration data and / or further calibration data. For operating points for which no offset value has been determined, calibration data and / or further calibration data can be generated by interpolation or extrapolation.
[0054] In one embodiment, it is proposed that at least one additional image of a uniformly reflecting and / or transmitting calibration object is also acquired with the aid of a stereo camera system, wherein illumination correction data are generated based on the at least one additional image acquired to correct the illumination geometry of at least one light source of the medical microscope, and the illumination correction data are taken into account during the generation of the calibration data and / or further calibration data. Thus, the inhomogeneous illumination geometry of the at least one light source can also be corrected. The illumination correction data can be included in the calibration data or the further calibration data. The uniformly reflecting and / or transmitting calibration object can also be part of the at least one three-dimensional calibration object or the at least one two-dimensional calibration object.
[0055] Further features of the configuration of the medical microscope assembly are apparent from the description of the configuration of the method. The advantages of the medical microscope assembly are correspondingly identical to the configuration of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The present invention will be described in detail below with the aid of preferred embodiments and with reference to the accompanying drawings. In the drawings:
[0057] Figure 1 A schematic diagram illustrating an embodiment of a medical microscope assembly;
[0058] Figure 2a a schematic diagram showing a three-dimensional calibration object and a two-dimensional calibration object in side view;
[0059] Figure 2b A schematic diagram showing a three-dimensional calibration object and a two-dimensional calibration object in a top view;
[0060] Figure 3 A schematic flow chart showing an embodiment of a method for calibrating a stereo medical microscope. DETAILED DESCRIPTION
[0061] exist Figure 1 Schematic diagram of an embodiment of a medical microscope assembly 100 is shown in FIG. The medical microscope assembly 100 comprises a stereoscopic medical microscope 1 (eg, a surgical microscope) and a data processing device 2. The method described in the present disclosure will be explained below with the aid of the medical microscope assembly 100.
[0062] The stereo medical microscope 1 comprises a stereo camera system 3, which comprises a left camera 31 and a right camera 3r. The cameras 31, 3r acquire an acquisition region which is imaged by means of a stereo imaging optical system 4 (shown only schematically) of the medical microscope 1. In order to correct errors which occur, in particular during adjustment, calibration data 30 and further calibration data 31 can be stored for the imaging optical system 4, for example for an actuator 5 which is configured to change the properties of the imaging optical system 4.
[0063] Furthermore, the medical microscope 1 comprises a signal processing device 6, which processes the raw signal 101 provided by the image sensor of the left camera 31 and the raw signal 10r provided by the image sensor of the right camera 10r and generates and provides an image 20 and a further image 21 based on the raw signals 101, 10r. The images 20, 21 can be displayed, for example, on at least one display device 7, for example one or more monitors or head mounted display devices (English: Head Mounted Display, HMD), which can in particular also be part of the medical microscope 1. The signal processing can in particular include pixel-dependent changes in brightness and / or the generation of offsets and / or rotations around the image center point (or other points) and further manipulations (filtering, color correction, etc.). In order to correct the raw signals 101, 10r, calibration data 30, 31 can be stored in the signal processing device 6.
[0064] It can also be provided that the calibration data 30 , 31 are stored exclusively in the image processing device 6 .
[0065] The data processing device 2 comprises a computing device 2 - 1 (eg a microprocessor or a microcontroller) and a memory 2 - 2 . The data processing device 2 may also be a part of the medical microscope 1 .
[0066] The data processing device 2 is configured to generate calibration data 30 based on images 20 of at least one three-dimensional calibration object 40 acquired by means of the cameras 31, 3r of the stereo camera system 3, and to store these calibration data for correction. In addition, the data processing device 2 is configured to generate further calibration data 31 based on further images 21 of at least one two-dimensional calibration object 41 acquired by means of the cameras 31, 3r of the stereo camera system 3, and to store these further calibration data for correction.
[0067] In particular, a method for calibrating a stereo medical microscope 1 is carried out with the aid of the medical microscope assembly 100, the method comprising:
[0068] a) acquiring an image 20 of the at least one three-dimensional calibration object 40 by means of the cameras 31 , 3r of the stereo camera system 3 of the medical microscope 1 ;
[0069] b) generating calibration data 30 based on the acquired imaging 20, wherein the generated calibration data 30 is stored for correction;
[0070] c) acquiring a further image 21 of the at least one two-dimensional calibration object 41 by means of the cameras 31 , 3r of the stereo camera system 3 ;
[0071] d) Generating further calibration data 31 based on the acquired further images 21 , wherein the further calibration data 31 are stored for correction purposes.
[0072] It can be provided that step a) and / or step c) are performed at different operating points of the medical microscope 1, wherein for each of the different operating points, calibration data 30 are generated in step b) and / or further calibration data 31 are generated in step d). In this case, the operating point can include, in particular, parameters for the following settings: the position (position and / or orientation) and / or magnification (zoom) of the camera(s), the working distance (position of the focus or focusing lens), the position of the diaphragm, the value of the diaphragm opening (aperture diaphragm), the presence (or absence) of a cover lens, etc.
[0073] Different operating points can be set manually and automatically. In the manual setting process, it can be provided that, for example, the parameters of the medical microscope 1 are displayed to the technician so that the technician can set the parameters for the respective operating point. In the automatic setting process, the data processing device 2 can be configured to generate suitable control parameters and supply them to the medical microscope 1, in particular the actuator 5, in order to set different operating points.
[0074] It can be provided that the calibration data 30 generated in step b) and / or the further calibration data 31 generated in step d) are at least partially applied to the acquired image 20 and / or the acquired further image 21 for correction, wherein step b) is performed and / or repeated for the corrected image 20 and / or step d) is performed and / or repeated for the corrected further image 21). In this case, the correction can be carried out both in the signal processing device 6 and in the data processing device 2.
[0075] It can be provided that the calibration and step c) and step d) are repeated until at least one predetermined optimization criterion is met. For this purpose, in particular, before each repetition, further calibration data 31 are stored in the medical microscope 1, in particular in the image processing device 6. Step c) and step d) are then performed again with the changed further calibration data 31. As a result, the calibration of the medical microscope 1 can be improved step by step and iteratively.
[0076] In particular, it is proposed that step b) includes one or more of the following measures: determining external calibration data, determining internal calibration data, determining distortion correction field data, determining edge attenuation correction data, and determining color shift field correction data. Thus, the generated calibration data 30 includes external calibration data and / or internal calibration data and / or distortion correction field data and / or edge attenuation correction data and / or color shift field correction data.
[0077] It can be stated that step b) includes determining the respective focus positions of the cameras 31, 3r of the stereo camera system 3 and setting the respective focus positions. The (actual) focus position can be set or adjusted based on the values determined for the focus positions. In principle, this can be done manually by a technician or automatically and / or motorized by means of an actuator 5. The purpose is to always have the same focus position of the two cameras 31, 3r at different operating points.
[0078] It can be proposed that step d) comprises determining an offset between the acquired further images 21 of the cameras 31, 3r of the stereo camera system 3 and determining offset correction data, and / or determining a rotation between the acquired further images 21 of the cameras 31, 3r of the stereo camera system 3 and determining rotation correction data. Thus, the generated further calibration data 31 comprises offset correction data and / or rotation correction data.
[0079] It can be proposed that, within the scope of step a) or step c), different operating points of the movable lens system of the stereo medical microscope 1 are selected when acquiring at least a portion of the images 20, 21, wherein within the scope of step b) or step d), offset values are determined for individual operating points of the operating points of the movable lens system, wherein calibration data 30 and / or further calibration data 31 are generated taking into account the correspondingly determined offset values.
[0080] It can be provided that at least one additional image 22 of a uniformly reflecting and / or transmitting calibration object 42 is also acquired by means of the stereo camera system 3, wherein illumination correction data are generated based on the acquired at least one additional image 22 for correcting the illumination geometry of at least one light source (not shown) of the medical microscope 1, and the illumination correction data are taken into account during the generation of the calibration data 30 and / or the further calibration data 31. For this purpose, the illumination correction data are determined by means of the data processing device 2. For this purpose, the data processing device 2 determines a brightness curve in the acquired at least one additional image 22, in particular for different operating points. Since the brightness in the acquired at least one additional image 22 should remain constant, the illumination correction data can be generated based on the corresponding brightness curve, in particular the illumination correction data are determined from the brightness curve. The illumination correction data are stored in the signal processing device 6 in particular as calibration data 30, 31 for correction in continuous operation. The signal processing device 6 then applies the illumination correction data to the raw signals 10l, 10r.
[0081] In particular, the at least one additional image 22 of the uniformly reflecting and / or transmissive calibration object 42 is acquired according to steps a) to d), and the illumination correction data are generated. In particular, the correction can already be performed based on edge attenuation correction data (for correcting camera vignetting) before the brightness curve and the illumination correction data are determined. However, it can also be provided that the uniformly reflecting and / or transmissive calibration object 42 is part of the two-dimensional calibration object 41. The acquisition of the at least one additional image 22 of the uniformly reflecting and / or transmissive calibration object 42 can then also be performed simultaneously with other steps, for example step c).
[0082] Figure 2a and Figure 2b A schematic diagram of an embodiment of a three-dimensional calibration object 40 and a two-dimensional calibration object 41 is shown. Figure 2a Here a side view is shown, Figure 2bA top view is shown, which corresponds to how the medical microscope 1 will acquire a view of the calibration objects 40, 41 by means of the cameras 31, 3r. The three-dimensional calibration object 40 has a triangular shape with inclined surfaces (it can therefore also be referred to as a "tilted target"), so that different spacings (e.g. in the z direction) can be acquired simultaneously. The two-dimensional calibration object 41 has a checkerboard pattern on the top side. With the help of the checkerboard pattern, in particular, the offset and rotation (e.g. in the x-direction / y-direction) between the (further) images 20, 21 acquired by the cameras 31, 3r of the stereo camera system 3 can be determined. In addition or alternatively, the two-dimensional calibration object 41 can in principle also have other suitable structures, such as rectangular, linear and / or other suitable two-dimensional shapes, etc.
[0083] In the example shown, the calibration objects 40, 41 are shown together as a combined object. However, the calibration objects 40, 41 can also be formed separately from each other and / or used separately from each other.
[0084] In particular, it can also be provided that the three-dimensional calibration object 40 is formed according to one of the embodiments as described in DE 10 2019 131 646 A1, in particular according to the embodiment of FIG. 5 or FIG. 6 shown there.
[0085] exist Figure 3 A schematic flow chart of an embodiment of a method for calibrating a stereoscopic medical microscope is shown in FIG. Figure 1 The medical microscope components shown are implemented.
[0086] After the method start 200, which can be triggered, for example, by a technician, an image of at least one three-dimensional calibration object is acquired by means of a camera of a stereo camera system of a medical microscope in method step 201. For this purpose, (at least one) three-dimensional calibration object is arranged in the acquisition area of the medical microscope before the acquisition. In particular, it is proposed that method step 201 is performed for different working points of the medical microscope.
[0087] In method step 202, calibration data are generated based on the acquired imaging by means of a data processing device, in particular calibration data are determined based on the acquired imaging. The generated calibration data are stored for correction, in particular in a memory provided for this purpose of the medical microscope, for example in a memory of a signal processing device and / or a control device and / or an actuator of the medical microscope. In particular, calibration data are generated and in particular determined for different working points. The generated and in particular determined calibration data in particular include a camera projection matrix (extrinsic calibration data and / or intrinsic calibration data) and / or distortion correction field data and / or edge attenuation correction data and / or color shift field correction data.
[0088] Depending on the type of calibration data to be determined, it may be provided that different three-dimensional calibration objects are used in each case. For each of these three-dimensional calibration objects, an image is then acquired in method step 201 (in particular at different working points).
[0089] In method step 203, an image of at least one three-dimensional calibration object is acquired by means of a camera of a stereo camera system. Figure 2a The acquisition can be performed with the three-dimensional calibration object 40 (“tilted object”) shown in FIG. 4 . Due to the tilted surface, the blur curve in the acquired image can be evaluated along the optical axis (z direction), so that the corresponding focus position of the camera can be determined. In particular, the acquisition can be performed for different working points, in particular at different working distances from the three-dimensional calibration object. The corresponding focus position of the camera of the stereo camera system is determined based on the acquired image.
[0090] In method step 204, the focal positions of the cameras are set or brought closer to one another based on the determined values. This is done in particular manually by a technician. This is achieved in particular in such a way that the focal positions are always the same for both cameras at different working points (in particular working distances).
[0091] In method step 205 , a further image of the at least one two-dimensional calibration object is acquired by means of the camera of the stereo camera system.
[0092] In method step 206, further calibration data are generated, in particular determined, based on the acquired further images. In particular, it is provided that an offset (x / y offset) and offset correction data between the acquired further images of the cameras of the stereo camera system and a rotation and rotation correction data between the acquired further images of the cameras of the stereo camera system are determined. The generated, in particular determined, further calibration data are stored for correction purposes, in particular in a memory provided for this purpose of the medical microscope, for example in a memory of a signal processing device and / or a control device and / or an actuator of the medical microscope.
[0093] Depending on the type of calibration data to be determined, it may be provided that different two-dimensional calibration objects are used in each case. For each of these two-dimensional calibration objects, an image is then acquired in method step 205 (in particular at different operating points).
[0094] In method step 207, the calibration data and the further calibration data are applied to the acquired further image of the two-dimensional calibration object; in particular, the acquired further image is corrected in this way. This can be done, for example, by means of the signal processing device 6 ( Figure 1) in that the signal processing device corrects the raw signals 101, 10r with the aid of stored calibration data and stored further calibration data and provides a result (ie the image corrected in this way).
[0095] It can be provided that method steps 205, 206 and 207 are then repeated. In particular, it is provided that method steps 205, 206 and 207 are repeated until at least one predetermined optimization criterion is met. The predetermined optimization criterion can be, for example, a maximum value of an offset and / or rotation between the images of the two cameras of the stereo camera system (for example + / - 1 pixel and / or + / - 0.001°).
[0096] In method step 207, provision may be made to check the generated calibration data by at least partially applying them to the acquired image and / or the acquired further image and evaluating the result of the application, wherein if the evaluation result does not meet at least one predetermined criterion, method steps 201 to 207 are at least partially repeated. The at least one predetermined criterion may, for example, include a maximum value of an offset and / or rotation between the images of the two cameras of a stereo camera system, a corresponding maximum value of a distortion of the camera and / or a corresponding maximum value of an edge falloff of brightness (camera vignetting), etc.
[0097] If the calibration data have been optimized and / or checked, the calibration data and the further calibration data are stored for use in the field of the medical microscope in method step 208. The method then ends 209.
[0098] List of Reference Numerals
[0099] 1Medical Microscope
[0100] 2 Data processing device
[0101] 2-1 Computing Device
[0102] 2-2 Memory
[0103] 3 Stereo Camera System
[0104] 3l left camera
[0105] 3r right camera
[0106] 4 Imaging optics
[0107] 5. Actuator
[0108] 6Signal processing device
[0109] 7 Display device
[0110] 10l original signal (left camera)
[0111] 10r original signal (right camera)
[0112] 20 Imaging
[0113] 21 Additional Imaging
[0114] 22 Additional Imaging
[0115] 30 calibration data
[0116] 31 Additional calibration data
[0117] 40 3D calibration objects
[0118] 41 2D calibration objects
[0119] 42 Calibration objects for uniform reflection and / or transmission
[0120] 100 Medical Microscope Components
[0121] Method steps of 200-209 method
Claims
1. A method for calibrating a stereoscopic medical microscope (1), include: a) acquiring an image (20) of at least one three-dimensional calibration object (40) by means of a camera (31, 3r) of a stereo camera system (3) of the medical microscope (1); b) generating calibration data (30) based on the acquired imaging (20), wherein the generated calibration data (30) is stored for correction; c) acquiring a further image (21) of at least one two-dimensional calibration object (41) by means of the cameras (31, 3r) of the stereo camera system (3); d) generating further calibration data (31) based on the acquired further images (21), wherein the further calibration data (31) are stored for correction purposes.
2. The method according to claim 1, characterized in that Step a) and / or step c) are performed at different operating points of the medical microscope (1), wherein for each of the different operating points the calibration data (30) are generated in step b) and / or the further calibration data (31) are generated in step d).
3. The method according to claim 1 or 2, characterized in that: The calibration data (30) generated in step b) and / or the further calibration data (31) generated in step d) are at least partially applied to the acquired imaging (20) and / or the acquired further imaging (21) for correction, wherein step b) is performed and / or repeated for the corrected imaging (20) and / or step d) is performed and / or repeated for the corrected further imaging (21).
4. The method according to claim 3, characterized in that The calibration and the steps c) and d) are repeated until at least one predetermined optimization criterion is met.
5. The method according to any one of the preceding claims, characterized in that The generated calibration data (30) are checked by at least partially applying to the acquired image (20) and / or the acquired further image (21) and evaluating the result of the application, wherein if the evaluation result does not meet at least one predetermined criterion, steps a) to d) are at least partially repeated.
6. The method according to any one of the preceding claims, characterized in that Step b) and / or step d) includes one or more of the following measures: determining external reference calibration data, determining internal reference calibration data, determining distortion correction field data, determining edge attenuation correction data, and determining color shift field correction data.
7. The method according to any one of the preceding claims, characterized in that Step b) comprises determining the corresponding focus positions of the cameras (31, 3r) of the stereo camera system (3) and setting the corresponding focus positions.
8. The method according to any one of the preceding claims, characterized in that Step d) comprises: determining an offset between the further images (21) acquired by the cameras (3l, 3r) of the stereo camera system (3) and determining offset correction data, and / or determining a rotation between the further images (21) acquired by the cameras (3l, 3r) of the stereo camera system (3) and determining rotation correction data.
9. The method according to any one of the preceding claims, characterized in that Within the scope of step a) or step c), different operating points of the movable lens system of the stereo medical microscope (1) are selected when acquiring at least a portion of the imaging (20, 21), wherein within the scope of step b) or step d), offset values are determined for individual operating points of the operating points of the movable lens system, wherein the calibration data (30) and / or the further calibration data (31) are generated taking into account the correspondingly determined offset values.
10. The method according to any of the preceding claims, characterized in that At least one additional image (22) of a uniformly reflecting and / or transmitting calibration object (42) is also acquired with the aid of the stereo camera system (3), wherein illumination correction data are generated based on the acquired at least one additional image (22) for correcting the illumination geometry of at least one light source of the medical microscope (1), and the illumination correction data are taken into account during the generation of the calibration data (30) and / or the further calibration data (31).
11. A medical microscope assembly (100), comprising: A stereoscopic medical microscope (1) having a stereoscopic camera system (3) with cameras (3l, 3r) and storable calibration data (30, 31), and a data processing device (2). The data processing device (2) is configured to: i) generating calibration data (30) based on images (20) of at least one three-dimensional calibration object (40) acquired by means of cameras (31, 3r) of the stereo camera system (3), and storing the calibration data for correction; and ii) generating further calibration data (31) based on further images (21) of at least one two-dimensional calibration object (41) acquired by means of the cameras (31, 3r) of the stereo camera system (3), and storing the further calibration data for correction.
