Optical imaging device and method for improving displayed images

Abstract

To provide an optical imaging device and a method for improving a displayed image.SOLUTION: An optical imaging device includes a first image recording unit 2 and a second image recording unit 3 for generating at least a first and a second original images of an object 4, where the first and second original images differ with regard to at least an image parameter, where the image recording units 2 and 3 are arranged in such a method that the first and second original images are recorded from the same perspective. An image processing unit 12 configured to further process the first and second original images and an image display unit 16 configured to reproduce displayed images generated from the processed first and second original images are present. The image processing unit 12 is configured to supplement at least one of the first and second original images with image information from the other original image to generate a displayed image. Preferably the optical imaging device is a surgical digital microscope.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application claims the priority of German Patent Application No. 10 2019 118 508.4, the content of which is incorporated herein by reference in its entirety.

[0002] The present invention relates to an optical imaging device, in particular a digital surgical microscope, with at least two image recording units per observation direction, and a method for generating an image for an observer. The digital surgical microscope serves to ensure an improved spatial representation of a selected optical field for an observer, for example a medical specialist. In particular, adaptation of optical parameters such as optical magnification, object resolution and inherent depth of field is desirable. [Background technology]

[0003] Due to the digital processing, the surgical digital microscope has the advantage that the obtained images can be digitally post-processed and therefore improved with respect to optical parameters. However, this optical device also has technical limitations, namely, in this respect, for example, in the case of the simple optical system of a camera, a large depth of field is combined with a low object resolution.

[0004] One possible way to combat this problem is described in US Pat. No. 5,399,633. It discloses a stereomicroscope with respective beam paths for the observer's eyes, the optical effective diameter of the optical elements in the first beam path being implemented to be different from the optical effective diameter of the optical elements in the second beam path. In this way, it is achieved that the object resolution and depth of field of the images provided to the two beam paths, and therefore to both observer's eyes, are different. In the observer's brain, these two images are then combined to form an image optimized in terms of resolution and depth of field. However, the solution disclosed in the cited document has the drawback that the observer quickly tires, in particular as a result of the effort required to combine the different images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] German Patent No. 10 2006 036 300 B4 Specification Summary of the Invention [Problem to be solved by the invention]

[0006] The object of the present invention is to provide an improved optical imaging device, in particular a surgical digital microscope, with respect to the prior art and an image generating method such that the generated images are viewable with less user effort and contain more information content compared to the solutions known from the prior art. [Means for solving the problem]

[0007] This object is achieved by an apparatus and a method having the features of the independent claims. The dependent claims relate to advantageous developments and variants of the invention.

[0008] The optical imaging device according to the invention comprises a first and a second image recording unit for generating at least a first and a second original image of an object, where the original images differ with respect to at least some image parameters. In this case, the image recording units are arranged in such a way that the original images are recorded from the same viewpoint. Furthermore, the optical imaging device according to the invention comprises an image processing unit for further processing the original images and an image display for reproducing a display image generated from the processed original images. In this case, the image processing unit is configured to generate the display image by supplementing at least one of the two original images with image information from the other original image.

[0009] By supplementing one original image with image information from another original image, an overall improved displayed image can be created, and the new image can have improved characteristics, such as a higher dynamic range or lower noise.

[0010] This kind of image conditioning is particularly advantageous for applications in which the optical imaging instrument is a surgical digital microscope, in particular a surgical stereo microscope. In such cases, improved images are provided to the surgeon, thereby achieving a safer execution of potentially complex surgeries. In this case, the inventive solution can be implemented in one or both of the stereo channels.

[0011] It is particularly advantageous if the image parameters are depth of field and / or resolution In this respect, for example, the first image recording unit can be a colour camera and the second image recording unit can be a monochrome camera.

[0012] In this case, the maximum depth of field of the monochrome camera can be greater than that of the color camera, and the maximum resolution of the color camera can be greater than that of the monochrome camera.

[0013] As mentioned above, both cameras capture an object, for example an operating field, from at least approximately the same viewpoint. The image is then conditioned by an image processing unit and displayed to the observer on a planar or stereoscopic digital display unit. In this case, the maximum object resolution in the displayed image for objects in and near the focal plane can correspond almost or completely to the original image captured by the color camera. In both cases, the maximum object resolution in the displayed image is higher than the maximum object resolution in the image captured by the monochrome camera for objects in and near the focal plane. Objects that are outside the depth of field range of the color camera can have an object resolution in the displayed image that corresponds almost or completely to the object resolution in the original image captured by the monochrome camera. In both cases, the maximum object resolution in the displayed image for objects outside the focal plane is higher than the maximum object resolution in the image captured by the color camera. In this case, it is advantageous if both image recording units are focused on the same focal plane.

[0014] Similarly, the maximum depth of field of a color camera can be greater than that of a monochrome camera, and the maximum resolution of a monochrome camera is greater than that of a color camera.

[0015] In particular, both image recording units can also be colour cameras.

[0016] In one advantageous variant of the invention, a beam splitter is arranged in the light path between the object and the image recording unit. In this case, the beam splitter can provide both image recording units with exactly the same view of the object. Its characteristic curve can have a non-zero, as smooth as possible, spectral transmission profile in the visible spectrum range, in particular, so that the visible light is recorded by both image recording units with a predefined split ratio.

[0017] In particular, the beam splitter may be a dichroic beam splitter.

[0018] In one preferred embodiment of the present invention, an aperture diaphragm can be arranged upstream of one of the image recording units. In this regard, it is possible to use an adjustable aperture diaphragm, which can variably set the depth of field of the image recorded, for example, preferably by a monochrome camera. Optionally, another diaphragm can be used, for example, an additional diaphragm, preferably upstream of a color camera. An adjustable diaphragm is particularly advantageous for fluorescence recording, because then, by opening the diaphragm, it is possible to allow more fluorescence to reach the monochrome camera.

[0019] In order to vary the depth of field, the lens of the image recording unit, for example a monochrome camera, can also be designed such that the depth of field of the image is increased.

[0020] The method according to the invention for generating at least one image to be displayed on an image display unit from at least two original images of an object recorded by two different image recording units is specific in that the two original images differ at least with respect to certain image parameters. According to the invention, the at least two original images are recorded from the same viewpoint and at least one of the original images is supplemented by an image processing unit with image information of the other original image in order to generate the displayed image.

[0021] As mentioned above, the method can be carried out by means of a digital surgical microscope, for example by means of a digital surgical stereo microscope.

[0022] In this case, it is advantageous if image registration is performed for both image channels. In this case, image registration should be understood to mean a method step that ensures that the same image segment is provided to both eyes of the observer even at different viewpoints. Such registration can be performed, for example, by identification of two different image regions and subsequent matching. Matching should be understood to mean a measure that ensures that the two image segments are congruent. An improved overall visual impression can be achieved by image registration. Brightness matching can also be performed for both image channels, which can further improve the visual impression.

[0023] The display image does not necessarily have to be a color image. Alternatively or additionally, a new monochrome image can be calculated as the display image, which also has a combination of a large depth of field and a high maximum object resolution of objects in the focal plane.

[0024] Furthermore, color images can be synthesized to gradually transition to monochrome, e.g., grayscale, images in regions outside the depth of field range of the original color image, thereby avoiding blurred color or chromaticity information for finely structured brightness values ​​or luminance patterns.

[0025] Furthermore, the first image recording unit can record a monochrome image at a smaller optical imaging scale than the color image recorded by the second image recording unit, in which case the recorded monochrome image is first digitally zoomed and appropriately interpolated, so that the pre-processed monochrome image shows the same object segments as the color image.

[0026] Alternatively or additionally, the first image recording unit implemented as a monochrome camera can have a smaller number of pixels than the color camera. In this case, the monochrome image is first scaled up / interpolated to the number of pixels of the color image. In principle, an image distance unit that records an image with a higher depth of field can have a smaller number of pixels.

[0027] Exemplary embodiments and variants of the invention are explained in more detail below with reference to the drawings. [Brief description of the drawings]

[0028] [Figure 1] 1 shows in schematic form an exemplary surgical digital microscope according to the present invention; [Diagram 2] 1 illustrates one exemplary embodiment of a characteristic curve of a dichroic beam splitter. [Diagram 3] 1 shows a flow diagram of a first image processing method. [Figure 4] 13 shows a flow diagram of a second image processing method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] FIG. 1 shows a schematic diagram of an imaging device 1 according to the invention in one embodiment as a surgical digital microscope. It can be a planar microscope with one observation viewpoint, as shown, or a stereo microscope with two observation viewpoints. For each observation viewpoint, the surgical microscope comprises a pair of cameras, including a color camera 2 and a monochrome camera 3, as image recording units for recording an original image of the object 4. A split beam path of light from the object 4 to the cameras (2, 3) is realized by a dichroic beam splitter 5. In this case, the beam path passes from the object 4 via the main objective 6 to the zoom optical unit 17 and then to the dichroic beam splitter 5. A partial beam then passes via the video objective 8 to the color camera 2. On the other hand, a second partial beam passes via an adjustable aperture diaphragm 10 and a video objective 11 to the monochrome camera 3. In the case of fluorescence recording, the surgical digital microscope 1 comprises rotatable fluorescence observation filters 7 and 9 between the beam splitter 5 and the video objectives 8 and 11. For processing the raw images recorded by the two cameras 2 and 3, the surgical digital microscope 1 comprises an image processing unit 12 which is connected on the one hand to the colour camera 2 and the monochrome camera 3 and on the other hand to an image display unit 16. The object 4 is illuminated by a light source 13.

[0030] In this exemplary embodiment, the color camera 2 and the monochrome camera 3 are focused on the same focal plane. The imaging on the color camera 2 is realized in such a way that the image of the color camera 2 has a high resolution for objects that are in the focal plane. However, this comes at the expense of a shallow depth of field for the image of the color camera 2. The image of the monochrome camera 3 has a large depth of field, but in exchange for a lower resolution for objects that are in the focal plane.

[0031] It is here assumed that the depth of field of the monochrome camera 3 is adapted by both the adjustable aperture diaphragm 10 and the video objective 11. The adjustable aperture diaphragm 10 can likewise play a role in setting an appropriate light transmission during fluorescence recording. The monochrome camera 3 can include a sensor with a lower resolution compared to the color camera 2, since there is a lower resolution due to the large depth of field. A sensor with a lower resolution is more light sensitive, which is advantageous in the case of fluorescence recording. As a result, for example, more light can pass through the dichroic beam splitter 5 to the color camera 2.

[0032] According to the present invention, the image recorded by one camera is supplemented by the image processing unit 12 with the image information of the other camera, thereby generating a generated overall image that combines both a large depth of field and high resolution for each observation viewpoint.

[0033] 2 shows the characteristic curve of the dichroic beam splitter 5 in an exemplary embodiment. In the visible spectral range, the dichroic beam splitter 5 has a smooth transmittance profile with a constant transmittance, whereby visible light passes to the color camera 2 and the monochrome camera 3 at a predetermined split ratio. The dichroic beam splitter 5 has a high transmittance for near-infrared light (700-850 nm). It is also assumed that in addition to light in the visible wavelength range, light in the near-infrared wavelength range is also detected by the monochrome camera.

[0034] Fig. 3 shows a flow diagram of a local comparison algorithm 14 that can be implemented by the image processing unit 12. In this case, image F corresponds to the original image recorded by the color camera 2, and original image M corresponds to the image recorded by the monochrome camera 3. The local contrast and local image sharpness of images F and M are determined by a conventional image sharpness algorithm. A local comparison of contrast and image sharpness is performed for each image region of images M and F. The display image D1 is then synthesized such that in each image region of D1, the corresponding image region from image F or image M with higher sharpness or higher contrast is used, and color information is taken from image F. In case of higher resolution grayscale information from image M, it is assumed that the resulting color image D1 is further sharpened digitally. A blend of the two original images M and F is preferably used in the transition regions. The method can be used only on the luminance or luma channel ("Y") of a color image F, for example in YCbCr format, while the chrominance components of the color image F are used unchanged in F' (or after being further sharpened digitally based on the luminance channel). In case of a luminance-chrominance color model different from YCbCr, the method is likewise preferably performed separately for the luminance and chrominance channels. Other color models from the prior art can be used as well, with appropriate distinction between grayscale and color information.

[0035] Fig. 4 shows a flow diagram of an alternative image processing method, in which a depth map algorithm is used in the image processing unit 12. In this case, a depth map produced by a depth sensor is used for data processing. In the case of a surgical stereo microscope, it is assumed that the depth map is obtained by stereo reconstruction. The depth of the respective image regions, known from the depth map, determines which of the images M and F has locally a deeper depth of field or a higher object resolution. For the display image D2, for each image region, the image region with the deeper depth of field or the higher resolution is selected. The color information of the display image D2 is adopted from the image F. Optionally, a suitable mixed weighting of the images F and M can be performed in the transition regions. [Explanation of symbols]

[0036] 1 Imaging equipment 2 Color Camera 3. Monochrome Camera 4 objects 5 Beam splitter 6 Main objective lens 7 Fluorescence Observation Filters 8 Video Objectives 9 Fluorescence Observation Filters 10 Aperture 11 Video Objective Lens 12 Image Processing Unit 13 Light source 14 Local Comparison Algorithms 15 Depth Map Algorithm 16 Image display unit 17 Zoom optical unit F Color image M Monochrome Image D1 Display image D2 Display image

Claims

1. an optical imaging device (1) comprising first (2) and second cameras (3) for generating at least first and second original images (F, M) of an object (4), said original images (F, M) differing at least with respect to certain image parameters, said first and second cameras (2, 3) are positioned in such a way that the original images (F, M) are recorded from the same point of view; the presence of an image processing unit (12) for further processing said original images (F, M) and an image display unit (16) for reproducing display images (D1, D2) generated from said processed original images (F, M); - said image processing unit (12) is adapted to supplement at least one of the two original images (F, M) with image information from the other original image (F, M) to generate a display image (D1, D2); - said image parameter is the depth of field; - the first camera (2) is a color camera and the second camera (3) is a monochrome camera; - the maximum depth of field of said monochrome camera (3) is greater than that of said color camera (2); An optical imaging device (1).

2. 2. Optical imaging instrument (1) according to claim 1, characterized in that the optical imaging instrument (1) is a surgical digital microscope.

3. 3. Optical imaging device (1) according to claim 1 or 2, characterized in that the maximum resolution of the colour camera (2) is higher than that of the monochrome camera (3).

4. 3. Optical imaging device (1) according to claim 1 or 2, characterized in that the maximum resolution of the monochrome camera (3) is higher than that of the colour camera (2).

5. Optical imaging device (1) according to any one of claims 1 to 4, characterized in that a beam splitter (5) is arranged in the optical path between the object (4) and the first and second cameras (2, 3).

6. 6. Optical imaging instrument (1) according to claim 5, characterized in that the beam splitter (5) is a dichroic beam splitter.

7. Optical imaging apparatus (1) according to any one of the preceding claims, characterized in that an aperture stop (10) is arranged upstream of one of the first and second cameras (2, 3).

8. 8. Optical imaging instrument (1) according to claim 7, characterized in that the aperture stop (10) is implemented so as to be adjustable.

9. Optical imaging instrument (1) according to any one of claims 1 to 8, characterized in that the optical imaging instrument (1) is a surgical stereo microscope, and the features presented in at least one of claims 1 to 7 are realised for at least one of the stereo channels.

10. Optical imaging instrument (1) according to claim 9, characterised in that the features presented in at least one of claims 1 to 7 are realised for both stereo channels.

11. 1. A method for generating at least one display image (D1, D2) to be displayed on an image display unit (16) from at least two original images (F, M) of an object (4) recorded by two different cameras (2, 3), comprising: - said two original images (F, M) differ with respect to at least some image parameters, - said at least two original images (F, M) are recorded from the same viewpoint, at least one of said original images (F, M) is supplemented by an image processing unit (12) with image information of the other original image (F, M) to generate said display image (D1, D2), said original images (F, M) differing in terms of depth of field, the first original image (M) is a monochrome image and the second original image (F) is a color image, A method characterized in that said monochrome image (M) has a greater depth of field than said colour image (F).

12. 12. The method of claim 11, wherein the method is performed by a surgical digital microscope.

13. 13. The method according to claim 11 or 12, characterized in that the method is performed by means of a surgical stereomicroscope and image registration is performed for both image channels.

14. Method according to any one of claims 11 to 13, characterized in that the method is carried out by means of a surgical stereomicroscope and brightness matching is carried out for both image channels.

Images