Medical imaging system

Abstract

Medical imaging system with - an image sensor designed to produce an image of at least one area of ​​interest, wherein the image has a plurality of pixels based on optical radiation from the area of ​​interest reaching the image sensor; - a depth map sensor configured to generate a depth map of a field of interest, wherein the field of interest at least partially overlaps the area of ​​interest and the depth map contains information that makes it possible to identify at least one first pixel of the plurality of pixels that is farther from the image sensor than at least one second pixel of the plurality of pixels, and - a brightness control configured to operate in a first mode that controls the brightness of the image on the basis of a first plurality of pixels, including at least one first pixel that is located at a distance from the image sensor greater than a predetermined percentage of the distance of the pixel furthest from the image sensor, the predetermined percentage being at least 50%.

Description

Technical area

[0001] The disclosure relates generally to the field of image acquisition and specifically to brightness control in a medical imaging system. Technical background

[0002] When generating images, especially video images, in a surgical context, e.g., in neurosurgery, ENT, or involving tubular hollow organs, it is difficult to control the brightness or exposure of an image. Without a dedicated control, the medical imaging system typically produces images that appear too dark or too bright for the user. To give the user control over the brightness of an image, medical imaging systems have employed what is commonly known as automatic exposure control. This means that the more light reaching the image sensor, the shorter the time the shutter remains open or the imaging chip remains sensitive to light. However, this automatic approach is usually unable to meet user expectations.

[0003] Another approach was to give the user direct control over the light source illuminating the surgical site. This means that if the user finds the image too dark, they can increase the light source's intensity, and vice versa. However, this approach requires continuous adjustment of the light intensity.

[0004] Another approach, as described in EP 2 256 688 B1, allows the user to define a region of interest in an image, where the output of the imaging system is a combination of the overall image generated with a first exposure and the region of interest captured with a second exposure. However, this approach places a significant burden on the user in defining the desired region.

[0005] US 2016 / 0 057 332 A1 relates to systems and methods for capturing high dynamic range (HDR) images using camera array arrays. A method for generating an HDR image using an array camera comprises: defining at least two subsets of active cameras, determining capture settings for each subset, the capture settings including at least two exposure settings, configuring the active cameras with the determined settings, acquiring image data with the active cameras, synthesizing an image for each subset of cameras, and generating an HDR image from the synthesized images.

[0006] US 2016 / 0 191 896 A1 relates to a method and an electronic information processing system comprising capturing a first image of a scene with a first exposure setting using a 3D camera, correlating distances from the 3D camera with exposure values ​​across a plurality of image elements of the first image, selecting an exposure parameter for at least one of the image elements within a specific depth range, capturing a second image of the scene according to the exposure parameter, and constructing a composite image based on at least a portion of the second image for the corresponding image elements.

[0007] Therefore, there is still a need for a medical imaging system and method that better adjusts the brightness of an image to the user's expectations. Summary of the invention

[0008] The aim of the invention is to control the brightness of images, e.g. from endoscopes or other imaging devices, in such a way that the user has a good visual perception of a surgical site and can recognize individual features of the surgical site that are imaged.

[0009] According to a first aspect of the disclosure, a medical imaging system according to claim 1 is provided.

[0010] In solving the above problem, the inventors realized that a user viewing a medical image, especially a series of medical images as a video, is typically only interested in a small portion of the overall image generated by the image sensor. This portion is referred to below as the region of interest.

[0011] Consequently, the inventors have found that when controlling the brightness of an image to the overall image, the actual region of interest can play only a minor role in controlling the brightness, even if it is indeed this region of interest that the user wants to clearly see.

[0012] Finally, the inventors found that, especially in neurosurgery, otolaryngology, or tubular hollow organs, the region of interest is often associated with the region of the overall image that is further away from the imaging sensor, in particular the region furthest away from the imaging sensor.

[0013] In view of these new findings identified by the inventors, the medical imaging system according to the present disclosure comprises a depth map sensor configured to generate a depth map of a region of interest in which the region of interest at least partially overlaps.

[0014] As required in view of the present teachings, the depth map will contain information that enables at least a first pixel from the plurality of pixels of the image generated by the image sensor to be identified, wherein this first pixel is farther from the image sensor than at least a second pixel from the plurality of pixels.

[0015] In example embodiments, the depth map contains information that allows a first plurality of pixels to be identified that are farther away than a second plurality of pixels. For other example embodiments, the depth map will contain, for at least a subset of pixels within the region of interest, the individual distance for each pixel from the subset of pixels to the image sensor. For some example embodiments, all pixels of the region of interest covered by the region of interest have associated depth information. Furthermore, in some example embodiments, the depth map sensor generates a depth map that includes all pixels of the region of interest or even all pixels that make up the image generated by the image sensor.

[0016] It should be noted that the terminology "a pixel at a distance from the image sensor" is shorthand for the fact that an area of ​​a scene that emits optical radiation, which is received by the image sensor and then represented by a pixel in the image, is located at a distance from the image sensor. In other words, it is the distance from the image sensor of an area of ​​a scene imaged by a pixel, which is referred to in shorthand as the "distance of a pixel from the image sensor."Or, the terminology "a first pixel farther from the image sensor than a second pixel" is shorthand for the statement that a first region of a scene that emits optical radiation received by the image sensor and then represented by the first pixel in the image is at a greater distance from the image sensor than a second region of the scene that emits optical radiation received by the image sensor and then represented by the second pixel in the image. The various shorthands are intended to simplify the explanation of the technical concept and make the new approach easier to convey.

[0017] Using the information from the depth map, the at least one first pixel, in particular a first plurality of pixels, can be identified as the pixel(s) that are likely to be of interest to the user viewing the medical image. Therefore, at least in a first mode, the brightness control controls the brightness of the image based on the at least one first pixel. That is, similar to controlling the brightness of an image based on the entire image, i.e., taking all pixels of the image into account, the brightness is now controlled based on the at least one first pixel or the first plurality of pixels. In exemplary embodiments, all other pixels that are not contained in the at least one first pixel or not in the first plurality of pixels are not taken into account for the brightness control when the brightness control operates in the first mode.

[0018] According to some examples, the depth map sensor can acquire depth information from the time-of-flight information of the image sensor or a camera module. Some corresponding time-of-flight systems use a special image sensor, typically a near-infrared sensor, that determines the time-of-flight of light emerging from the scene and arriving at the image sensor. Other example embodiments use a system that projects specific patterns into the scene and analyzes the patterns reflected from the scene to obtain depth information. Such systems can use structured light, which allows regions farther from the image sensor to receive more light than other regions closer to the image sensor. These systems can also be used for triangulation.Further exemplary embodiments provide the depth map sensor with stereoscopic images from which the desired depth map can be derived. Some systems use two different objectives or lens systems to obtain two images captured from two different positions. This can be, for example, an image sensor in which a first region is associated with the first image and a second region with the second image, or an image sensor is associated with the first image and a second image sensor with the second image. For each embodiment, the image sensor configured to generate an image of at least one region of interest can also be used to create the depth map, in particular for triangulation purposes.

[0019] In the first mode, the brightness of the image is controlled based on at least one first pixel that is farther from the image sensor than at least one second pixel that is different from the at least one first pixel. Since the depth map can be generated automatically, the control of the brightness can be detected automatically. Furthermore, it is possible to adjust the brightness repeatedly or even continuously because the depth map can be updated continuously or even continuously and the control of the brightness can be performed or adjusted on an ongoing basis. For some example embodiments, the depth map is generated for each image generated by the image sensor. In certain example embodiments, the control of the brightness is performed or adjusted each time the depth map is created.In some example embodiments, at least one of the regions and the area of ​​interest includes all pixels of the image as provided by the image sensor.

[0020] In some example embodiments, the predetermined percentage may be greater than 99%, greater than 98%, greater than 95%, or greater than 90% of the distance of the pixel farthest from the image sensor. In other example embodiments, the predetermined percentage may be greater than 85%, greater than 75%, or greater than 60% of the pixel farthest from the image sensor.

[0021] According to another exemplary embodiment, the brightness controller is further configured to operate in a second mode that controls the brightness of the image based on an overall brightness of the image that includes at least a second pixel of the plurality of pixels that is different from the at least one first pixel.

[0022] This refinement allows the brightness control to be modified so that other pixels are also taken into account by the brightness control. In some exemplary embodiments, the at least one second pixel consists of a second plurality of pixels that are adjacent to or located near the at least one first pixel. In other exemplary embodiments, the overall brightness of the image is determined from all pixels.

[0023] According to a further exemplary embodiment, the system comprises an image processing unit configured to receive a first image and a second image from the image sensor and configured to overlay the second image with at least one pixel from the first image that has a greatest distance from the image sensor.

[0024] This refinement allows the first image to be combined with a selected portion of the second image. In some example embodiments, the brightness of the first image is controlled differently than the brightness of the second image. In certain example embodiments, the first image is obtained when the brightness control operates in the first mode, and the second image is obtained when the brightness control operates in the second mode. Such an embodiment allows a composite image to be obtained that includes the overall image with a brightness controlled based on the overall brightness overlaid with selected pixels from an image whose brightness was controlled based on the at least one first pixel.This makes it possible to create an image that has a suitable brightness to detect features outside the region of interest and also has a suitable brightness that allows features within the region of interest to be detected. In some example embodiments, the illumination of the scene and / or the shutter speed is different when the first image is captured than when the second image is captured. In conjunction with this embodiment or another embodiment, regions of the image that are close to the image sensor, i.e., pixels that are located in regions of the image that correspond to regions of the scene imaged by the image sensor and that are close to the image sensor, can also be provided with a constant brightness value in order to avoid flare effects on the image.For example, a medium brightness can be selected for the pixels of such areas.

[0025] In some exemplary embodiments, the overlay is performed such that the pixels of the first image replace the corresponding pixels of the second image. In other exemplary embodiments, the pixels of the first image are applied to the corresponding pixels of the second image with a certain opacity or transparency.

[0026] The individual pixels taken from the first image to overlay the corresponding pixels in the second image can be determined in different ways to meet the user's expectations.In some exemplary embodiments, these pixels are selected taking into account at least one of the following: a) the at least one pixel from the first image having the at least one first pixel, b) the at least one pixel from the first image that is / are the same as the at least one first pixel, c) the at least one pixel from the first image that has / have all pixels of the image that do / do not exceed a certain brightness threshold, d) the at least one pixel from the first image that is / are within a predefined area or shape of the image, and e) the at least one pixel from the first image that is / are within a predetermined distance from the at least one first pixel.

[0027] For exemplary embodiments, the area or shape may be predefined, e.g., a circular area centered on the image, it may be defined with parameters that can be controlled by the user, e.g., the position and / or radius of a circular area on the image may be set by the user, or it may be defined interactively by the user, e.g., by defining the area with a touchscreen or drawing on the screen with a mouse.

[0028] Considering the greatest distance for a number n of pixels from the image sensor with a total of m pixels, this can be understood as meaning that each of the n pixels has a greater distance than each of the other remaining mn pixels in the image. According to an exemplary embodiment, all m pixels are sorted in descending order of distance from the image sensor, with the top n pixels in this sorted list being considered the n pixels with the greatest distance from the image sensor.

[0029] Furthermore, an alternative is disclosed in which, in the first mode, the brightness of the image is controlled based on a first plurality of pixels, including the at least one first pixel that lies within a closed boundary enclosing the pixels with the greatest distance from the image sensor.

[0030] This alternative uses a closed region based on which the brightness of the image is controlled. For some example embodiments of the alternative, the brightness control determines an isoline or contour line connecting pixels equidistant from the image sensor or pixels similarly distanced from the image sensor. In some example embodiments of the alternative, those pixels are selected that are approximately 99%, 98%, 95%, or 90% of the maximum distance to the image sensor. In other embodiments, the pixels are selected that are approximately 85%, 75%, 60%, or 50% of the maximum distance to the image sensor. Certain example embodiments of the alternative apply a tolerance window to these values ​​that does not exceed 1%, 2%, 5%, 10%, or 25%. All pixels within this isoline or contour line defining the closed boundary are used to control the brightness of the image.

[0031] Furthermore, an alternative is disclosed in which, in the first mode, the brightness of the image is controlled based on a first plurality of pixels including at least a first pixel belonging to a cluster of pixels having the greatest distance from the image sensor.

[0032] This alternative uses well-known clustering methods, specifically collectivity-based clustering, hierarchical clustering, centroid-based clustering, distribution-based clustering, density-based clustering, k-means clustering, and seat-based clustering. Once the pixel block with the greatest distance from the sensor has been identified, these pixels from the cluster are used to control the image brightness.

[0033] Further, an alternative is disclosed wherein, in the first mode, the brightness of the image is controlled based on a first plurality of pixels including the at least one first pixel, the first plurality of pixels having a preset percentile of pixels farthest from the image sensor.

[0034] The percentile is determined by the percentage of pixels that exceed a certain distance from the image sensor. This means that the lower percentiles contain only pixels that are far from the image sensor. As the percentile increases, pixels that are medium or closer to the image sensor begin to be included, and the upper percentiles further include pixels that are close to the image sensor. Finally, the 100th percentile contains all pixels.

[0035] The percentile that determines which pixels are considered for brightness control may be selected as the first percentile, second percentile, fifth percentile, or tenth percentile. For other example embodiments of the alternative, the percentile that determines which pixels are considered for brightness control may be selected as the 15th percentile, 25th percentile, 40th percentile, or 50th percentile. In certain example embodiments of the alternative, no pixels above the specified percentile are considered for brightness control. In some example embodiments of the alternative, all pixels of this preset percentile are considered for brightness control.

[0036] Furthermore, an alternative is disclosed in which, in the first mode, the brightness of the image is controlled on the basis of a first plurality of pixels including the at least one first pixel, the first plurality of pixels comprising pixels having a distance d from the image sensor, where d > dmin + f* (dmax - dmin), where dmin denotes the minimum distance from the image sensor of all pixels, dmax denotes the maximum distance from the image sensor of all pixels, and f denotes a factor between 0 and 1.

[0037] In this refinement, the factor f may be a preset value. For some example embodiments of the alternative, the factor is chosen as 0.99, 0.98, 0.95, or 0.9. For certain example embodiments of the alternative, the factor is chosen as 0.85, 0.75, 0.6, or 0.5. For other example embodiments of the alternative, the factor may be selected by the user from a number of preset factors. For further example embodiments of the alternative, the user may enter the factor as a numerical value and / or change the value via a user control. For some example embodiments of the alternative, no pixels are used that have a distance smaller than d. For certain example embodiments of the alternative, all pixels that satisfy this inequality are used to control the brightness of the image.

[0038] According to a further exemplary embodiment, the medical system further comprises a display configured to display the image and controls that allow the user to define a closed contour on the image, wherein the brightness control is further configured to control the brightness of the image based on inner pixels enclosed by the contour.

[0039] This design, which allows the omission of the depth map sensor, gives the user direct control over the area of ​​the image on the basis of which the brightness control should control the brightness of the image.

[0040] According to a second aspect of the disclosure, a method for controlling the brightness of an image according to claim 5 is provided.

[0041] Implementations of the disclosure may also be embodied as software or firmware, stored in a suitable medium, and executable to perform various versions of the methods described herein. These and other features of the disclosure will become apparent from the following description of exemplary embodiments, considered in conjunction with the accompanying drawings. Short description of the drawings Fig. 1 shows an exemplary embodiment of a medical system according to the present disclosure. Fig. 2 shows an example image generated by the image sensor. Fig. 3 shows another exemplary embodiment of a medical imaging system. Fig. Figure 4 shows an overlay of images created with varying exposure or brightness settings. Fig. 5 shows an example of a method for controlling the brightness of an image. Detailed description of the exemplary embodiments

[0042] Fig. 1 shows an exemplary embodiment of a medical imaging system 10 with a video endoscope 12. The medical imaging system 10 further comprises an image sensor 14 in combination with a depth map sensor 16. The system further comprises a brightness controller 18 and a display 20.

[0043] The image sensor 14 has a field of view 22. The visual information in the field of view is provided by the optical radiation of the viewed scene and received by the image sensor 14. The image sensor 14 generates an image 40 (see Fig. 2), which consists of several pixels 42 (see Fig. 2). The 42 pixels are only symbolic and cover the image sensor.

[0044] As in Fig. 1, the field of view 22 is directed toward an opening in the cranial bone 24 and the dura mater 26 in a skull 28. The opening leads to a surgical site containing a malignant tumor 30.

[0045] Starting from the central axis 32 of the field of view 22, the field of view 22 is bounded by a first radius, identified by the reference number 34. The opening through the skull bone 24 has a second radius, identified by the reference number 36. The opening through the dura mater has a third radius, identified by the reference number 38.

[0046] Fig. 2 shows an exemplary embodiment of an image 40 generated by the image sensor 14. The image 40 consists of a plurality of pixels 42, some of which are illustrated by small squares. Fig. Figure 2 also shows the first radius 34, the second radius 36, and the third radius 38. It should be noted that the present disclosure is not based on circular shapes. All other shapes, including rectangular shapes or polygonal shapes, are also suitable for practicing according to the present disclosure.

[0047] The area of ​​the image 40 in which the user is interested lies within the third radius 38 and is referred to as the region of interest (ROI) 44.

[0048] The depth map sensor 16 generates a depth map of a field of interest 46, indicated by the dashed line. The depth map sensor 16 generates depth information for each pixel 42 within the field of interest 46. For other exemplary embodiments, the depth map sensor 16 provides depth information only for a subset of pixels 42 within the field of interest 46.

[0049] Fig. Figure 1 also shows an example of dmin, the minimum distance of all pixels from the image sensor, and dmax, the maximum distance of all pixels from the image sensor. As explained above, the terminology "a pixel at a distance from the image sensor" is a shorthand way of saying that an area of ​​a scene that emits optical radiation, which is received by image sensor 14 and then represented by a pixel 42 in the image, is at a distance from image sensor 14.

[0050] Fig. 2 shows an exemplary image 40 generated by the image sensor 14, including two examples of areas that typically result in very strong reflection and make other areas of the image 40 appear very dark. One example is the first area 48, which is due to a reflection from the skull bone 24. Another example is the second area 50, which is a reflection from the dura mater 26. Since in the first mode, the corresponding pixels 42 are not far from the image sensor 14, they are not taken into account when controlling the brightness of the image 40.

[0051] Fig. 3 shows a further exemplary embodiment of a medical imaging system 10 in addition to the one shown in Fig. 1, a camera control unit 60 consisting of the brightness control 18, an image processing unit 62, a light source control unit 64, and a human-machine interface (MMI) 66, which enables the user to control the medical imaging system 10. Furthermore, the video endoscope 12 is held by a holder 68.

[0052] Fig. 4 shows an exemplary functionality of the image processing unit 66. The medical imaging system 10 captures, for example, first images with a brightness in a first mode and second images 72 in a second mode. In some embodiments, the exposure time of the second images 72 is shorter than that of the first images 70. The image processing unit 66 overlays a second image 72 with pixels from a previous or subsequent first image 70 to obtain a composite image 74, which is then displayed on the display 20.

[0053] Fig.5 shows an exemplary embodiment of a method. First, an image 40 from the image sensor 14 is displayed. In a further step, the region of interest is defined, either manually by the user or automatically based on the information from the depth map sensor 16. A first image 70 is generated with the brightness control in a first mode, and a second image 72 is generated with the brightness control 18 in a second mode. In this exemplary embodiment, the shaded areas represent the parts of the overall image that are used to control the brightness. As can be seen, in this exemplary embodiment, those pixels 42 on the basis of which the brightness was controlled in the first mode are excluded from the brightness control in the second mode. In a final step, the second image 72 is overlaid with the pixels from the first image 70 that are elements of the desired region.

[0054] The terms "comprising," "including," "carrying," "having," "containing," "involving," and the like, as used herein, are to be construed as non-limiting—that is, including but not limited to. The use of ordinal terms such as "first," "second," "third," etc., in the claims to amend a claim element does not, of itself, imply priority, precedence, or ordering of one claim element over another, or the chronological order in which acts of a method are performed. Rather, unless expressly stated otherwise, such ordinal terms are used merely as labels to distinguish one claim element with a particular name from another element with the same name (but for the use of the ordinal term).

[0055] The foregoing has presented the features and technical advantages of the invention rather broadly so that the following detailed description of the invention may be more fully understood. It should be understood by those skilled in the art that the disclosed conception and specific embodiments can be readily utilized as a basis for modifying or designing other structures to carry out the same purposes of the invention. It should also be recognized by those skilled in the art that such equivalent constructions do not depart from the scope of the invention as set forth in the appended claims.

[0056] Although the invention and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations may be made herein without departing from the spirit and scope of the invention as defined by the appended claims. The combinations of features described herein are not to be interpreted as limiting, and the features contained herein may be employed in any functional combination or subcombination consistent with the invention. This description, therefore, is to be construed as providing written disclosure under U.S. patent law and all applicable foreign patent laws for any functional combination or subcombination of the features contained herein.

[0057] Furthermore, the scope of the present application is not limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods, and steps described in the specification. Those skilled in the art will recognize from the disclosure of the invention processes, machines, manufacture, composition of matter, means, methods, or steps now or later developed which perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be used in accordance with the invention. Accordingly, the appended claims are intended to include such processes, machines, manufacture, compositions of matter, means, methods, or steps within their scope.

Claims

[1] Medical imaging system with - an image sensor configured to generate an image of at least one region of interest, the image comprising a plurality of pixels based on optical radiation from the region of interest reaching the image sensor; - a depth map sensor configured to generate a depth map of a field of interest, wherein the field of interest at least partially overlaps the region of interest, and the depth map includes information enabling at least a first pixel of the plurality of pixels to be identified that is farther away from the image sensor than at least a second pixel of the plurality of pixels, and - a brightness controller configured to operate in a first mode that controls the brightness of the image based on a first plurality of pixels, including the at least one first pixel, that have a distance from the image sensor that is greater than a predetermined percentage of the distance of the pixel furthest from the image sensor, the predetermined percentage being at least 50%. [2] The medical imaging system of claim 1, wherein the brightness controller is further configured to operate in a second mode that controls the brightness of the image based on an overall brightness of the image having at least a second pixel of the plurality of pixels that is different from the at least one first pixel. [3] A medical imaging system according to any one of the preceding claims, further comprising an image processing unit configured to obtain a first image and a second image from the image sensor and configured to overlay the second image with at least one pixel from the first image at a greatest distance from the image sensor. [4] A medical imaging system according to any preceding claim, further comprising a display configured to display the image and user controls enabling the user to define a closed contour on the image, wherein the brightness control is further configured to control the brightness of the image based on interior pixels enclosed by the contour. [5] A method for controlling the brightness of an image, the method comprising the following steps: - generating an image of a region of interest using an image sensor, the image comprising a plurality of pixels based on optical radiation from the region of interest reaching the image sensor, - generating a depth map of a field of interest using a depth map sensor, wherein the field of interest at least partially overlaps the region of interest and the depth map comprises information enabling at least a first pixel of the plurality of pixels to be identified that is farther away from the image sensor than at least a second pixel of the plurality of pixels, and - the brightness of the image is controlled in a first mode by means of a brightness controller on the basis of a first plurality of pixels, including the at least one first pixel, which have a distance from the image sensor that is greater than a predetermined percentage of the distance of the pixel furthest from the image sensor, the predetermined percentage being at least 50%.

Images