Medical imaging device, medical system, and method for colour adjustment of a medical imaging device

Abstract

The invention relates to a medical imaging device (10) comprising: an imaging unit (12) which is designed to perform multispectral and / or hyperspectral image capturing, wherein the imaging unit (12) is designed to take at least one intraoperative calibration image (14) of a calibration object (18) that can be intraoperatively inserted into a cavity (16) while the calibration object (18) is inserted in the cavity (16); and a calibration unit (20) which is designed to perform, in accordance with the calibration image (14), colour adjustment for the image capturing, which includes adjusting a colour calibration, in particular a white-balance calibration. The invention also relates to: a medical system (34) having a medical imaging device (10); a calibration object (18); and a method for colour adjustment of a medical imaging device (10).

Description

[0001] Medical imaging device, medical system and method for color adjustment of a medical imaging device

[0002] The invention relates to a medical imaging device, a medical system and a method for color adjustment of a medical imaging device.

[0003] Imaging devices such as endoscopic or exoscopic devices that generate multispectral or hyperspectral images are known from the prior art. Multispectral or hyperspectral images have, in addition to two spatial dimensions, such as a conventional camera image, a spectral dimension. The spectral dimension encompasses several spectral bands (wavelength bands). Multispectral and hyperspectral images differ primarily in the number and width of their spectral bands.

[0004] Several imaging devices for generating such multispectral or hyperspectral images are known, particularly in the context of medical applications. For example, DE 20 2014 010 558 U1 describes a device for capturing a hyperspectral image of an examination area of ​​a body. The device includes an input lens for generating an image in an image plane and a slit-shaped aperture in the image plane for masking out a slit-shaped region of the image. The light passing through the aperture is spread out by a dispersive element and recorded by a camera sensor. As a result, the camera sensor can record a plurality of spectra, each with an associated spatial coordinate, along the longitudinal direction of the slit-shaped aperture.The described device is further configured to record additional spectra along the longitudinal direction of the slit-shaped aperture in a direction different from the longitudinal direction of the slit-shaped aperture. The method underlying this disclosure for generating multispectral or hyperspectral images is also known as the so-called pushbroom method.

[0005] In addition to the pushbroom method, there are other methods for generating multispectral or hyperspectral images. In the so-called whiskbroom method, the examination area or object is scanned point by point, and a spectrum is obtained for each point. In contrast, the staring method acquires multiple images with the same spatial coordinates. Different spectral filters and / or illumination sources are used from image to image to resolve spectral information. Furthermore, there are methods in which a two-dimensional multicolor image is decomposed into several individual spectral images using suitable optical elements such as optical slicers, lenses, and prisms. These individual images are simultaneously acquired on different detectors or detector areas. This is sometimes referred to as the snapshot approach.

[0006] As described in DE 10 2020 105 458 A1, multispectral and hyperspectral imaging devices are particularly suitable as endoscopic imaging devices. In this context, multispectral and / or hyperspectral imaging is a fundamental field of application, for example, for diagnostics and for assessing the success or quality of a procedure.

[0007] The calculation of parameters based on multispectral or hyperspectral image data can be significantly influenced by even minor spectral changes. Deviations between a spectrum determined during imaging and the actual spectrum of the imaged object can therefore lead to inaccurate or incorrect results. Conventional systems therefore undergo comparatively complex and precise white balancing to achieve precise color calibration. Regular recalibration is required, for example, if the optics used are changed or a different light source is used. Recalibration may also be necessary if imaging is to be performed in a different medium than previously used. Furthermore, recalibration is required at regular intervals even if the hardware remains unchanged, because aging effects, contamination, etc. can affect the imaging.Therefore, a full white balance is often performed before every procedure. This is time-consuming and reduces the effective uptime of the system. Furthermore, calibration consumes the time and attention of trained personnel, which in turn reduces their availability for more complex procedures.

[0008] Based on the prior art, it is an object of the present invention to enable efficient color matching of a medical imaging device.

[0009] This object is achieved according to the invention by a medical

[0010] Imaging device with the features of claim 1, a medical system with the features of claim 14, a calibration object for a medical imaging device with the features of claim 21 and a method for color adjustment of a medical imaging device with the features of claim 22. Further developments of the invention can be found in the dependent claims.

[0011] According to the invention, a medical imaging device can comprise an imaging unit configured to perform multispectral and / or hyperspectral image acquisition, wherein the imaging unit is configured to acquire at least one intraoperative calibration image of a calibration object that can be introduced intraoperatively into a cavity while the calibration object is introduced into the cavity. Furthermore, the medical imaging device can comprise a calibration unit configured to perform a color adjustment for the image acquisition based on the calibration image, which includes an adjustment of a color calibration, in particular a white balance calibration. The medical imaging device can be, for example, an endoscope device, an exoscope device, and / or a microscope device.

[0012] In addition, a medical system can comprise a calibration object that can be introduced intraoperatively into a cavity, as well as a medical imaging device according to the invention.

[0013] The invention also relates to a calibration object for a medical imaging device according to the invention and / or for a medical system according to the invention.

[0014] Furthermore, a method for color adjustment of an imaging device configured to perform multispectral and / or hyperspectral image acquisition is proposed. This method may be a method for white light calibration. The method may comprise a step of intraoperatively introducing a calibration object into a cavity. Furthermore, the method may comprise capturing at least one intraoperative calibration image of the calibration object introduced intraoperatively into the cavity. Furthermore, the method may comprise performing a color adjustment for the image acquisition in accordance with the calibration image, wherein the color adjustment comprises adjusting a color calibration. This may, in particular, comprise adjusting a white light calibration. The features according to the invention allow for efficient color calibration of an imaging device.Color matching can be performed during surgery and while an imaging unit is in operation, eliminating the need to allocate time and resources to perform calibration prior to a procedure. A high degree of accuracy can also be achieved because any spectral shifts can be avoided by performing color matching, correction, or adjustment of color calibration directly in the imaging environment. As a result, parameters from multispectral or hyperspectral images can be obtained with high accuracy in a time-efficient manner.

[0015] The medical imaging device can be part of and / or comprise a medical imaging device and / or imaging instrument. The imaging device can be configured to record tissue parameters, images of wounds, images of body parts, etc. For example, the imaging device can be configured to image a surgical field. The imaging unit can comprise at least one optical system and at least one image acquisition sensor coupled to the optical system, which are configured to perform an image acquisition of an image region, generating spatially and spectrally resolved image data that includes both spatial and spectral information.

[0016] The imaging unit, and in particular the optics and / or the image acquisition sensor system, can be configured for multispectral and / or hyperspectral imaging, specifically for capturing and / or generating multispectral and / or hyperspectral image data. Multispectral imaging or multispectral image data can refer in particular to imaging in which at least two, in particular at least three, and in some cases at least five spectral bands can be and / or are captured independently of one another. Hyperspectral imaging or hyperspectral image data can refer in particular to imaging in which at least 20, at least 50, or even at least 100 spectral bands can be and / or are captured independently of one another.The imaging device may operate according to the pushbroom method and / or the whiskbroom method and / or the staring method and / or a snapshot principle.

[0017] In some embodiments, the imaging device comprises a white-light camera and / or sensor system for white-light image capture. The imaging device can be configured for white-light imaging in addition to spectrally resolved imaging. Separate optics and / or shared optics can be used for this purpose. The white-light imaging and the spectrally resolved imaging can be performed simultaneously or alternately, or at times simultaneously and at times sequentially.

[0018] In some embodiments, the imaging device comprises sensors for fluorescence imaging. The imaging device can be configured for fluorescence imaging in addition to spectrally resolved imaging and, if appropriate, in addition to white-light imaging. Separate optics and / or shared optics can be used for this purpose. The fluorescence imaging, if appropriate, the white-light imaging, and the spectrally resolved imaging can be performed simultaneously or alternately, or at times simultaneously and at times sequentially.

[0019] For some applications, it may be advantageous to be able to use a high spectral resolution. In these cases, hyperspectral imaging is a suitable option. This can be combined with white-light imaging and / or fluorescence imaging. This enables real-time observation via a white-light image and / or a fluorescence image, even if the acquisition of spectrally resolved image data is essentially only real-time, meaning that, for example, several seconds are required to create a spectrally resolved image.

[0020] For some applications, it may be advantageous to generate spectral image data in real time, or at least essentially in real time. This includes, for example, generating a spectrally resolved image in less than one second, or even multiple times per second. In this case, it may be appropriate to use multispectral imaging. A possibly lower spectral resolution is then offset by a higher frame rate. Depending on the application, it may be sufficient to consider only a few different spectral ranges and / or wavelengths, for example, two or three or four, or generally fewer than ten. In this case, additional white light imaging can optionally be dispensed with. Spectrally resolved image data that is acquired in real time, ordeliver several images per second, can also be used for surveillance purposes, whereby it is not necessarily necessary to create an image for a user to display, but the image data can also be processed in the background.

[0021] The calibration object can be configured to be imaged in a state introduced into the cavity, such that information can be determined based on an image of the calibration object in this state introduced into the cavity, on the basis of which information a color adjustment for the image acquisition can be carried out. The cavity can, for example, be a body cavity or be located in the body of a patient. The cavity can in particular be delimited by tissue and / or body parts. The calibration image is in particular recorded in a state in which the imaging unit is inserted into the cavity and / or is directed towards an interior of the cavity. For example, in the case of an endoscope device, the intraoperative calibration image on which the calibration object can be seen is recorded in a state in which a shaft of the endoscope device is inserted into the cavity.In other words, the intraoperative calibration image can be called an intracorporeal calibration image.

[0022] The color adjustment may include an adjustment of at least one parameter of a color calibration, in particular a white balance calibration. In some embodiments, the color adjustment may also include a complete color calibration. In some cases, the color adjustment may serve to correct a previously performed color calibration.

[0023] A color adjustment to be performed can be easily determined, especially if the calibration object comprises at least one color area with predetermined spectral properties. The color area can be homogeneously colored. The color area can be suitable for performing a white balance calibration. In some embodiments, the color area has an area of ​​at least 1 mm 2 , at least 10 mm 2 or even at least 1 cm 2The colored surface can, for example, be monochrome and / or white. In this context, "white" refers to the property of the colored surface to scatter illuminating light at least substantially independently of wavelength in a spectral range that includes at least visible light, for example a spectral range that extends from a wavelength of at least 300 nm, at least 350 nm, or at least 400 nm to a wavelength of at least 600 nm, at least 700 nm, or at least 800 nm. In particular, in the case of hyperspectral imaging, the range can extend at least from 500 nm to 1000 nm.

[0024] The calibration object may have a sterile and / or sterilized surface. In some embodiments, the medical system may include an outer packaging for the calibration object that encloses the calibration object in a sterile manner. The calibration object may be removable from the outer packaging before entering the cavity and capturing the calibration image.

[0025] The calibration object can be part of a medical instrument. This allows a calibration image to be acquired without significant loss of time, for example, if the instrument is already inserted into the cavity.

[0026] Furthermore, the at least one colored surface can be autoclave-resistant such that the spectral properties of the colored surface are at least substantially retained during autoclaving of the calibration object. For example, the spectral properties of the colored surface can remain substantially unchanged over at least ten, at least 20, at least 50, or even at least 100 autoclaving cycles. This allows the calibration object to be used multiple times. This configuration can be particularly advantageous if the calibration object is located on a reusable medical instrument and / or is part of such an instrument.

[0027] In some embodiments, the spectral properties of the color area emulate spectral properties of human and / or animal tissue.

[0028] In particular, an absorption spectrum and / or a reflection behavior and / or a scattering behavior of the colored area at least in the visible spectral range and / or in the near-infrared range corresponds at least substantially to an absorption spectrum and / or a reflection behavior and / or a scattering behavior of human and / or animal tissue. The tissue can be epithelial tissue, muscle tissue, nerve tissue and / or connective tissue. This allows a color adjustment with regard to relevant spectral ranges to be carried out particularly precisely. Under certain circumstances, additional weighting of spectral ranges in the calibration image can be omitted because it is known that the image of the calibration object reflects the spectral range in which tissue is to be imaged. Integration into existing instruments or instruments that are to be used anyway can be achieved in particular if the calibration object includes a label.This eliminates the need for any further adjustments to the instrument, which could otherwise affect its functionality. Furthermore, the calibration object can be easily identified in the calibration image. The label can include at least one character, such as letters or numbers. The label can include multiple characters in different colors, allowing the calibration object to be used for calibration in different spectral ranges.

[0029] In some embodiments, the calibration object can be designed for single use. For example, the calibration object can be inserted into the cavity once to capture the calibration image and then discarded. This makes it possible to provide a calibration object that can be manufactured cost-effectively and with low requirements regarding material quality and durability.

[0030] The color adjustment can be based on base calibration data that can relate to the calibration object, with the color calibration comprising at least one calculation based on both the calibration image and the base calibration data. This allows a high degree of accuracy to be achieved, since, starting from a precisely performed base calibration, comparatively straight adjustments must be made to perform the color adjustment. The base calibration data can include data sets for multiple configurations of the imaging device. These can be based, for example, on base calibrations performed with different optics, optical filters, interchangeable shafts, and / or in different illumination modes or imaging modes.In some cases, the base calibration data comprise at least one image with a large number of pixels, wherein a spectrum is available for the majority of the pixels and in particular for each pixel. When recording the calibration image, however, it can be provided that a spectrum is recorded only for some and / or for selected pixels of the calibration image. The spectra contained in the calibration image can then be compared with the spectra of the base calibration data assigned to the corresponding pixels. Approximations can then be calculated for further pixels using a deviation determined therefrom. Base calibration data can be generated efficiently and with little effort, in particular if the imaging unit is configured to record at least one base calibration image in a color-calibrated state. The base calibration data can be based at least partially on the base calibration image.The acquisition of the base calibration image is particularly performed extraoperatively and outside a body cavity. In other words, the base calibration image is acquired in a state in which the imaging unit is correctly color calibrated and, in particular, is subject to correct white balance. To acquire the base calibration image, the calibration object can, in particular, be imaged in an extraoperative state. The intraoperatively acquired calibration image of the calibration object can then be compared with the base calibration image.

[0031] The calibration unit can be configured to determine spectral information of an image region of the calibration image associated with the calibration object. This provides spectral information related to the calibration object. This information can be compared with the known or expected spectral properties of the calibration object, allowing color matching or correction to be performed with a high degree of accuracy.

[0032] In some embodiments, the calibration unit is configured to perform image recognition to identify the calibration object. This allows a high degree of automation to be achieved. The image recognition can detect a segmentation, based on which image regions of the calibration image can be determined in which at least parts of the calibration object are located. The image recognition can be based on information obtained from the base calibration image. In some embodiments, a user can specify specifications regarding recognition of the calibration object based on the base calibration image, which specifications are used by the calibration unit to recognize the calibration object in the calibration image. This allows a high degree of reliability in recognition of the calibration object to be achieved.

[0033] The detection of the calibration object can comprise detecting whether a calibration object is present in the image. This can be implemented, for example, using suitable Cl classification models. Furthermore, if the presence of a calibration object has been detected, the image detection can comprise placing the imaging unit, and in particular a camera of the imaging unit, into a calibration mode. This can comprise, for example, specifying a specific exposure rule, particularly in the case of a multispectral imaging unit, switching from a white light mode to a multispectral mode in which no image is displayed to the user, possibly at least temporarily, or moving to a suitable location in an imaging slit of a hyperspectral detection sensor system. Furthermore, the detection of the calibration object can comprise detecting pixels and / or image regions that belong to the calibration object.This can be achieved, for example, using suitable KL classification models. This allows the image regions relevant for calibration to be selected from the calibration image. Furthermore, spectral properties can be determined for these image regions and / or pixels.

[0034] Generally speaking, the imaging unit can be configured to automatically switch to a calibration mode upon detection of the calibration object. This achieves a high degree of operating reliability and ease of use.

[0035] The detection of the calibration object can be carried out particularly efficiently if the image recognition is based on multispectral and / or hyperspectral image data. In particular, the image recognition can be performed using the same image data that underlies the calibration image. In other words, the image recognition can include detection of the calibration object in the calibration image.

[0036] In some embodiments, the imaging unit is configured to perform white light imaging, wherein the image recognition is based on white light image data. This allows a high degree of accuracy in the recognition of the calibration object to be achieved. The calibration unit can be configured to match white light image data with the calibration image or to map them to one another. An image area found in a white light image in which the calibration object is located can then be recognized in the calibration image. This means that the calibration object is reliably and precisely recognized in the calibration image and an image area that is relevant for the color adjustment is used. It is advisable to use a calibration object with a distinctive shape and / or a distinctive color design that favors image recognition. For example, it can be a color pattern with defined areas that are easy to recognize and, if necessary,have distinguishable geometric shapes and / or colors. This allows the calibration object to be reliably detected even in the case of a poor white balance condition. A calibration image of high quality and relevant information content can be obtained in particular if the imaging unit is configured to acquire at least one test image of the calibration object. The calibration unit can be configured to create an analysis of the test image and to specify at least one image acquisition parameter for acquiring the calibration image in accordance with the analysis. The image acquisition parameter can, for example, relate to settings relating to multispectral and / or hyperspectral imaging during the acquisition of the calibration image.For example, in the case of a pushbroom camera, this can be used for targeted column images after determining the image area of ​​the calibration object based on the test image. The calibration image can then be targeted at these image areas, thereby achieving high spectral resolution and / or enabling fast image acquisition.

[0037] Information relevant for color adjustment can be determined in a targeted manner, particularly if the calibration unit is configured to disregard overexposed image areas and / or underexposed image areas of the calibration image for color adjustment. The detection of overexposed image areas can be performed, in particular, in addition to the detection of image areas in which the calibration object is located. Thus, the color adjustment can be based, in particular and preferably, only on those image areas in which the calibration object is correctly exposed.

[0038] The color adjustment can be performed precisely and / or adapted to the situation if the imaging unit is configured to acquire multiple intraoperative calibration images of the calibration object while the calibration object is inserted into the cavity. The color adjustment can be based on the multiple different calibration images. In some embodiments, for example, a combined image can be determined from multiple calibration images. This can, for example, obtain averaged spectral information. Alternatively or additionally, multiple calibration images can be acquired at time intervals, and a color adjustment can be performed for each one. The color adjustment can thus track a changing imaging situation.

[0039] The color adjustment may include spatial and / or temporal weighting of the multiple different calibration images. Multiple calibration images can thus be combined in a targeted manner, taking into account the relevant information and disregarding less relevant or irrelevant information. The weighting may depend on additionally available information, such as focus, image area, lighting situation, image acquisition parameters, etc., during the acquisition of the multiple calibration images. The weighting may additionally or alternatively include results from image analyses. For example, a self-learning system and / or a KL algorithm may be used to assess the quality, suitability, image area, relevance, or the like of calibration images.In addition, image errors in calibration images can be detected and corresponding calibration images can be sorted out or at least considered with a low weighting.

[0040] In some embodiments, the medical imaging device may further comprise an assessment unit configured to compare a degree of color calibration adjustment with a threshold value and to issue a recommendation for external and / or extraoperative recalibration if the degree of color calibration adjustment exceeds the threshold value. This allows for automated detection of whether recalibration is necessary, for example, to perform a baseline calibration and / or acquire a baseline calibration image. In the case of minor deviations, it may be sufficient to perform the color adjustment according to the calibration image, thereby enabling the required spectral correction. For larger deviations, however, it may be necessary to externally recalibrate the medical imaging device.

[0041] In some embodiments, the expected quality of the color calibration can be presented to a user. If, for example, there is no instrument or, more generally, no calibration object in the image area at the start of an intervention, the expected quality of the color calibration is low. Under certain circumstances, a hyperspectral image can even be automatically deactivated if it is expected that it cannot be carried out with reasonable quality. Once a calibration object, in particular an instrument with one, is recognizable in the image, a color calibration can be performed. User safety is then high because reliable image data can be generated. After the instrument / calibration object is removed, safety can decrease again, particularly due to the slow heating of the camera and light source.In this case, it can be provided that if a predetermined and / or predefinable time limit is exceeded, a recording function is automatically deactivated again in order to prevent faulty images from being recorded. The devices and systems according to the invention, as well as the methods according to the invention, are not intended to be limited to the application and embodiment described above. In particular, in order to fulfill a functionality described herein, they can have a number of individual elements, components and units, as well as method steps, that differs from the number stated herein. Furthermore, in the value ranges specified in this disclosure, values ​​lying within the stated limits are also to be considered disclosed and can be used arbitrarily.

[0042] It is particularly noted that all features and properties described with reference to a device, as well as procedures, are transferable to methods and applicable within the meaning of the invention and are considered to be included in the disclosure. The same applies in reverse. This means that structural features mentioned with reference to methods, i.e., features related to the device, can also be considered, claimed, and included in the disclosure within the scope of the device claims.

[0043] The present invention is described below by way of example with reference to the accompanying figures. The drawings, the description, and the claims contain numerous features in combination. Those skilled in the art will expediently consider the features individually and use them in meaningful combination within the scope of the claims.

[0044] If there is more than one instance of a particular object, only one of them may be provided with a reference symbol in the figures and in the description. The description of this instance can be transferred accordingly to the other instances of the object. If objects are named using numerical terms, such as first, second, third object, etc., these serve to name and / or assign objects. Accordingly, for example, a first object and a third object, but not a second object, may be included. However, a number and / or sequence of objects could also be derived using numerical terms.

[0045] They show:

[0046] Fig. 1 is a schematic representation of a medical system with a medical imaging device; Fig. 2 is a schematic representation of a calibration image with an image area showing a calibration object;

[0047] Fig. 3 is a schematic spectrum of the image area of ​​the calibration image showing the calibration object;

[0048] Fig. 4 is a schematic representation of a base calibration image with an image area showing a calibration object;

[0049] Fig. 5 is a schematic spectrum of the image area of ​​the base calibration image showing the calibration object;

[0050] Fig. 6 is a schematic representation of a calibration image with overexposed and underexposed image areas;

[0051] Fig. 7 is a schematic representation of a test image of the calibration object;

[0052] Fig. 8 is a schematic representation of a calibration image showing another

[0053] Calibration object shows;

[0054] Fig. 9 is a schematic representation of a calibration image showing yet another calibration object;

[0055] Fig. 10 is a schematic perspective view of another medical system; and

[0056] Fig. 11 is a schematic flow diagram of a method for color adjustment of a medical imaging device.

[0057] Fig. 1 shows a schematic representation of a medical system 34 with a medical imaging device 10. In the exemplary case shown, the imaging device 10 is an endoscopic imaging device, specifically an endoscope device. Alternatively, the imaging device 10 could be an exoscopic, a microscopic, or a macroscopic imaging device. The imaging device 10 is provided, for example, for examining a cavity. The imaging device 10 comprises an imaging unit 12 configured to perform multispectral and / or hyperspectral image acquisition. The imaging unit 12 comprises at least one optical system and image acquisition sensors, which are not shown in Fig. 1. These can be at least partially integrated into a camera unit 45.

[0058] In the case of a multispectral imaging unit 12, the optics and image acquisition sensors can optionally comprise optical filters and / or image sensors operating in different spectral ranges. In the case of a hyperspectral imaging unit 12, the optics and image acquisition sensors can operate, for example, according to the pushbroom method or the whiskbroom method.

[0059] In addition to multispectral and / or hyperspectral image acquisition, the imaging unit 12 can be configured to capture white-light images. This can occur parallel or sequentially with the multispectral and / or hyperspectral imaging. For example, a real-time white-light image can be generated. Multispectral images and / or hyperspectral images and / or fluorescence images can then optionally be superimposed on the white-light image, displayed alternately with it, and / or in parallel.

[0060] In the illustrated case, the imaging device 10 comprises an imaging instrument 42, for example, an endoscope. Parts of the optics and / or the image acquisition sensor system can also be integrated into the imaging instrument 42. For example, in some embodiments, a camera ("Tipcam") arranged at a distal shaft end of the imaging instrument can be provided.

[0061] The imaging device 10 further includes an illumination unit 44. The illumination unit 44 can be connectable and / or connected to the imaging instrument and can provide illumination light for image acquisition. In some embodiments, the illumination unit 44 can be multimodal and generate, for example, white light, excitation light for fluorescence imaging, and / or excitation light for multispectral and / or hyperspectral imaging.

[0062] The medical system 34 comprises a medical instrument 38. The medical instrument 38 can be configured to be inserted into a patient's cavity for therapeutic and / or diagnostic purposes. By way of example only, the medical instrument 38 is a tissue coagulation forceps. However, it is understood that any medical instruments are possible according to the invention. These can be inserted into a cavity, in particular an opened cavity, and used therein through a working channel of an endoscope and / or through an access device and / or through an incision and / or otherwise.

[0063] The medical system 34 and / or imaging device 10 further includes a display unit 46. The display unit 46 is configured to display representations to a user based, for example, on acquired image data. For example, a user can view a site via the display unit 46 to observe the medical instrument 38, assess tissue to be treated and / or examined, verify the correct arrangement and / or orientation of the imaging instrument 42, or the like.

[0064] Reference is made below to Fig. 2. The imaging unit 12 is configured to record at least one calibration image 14 of a calibration object 18 that can be introduced intraoperatively into a cavity 16. In the illustrated case, the calibration image 14 is recorded in a state in which the imaging device 10 images the interior of the cavity 16 and, for example, is at least partially introduced into it and / or focused into it. The calibration object 18 is introduced into the cavity 16. The calibration object 18 serves to carry out a color adjustment for the image acquisition, by means of which a color calibration can be adjusted. In the present case, this allows a white balance calibration to be adjusted and / or carried out. The calibration image 14 is recorded under real operating conditions. As a result, the color adjustment is based on conditions that actually occur during image acquisition.

[0065] To perform the color adjustment, the imaging device 10 includes a calibration unit 20, which is shown schematically in Fig. 1. The calibration unit 20 can be configured as a physically separate unit. It can also be integrated into a controller and / or control unit of the imaging device 10.

[0066] In the case of Fig. 2, the calibration object 18 is part of the medical instrument 38. Specifically, the calibration object 18 can, for example, comprise a label 40.

[0067] The label 40 includes, for example, a logo and / or characters and / or symbols, such as a company name and logo or a type designation of the instrument. The calibration object 18 can thus be designed in such a way that it is not immediately recognizable as such by uninvolved persons.

[0068] As shown in Fig. 2, the calibration image 14 contains an image region 26 that is assigned to the calibration object 18. The calibration unit 20 is configured to determine such an image region 26. In the exemplary case, this is done automatically using image segmentation. The calibration object can include symbols and / or markings that support recognition. For example, a partial or complete frame and / or at least one boundary line or the like can be present. Such boundary elements can be easily and reliably recognized using common methods. Based on this, the image region 26 within which the calibration object 18 is located can then be determined.

[0069] The image area 26 shown in Fig. 2 is to be understood schematically. If the calibration object comprises, for example, one or more characters, logos, and / or symbols, the image area 26 can be delimited by edges and / or a contour thereof. The image area 26 can, for example, only comprise the characters, logos, and / or symbols. As a result, the image area 26 is limited to sections in which the calibration object 18 has known and defined optical properties, thus reliably enabling color matching.

[0070] The image recognition of the calibration object 18 can be based on multispectral and / or hyperspectral image data. In particular, the image recognition can be performed directly and, in particular, exclusively based on the calibration image 14. Alternatively or additionally, an associated white-light image can also be analyzed. For example, a position and a covered image area of ​​the calibration object 18 can be recognized in the white-light image, and a corresponding position / image area can be determined in the calibration image 14.

[0071] Image recognition can be performed using a KI algorithm based on image segmentation. The algorithm can be suitably trained, for example, using multiple images of the calibration object 18 and / or similar or identically designed objects. In some embodiments, the training is based on recognizing different lettering, logos, and symbols. The calibration unit 20 can thereby be configured to recognize different calibration objects 18, in particular calibration objects 18 that include different labels. As a result, the imaging device 10 can be color-adjusted intraoperatively when used with different calibration objects 18 and in particular with different instruments that include different calibration objects 18.It is then not necessary to introduce a specific calibration object into the cavity 16 specifically for the color adjustment, but the color adjustment can be carried out within the scope of any interventions.

[0072] In some embodiments, the imaging unit 18 can be automatically placed into a calibration mode upon detection of a calibration object 18. In calibration mode, for example, a white light display can be temporarily interrupted for the user. Furthermore, certain predefined image acquisition parameters can be automatically set in calibration mode, ensuring that a calibration is performed correctly.

[0073] The calibration object 18 comprises at least one colored area 36 with predetermined spectral properties. In the illustrated case, the inscription 40 forms the colored area 36. The colored area 36 can be homogeneously colored, for example, white, gray, or with a color that replicates spectral properties of human and / or animal tissue. In some embodiments, the colored area 36 can have an absorption spectrum in the visible range and / or the near-infrared range that is similar to human and / or animal tissue. This allows a color adjustment to be based on a spectrum that is oriented to the area of ​​application of the imaging device in the relevant spectral ranges.

[0074] In this case, the colored surface 36 is autoclave-resistant. Its spectral properties are therefore retained even upon repeated autoclaving. The calibration object 18 is thus autoclaved and thus prepared for reuse when the medical instrument is autoclaved. Autoclave resistance can be achieved by using an autoclave-resistant paint and / or a coating. A suitable coating can, for example, be based on a lacquer and / or be vapor-deposited and / or sputtered. Metal oxide coatings are particularly suitable, as they are translucent in the visible and near-infrared range, yet also withstand the harsh conditions of autoclaving.

[0075] Fig. 3 shows a schematic spectrum 48 of the image area 26 of the calibration image 14, which shows the calibration object 18. In the illustrated case, this is an intensity spectrum plotted against a wavelength. The covered spectral range extends, for example, at least from 400 nm to 750 nm, from 400 nm to 900 nm, or even further to smaller and / or larger wavelengths. The spectrum can extend over a wavelength range that corresponds to the wavelength range covered by the imaging unit 12. In the present case, this spectrum 48 is based on an averaging over pixels of the image area 26. The spectrum 48 is alternatively or additionally based on a temporal averaging. Depending on the configuration of the imaging unit 12, the spectrum 48 can be based on hyperspectral or multispectral image data. The spectrum shown in Fig. 3 is to be understood purely as an example.In particular, in the case of a multispectral imaging unit, it may also be a spectrum that comprises only a few points, in particular one point per multispectral support point.

[0076] The calibration unit 20 is configured to first determine the image area 26 and then to determine spectral information of the image area 26. This is, in particular, the schematically illustrated spectrum 48.

[0077] Based on the spectral information thus determined, the calibration unit 20 performs a color adjustment. The color adjustment is based on base calibration data. The base calibration data relates to the calibration object 18. Based on this, the calibration unit performs at least one calculation based on the base calibration data and on the calibration image 14, in particular on the spectral information determined with respect to the image area 26.

[0078] The basic calibration data may comprise a known spectrum of the calibration object 18, in particular of its at least one color area 36. The basic calibration data may therefore be stored and / or capable of being stored in the calibration unit 20.

[0079] In the present case, the base calibration data is based on the acquisition of at least one base calibration image 22. One such image is shown as an example in Fig. 4. The imaging unit 12 is configured to acquire the base calibration image 22 in a color-calibrated state. For this purpose, the imaging unit 12 is calibrated externally so that it has a correct white balance calibration. Then, at least one base calibration image 22 of a base calibration object 19 is acquired. The base calibration object 19 can be, for example, a neutral-colored screen and / or a neutral-colored color balance card, for example a gray card, a white surface, or the like. As shown in Fig. 4, the base calibration object 19 can comprise a target marking in a known manner to enable correct alignment.However, a completely uniform surface can also be imaged, which has the advantage that a spectrum can be obtained for each pixel. If a target marker is used, the relevant pixels can be masked out. Alternatively, the spectra corresponding to the pixels of the target marker can be replaced by spectra of nearby pixels and / or interpolated from these.

[0080] The calibration unit 20 is further configured to determine spectral information of the image area 50. From this, a spectrum 52 of the calibration object 18 can be obtained, as shown by way of example in Fig. 5. Due to the external color calibration of the imaging unit 12 prior to the acquisition of the base calibration image 22, the spectrum 52, or generally a spectral information determined from the base calibration image 22, is spectrally unadulterated. A calculation underlying the color adjustment can thus be performed by comparing spectral information determined intraoperatively, such as in particular the spectrum 48, with spectral information, in particular the spectrum 52, based on the base calibration image 22.

[0081] A color adjustment is performed using spectra 48, 52, for example, as follows. Let w0 denote spectrum 52. It is assumed that this corresponds to the spectrum with correct white balance calibration. Furthermore, Wi denotes spectrum 48. This thus corresponds to the intraoperative measurement of calibration object 18, which may yield a color deviation and necessitate a color adjustment. The color adjustment is then defined by the quotient w0 / wi, i.e., acquired images can be corrected using w0 / wi.

[0082] In addition, a black balance correction can also be performed analogously to the procedure described herein. This can be included in the color adjustment. A black level b can be determined by taking an image in the absence of illumination. Furthermore, it can be provided that the calibration object 18 has at least one black color area 36 that can be recognized in the calibration image 14. If a black level is taken into account, the color adjustment factor is determined as (w0-b) / (wi-b).

[0083] Fig. 6 shows a schematic representation of a calibration image 53 with an overexposed image area 28 and an underexposed image area 30. These image areas 28, 30 can influence spectral evaluations of the calibration image 53. In the example shown, they also overlap the image area 26 in which the calibration object 18 is imaged, so that the latter cannot be fully recognized. If the entire image area 26 were used as the basis for the color adjustment, the overexposed image area 28 and / or the underexposed image area 30 could affect the color adjustment. In the present case, the calibration unit 20 is configured to recognize overexposed image areas 28 and underexposed image areas 30 and to disregard them during the color adjustment. If necessary.Thus, only a part of the image area 26 is used and thus only a section of the calibration object 18 that is imaged correctly and / or with sufficient quality and / or accuracy is taken into account.

[0084] In some embodiments, the imaging unit can be configured to acquire at least one test image 24 of the calibration object 18. Such a test image 24 is shown as an example in Fig. 7. Generally speaking, the test image 24 can be used to verify the suitability of a selected image setup for performing an intraoperative calibration. Based on the test image 24, it can be assessed whether a calibration image 14 acquired in the next step can provide suitable spectral information.

[0085] In the present example, the calibration unit 20 is also configured to perform an analysis of the test image 24 and to specify at least one image acquisition parameter for acquiring the calibration image 14 based on the analysis. The test image 24 is thus used to adapt and, if necessary, optimize parameters for acquiring the calibration image 14.

[0086] The analysis of the test image 24 may include determining an image area 54 in which the calibration object 18 is located. The image area 54 may be selected according to the adjustability of the imaging unit 12. In the example, it differs from the image area 26 described above, which is primarily or exclusively based on the extent and contour of the calibration object 18. In the case of a hyperspectral imaging unit 10 with a pushbroom camera, the image area 54 is used to adjust parameters for column images such that spatially and spectrally resolved image data is acquired only for the image area 54. This can then be done in a short time and / or with high accuracy. For any imaging units 12, it may also be provided that an exposure and / or a focus and / or other image acquisition parameters are optimized solely with respect to the image area 54.For example, overexposure and / or underexposure in areas outside of image area 54 may be accepted. If a calibration image 14 is then acquired based on the analysis of test image 24, this calibration image is optimized to ensure reliable data can be obtained, particularly for image area 54.

[0087] Furthermore, the calibration unit 20 is configured to acquire multiple intraoperative calibration images 14 of the calibration object 18 while the calibration object 18 is inserted into the cavity 16. The color adjustment is then based on multiple different calibration images 14. These can show identical and / or different motifs. For example, multiple identical calibration images 14 can be acquired consecutively to compensate for image errors. Furthermore, multiple calibration images 14 can be acquired and analyzed with different acquisition parameters such as illumination, exposure time, focus, etc. Based on the individual analyses, the color adjustment can be performed by averaging, weighting, combining, balancing, etc.Alternatively or additionally, several calibration images 14, which were taken with matching parameters and from the same subject, can first be superimposed to form a single image, which is then subjected to analysis.

[0088] Generally speaking, the color adjustment can comprise a spatial and / or temporal weighting of the multiple different calibration images 14. This weighting can also refer only to selected image areas of the different calibration images 14. For example, for each calibration image 14, the image area 26 associated with the calibration object 18 can first be determined. The further analysis to determine the color adjustment is then based on the determined image areas 26.

[0089] Analogously, a plurality of base calibration images 22 and / or a plurality of test images 24 can be acquired, analyzed, and optionally weighted to obtain the base calibration data or to determine suitable parameters for acquiring one or more calibration images 14. Referring again to Fig. 1, the imaging device 10 further comprises an assessment unit 32 configured to compare a degree of color calibration adjustment with a threshold value and to output a recommendation for external recalibration if the degree of color calibration adjustment exceeds the threshold value. The recommendation can be output, for example, via the display unit 46. If only minor adjustments that do not exceed the threshold value are required, no external recalibration is necessary.

[0090] Fig. 8 shows a schematic representation of a calibration image 14', which shows another calibration object 18'. The calibration object 18' has several color areas 36' with predetermined spectral properties. These are represented as homogeneous color areas with an extension of a few mm. 2 formed, for example, as rectangles, squares, circles, polygons, ellipses, or other simple geometric figures. The color areas 36' can be easily and reliably located based on image segmentation. They can also have different colors, so that multiple spectra can be obtained that can serve as the basis for color matching. The calibration object 18' can be part of a medical instrument 38', analogous to the calibration object 18 described above.

[0091] Fig. 9 shows a schematic representation of a calibration image 14", which shows yet another calibration object 18". The calibration object 18" has at least one color area 36' with predetermined spectral properties. The calibration object 18" is intended for single use. The calibration object 18" is, for example, part of a calibration rod 56. This can be packaged sterile. If a color adjustment is to be performed, the calibration object 18" and / or the calibration rod 56 is unpacked and inserted into the cavity 16. Since the calibration object 18" is thus unused before use, its spectral properties are known very precisely and are not impaired by previous cleaning cycles, autoclaving and / or contamination.

[0092] It is understood that in other embodiments, a calibration object can also be used which is designed as a separate element and is not part of a medical instrument, but can nevertheless be intended for repeated use. Fig. 10 shows a schematic perspective illustration of another medical system 34'" with an alternative imaging device 10'". The alternative imaging device 10' basically operates analogously to the imaging device 10 described above. The alternative imaging device 10'" is designed as an exoscopic imaging device. It is directed, for example, into an opened cavity 16'" for multispectral and / or hyperspectral imaging. During color matching, a calibration object 18'" is introduced into the cavity 16'", as described above.

[0093] Fig. 11 shows a schematic flow diagram of a method for color matching of a medical imaging device. The sequence of the methods is also apparent from the above explanations. By way of example, the method is carried out with the imaging device 10.

[0094] The method comprises a step S1 of intraoperatively inserting a calibration object 18 into a cavity 16.

[0095] Furthermore, the method comprises a step S2 of recording at least one intraoperative calibration image 14 of the calibration object 18 introduced intraoperatively into the cavity 16.

[0096] In addition, the method comprises a step S3 of performing a color adjustment for the image acquisition according to the calibration image 14, wherein the color adjustment comprises an adjustment of a color calibration, in particular a white balance calibration.

[0097] List of reference symbols

[0098] 10 Imaging device

[0099] 12 Imaging Unit

[0100] 14 Calibration image

[0101] 16 cavity

[0102] 18 Calibration object

[0103] 19 Base calibration object

[0104] 20 Calibration unit

[0105] 22 Base calibration image

[0106] 24 test image

[0107] 26 Image area

[0108] 28 image area

[0109] 30 area

[0110] 32 assessment unit

[0111] 34 medical system

[0112] 36 color areas

[0113] 38 medical instrument

[0114] 40 labeling

[0115] 42 Imaging instrument

[0116] 44 lighting unit

[0117] 45 Camera unit

[0118] 46 display unit

[0119] 48 Spectrum

[0120] 52 Spectrum

[0121] 53 Calibration image

[0122] 54 image area

[0123] 56 calibration strips

Claims

Claims 1. A medical imaging device (10), in particular an endoscope device, exoscope device, and / or microscope device, comprising: an imaging unit (12) configured to perform multispectral and / or hyperspectral image acquisition, wherein the imaging unit (12) is configured to record at least one intraoperative calibration image (14) of a calibration object (18) that can be introduced intraoperatively into a cavity (16) while the calibration object (18) is introduced into the cavity (16); and a calibration unit (20) configured to perform a color adjustment for the image acquisition in accordance with the calibration image (14), which includes an adjustment of a color calibration, in particular a white balance calibration.

2. The medical imaging device (10) according to claim 1, wherein the color adjustment is based on base calibration data relating to the calibration object (18), and wherein the color calibration comprises at least one calculation based on both the calibration image (14) and the base calibration data.

3. The medical imaging device (10) according to claim 2, wherein the imaging unit (12) is configured to acquire at least one base calibration image (22) in a color-calibrated state, and wherein the base calibration data is based on the base calibration image (22).

4. Medical imaging device (10) according to one of the preceding claims, wherein the calibration unit (20) is configured to perform image recognition to identify the calibration object (18).

5. Medical imaging device (10) according to claim 4, wherein the image recognition is based on multispectral and / or hyperspectral image data.

6. Medical imaging device (10) according to claim 4 or 5, wherein the imaging unit (12) is configured to perform white light imaging and wherein the image recognition is based on white light image data.

7. Medical imaging device (10) according to one of claims 4 to 6, wherein the imaging unit (12) is configured to acquire at least one test image (24) of the calibration object (18), and wherein the calibration unit (20) is configured to create an analysis of the test image (24) and to specify at least one image acquisition parameter for acquiring the calibration image (14) in accordance with the analysis.

8. Medical imaging device (10) according to one of claims 4 to 7, wherein the imaging unit (12) is configured to automatically switch to a calibration mode in the event of detection of the calibration object (18).

9. Medical imaging device (10) according to one of the preceding claims, wherein the calibration unit (20) is configured to determine spectral information of an image region (26) of the calibration image (14) that is associated with the calibration object (18).

10. Medical imaging device (10) according to one of the preceding claims, wherein the calibration unit (20) is configured to disregard overexposed image areas (28) and / or underexposed image areas (30) of the calibration image (14) for the color adjustment.

11. Medical imaging device (10) according to one of the preceding claims, wherein the imaging unit (12) is configured to acquire a plurality of intraoperative calibration images (14) of the calibration object (18) while the calibration object (18) is introduced into the cavity (16), and wherein the color adjustment is based on the plurality of different calibration images (14).

12. The medical imaging device (10) of claim 11, wherein the color adjustment comprises a spatial and / or temporal weighting of the plurality of different calibration images.

13. The medical imaging device (10) according to any one of the preceding claims, further comprising an assessment unit (32) configured to compare a degree of adjustment of the color calibration with a threshold value and to issue a recommendation for external recalibration if the degree of adjustment of the color calibration exceeds the threshold value.

14. A medical system (34) comprising: a calibration object (18) that can be introduced intraoperatively into a cavity (16); and a medical imaging device (10) according to any one of the preceding claims.

15. Medical system (34) according to claim 14, wherein the calibration object (18) comprises at least one color surface (36) with predetermined spectral properties.

16. The medical system (34) according to claim 15, wherein the at least one color surface (36) is autoclave-resistant such that the spectral properties of the color surface (36) are retained during autoclaving of the calibration object (18).

17. The medical system (34) according to claim 15 or 16, wherein the spectral properties of the color area (36) simulate spectral properties of human and / or animal tissue.

18. Medical system (34) according to one of claims 14 to 17, wherein the calibration object (18) is part of a medical instrument (38).

19. Medical system (34) according to one of claims 14 to 18, wherein the calibration object (18) comprises a label (40).

20. Medical system (34) according to one of claims 14 to 19, wherein the calibration object (18) is intended for single use.

21. Calibration object (18) for a medical imaging device (ten) according to one of claims 1 to 13 and / or for a medical system (34) according to one of claims 14 to 20.

22. Method for color adaptation, in particular for white light calibration, of a medical imaging device (10), in particular according to one of claims 1 to 13, wherein the imaging device (10) is configured to perform multispectral and / or hyperspectral image acquisition, comprising: intraoperatively introducing a calibration object (18) into a cavity (16); recording at least one intraoperative calibration image (14) of the calibration object (18) introduced intraoperatively into the cavity (16); and performing a color adaptation for the image acquisition in accordance with the calibration image (14), wherein the color adaptation comprises an adaptation of a color calibration, in particular a white balance calibration.

Images