Imaging device, system, and imaging method

Abstract

The present invention relates to an imaging device (100), in particular an endoscopic imaging device, comprising: an image capturing unit (102), which is designed to map an object region and has a first capturing strand (104) which is designed to capture reference images (106) of the object region, and a second capturing strand (108) which is designed to capture spectral images of the object region; and a control unit (110), which is designed to carry out the following steps: - capturing an image stream of reference images (106) by means of the first capturing strand; - capturing spectral images by means of the second capturing strand (108) at the same time as capturing the image stream of reference images (106); and - generating a representation (112) of the image stream, wherein a sequence image (114) of the object region is superimposed on the representation (112), wherein the sequence image (114) is formed on the basis of the spectral images by the spectral images being locally assigned to the representation (112) of the image stream and by spectral representation parts (116) relating to the spectral images being sequentially added to the sequence image (114) such that the sequence image (115) sweeps over the representation (112) of the image stream.

Description

[0001] Imaging device, system and method for imaging

[0002] The present invention relates to an imaging device, a system, an imaging method, a program code and a computer program product.

[0003] Medical imaging devices, such as endoscopic or exoscopic devices, that generate multispectral or hyperspectral images are known from the prior art. In addition to two spatial dimensions, like a conventional camera image, multispectral or hyperspectral images have a spectral dimension. This spectral dimension comprises several spectral bands (wavelength bands). Multispectral and hyperspectral images differ primarily in the number and width of their spectral bands.

[0004] Several imaging devices for generating such multispectral or hyperspectral images are known, particularly in the context of medical applications. For example, DE 20 2014 010 558 U1 describes a device for acquiring a hyperspectral image of an examination area of ​​a body. The device comprises an input lens for generating an image in an image plane and a slit-shaped aperture in the image plane for blocking out a slit-shaped area of ​​the image. The light passing through the aperture is dispersed by means of a dispersive element and captured by a camera sensor. This allows the camera sensor to capture a multitude of spectra, each with an associated spatial coordinate, along the longitudinal direction of the slit-shaped aperture.The described device is further configured to acquire additional spectra along the longitudinal direction of the slit-shaped aperture in a direction different from that direction. The method underlying this disclosure for generating multispectral or hyperspectral images is also known as the pushbroom method.

[0005] Besides the pushbroom method, there are other techniques for generating multispectral or hyperspectral images. In the so-called whiskbroom method, the area under investigation or object is scanned point by point, and a spectrum is acquired for each point. In contrast, the staring method involves acquiring multiple images with the same spatial coordinates.

[0006] In the article “A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract” by Jonghee Yoon et al., published in Nature Communications 10, 1902 (2019), another method for generating hyperspectral images using an endoscope is described. In this method, a wide-field image and an inherently coregistered hyperspectral line scan are simultaneously acquired during a natural, freehand movement of the endoscope by the user. Similar to the pushbroom technique, the line scan comprises a spatial and a spectral dimension. However, unlike the pushbroom technique, the natural, freehand movement of the endoscope is used to acquire further hyperspectral line scans along the direction of movement, thus generating a hyperspectral image with two spatial dimensions and one spectral dimension.To display the hyperspectral image, spectral representation components are sequentially added to a representation based on line scans during its generation. This representation of hyperspectral data is displayed to the user separately from a representation based on wide-field images.

[0007] The separate display of hyperspectral data and wide-field images makes it impossible for the user to quickly and intuitively correlate features of the two representations spatially. This is especially true because the display is built sequentially based on line scans, preventing spatial mapping at an early stage of hyperspectral image generation. The display simply doesn't include enough pixels in two spatial dimensions. Therefore, if the user is performing an endoscopic examination, for example, to hyperspectrally image an abnormal tissue area within a cavity, they must manually correlate conspicuous areas in the hyperspectral data display with anatomical features in the wide-field images to determine precisely which tissue area exhibits which spectral properties. This leads to inaccuracies, misattributions, and an unsatisfactory user experience.

[0008] Based on the prior art, the invention aims to achieve an intuitive acquisition and presentation of multi- and / or hyperspectral data for the user.

[0009] The object of the invention is achieved by imaging devices, in particular endoscopic imaging devices, systems, methods for imaging, in particular endoscopic imaging methods, a program code and a computer program product as described herein and defined in the claims.

[0010] The invention provides for an imaging device, in particular an endoscopic imaging device. The imaging device comprises an image acquisition unit configured to image an object area. The image acquisition unit comprises a first acquisition line configured to acquire reference images of the object area and a second acquisition line configured to acquire spectral images of the object area. Furthermore, the imaging device comprises a control unit. The control unit is configured to perform the steps of acquiring a stream of reference images using the first acquisition line, acquiring spectral images using the second acquisition line simultaneously with acquiring the stream of reference images, and generating a display of the image stream.When generating the image stream, a sequence image of the object area is superimposed on the display. The sequence image is created from the spectral images by spatially assigning the spectral images to the image stream display and sequentially adding spectral representation components based on the spectral images to the sequence image in such a way that the sequence image sweeps over the image stream display.

[0011] Furthermore, the invention provides for an imaging device, in particular an endoscopic imaging device. The imaging device comprises an image acquisition unit configured to image an object area. The image acquisition unit comprises a first acquisition line configured to acquire reference images of the object area and a second acquisition line configured to acquire spectral images of the object area. The image acquisition unit is also configured to acquire the reference images with a first timestamp and the spectral images with a second timestamp. The imaging device further comprises a control unit.The control unit is configured to simultaneously perform the acquisition of a stream of reference images using the first acquisition line and the acquisition of spectral images using the second acquisition line. Furthermore, the control unit is configured to determine the temporal assignment of at least one spectral image to a reference image in the image stream by comparing the corresponding first timestamp of the respective spectral image with the corresponding second timestamp of the respective reference image. It is also configured to generate a sequence image of the object area based on spectral images by sequentially adding spectral representation components based on the spectral images to the sequence image according to the temporal assignment.

[0012] These features enable intuitive acquisition and display of multi- and / or hyperspectral data for the user. In particular, the data can be presented in such a way that it can be easily and intuitively mapped to specific anatomical locations. This can improve the usability and applicability of multi- and hyperspectral imaging. When the user examines prominent anatomical areas with the imaging device, in some embodiments the sequence image is superimposed on a video based on the reference images. The video can be displayed and shown to the user like a conventional endoscopic video. The sequence image can be spatially mapped to the reference images, allowing the user to intuitively and quickly map the spectral data to anatomical locations and sub-areas of the video display. Swiping over the video or...The representation of the image stream is achieved through local assignment.

[0013] By sweeping the endoscope over the image, a superimposed image of the sequence is built up on the video display before the user's eyes during image acquisition. The display of the multi- and / or hyperspectral data therefore appears to the user like a video. In contrast, with the conventional pushbroom method, the user must hold the endoscope still relative to an anatomical location and initiate a recording process. During the recording process, the area to be imaged is scanned, and subsequently, a display of the recorded hyperspectral data is generated. The present invention enables the user to move an endoscope exploratorily within a patient's cavity, with the sequence image building up similarly to a video. The hyperspectral data can be immediately assigned to the endoscopic video.If the user discovers an anatomical area of ​​interest, they can move the endoscope over this area and thus capture a complete hyperspectral image of this area.

[0014] The inventors recognized that a good overlay display, in which there is virtually no spatial offset between the sequence image and the video, improves the applicability of multi- or hyperspectral imaging. However, a temporal offset can occur during the acquisition of the spectral images and the reference images. This offset can be problematic if the sequence image is intended to overlap the image stream. In such an overlay display, the temporal offset can lead to a spatial misalignment of the two images, causing the overlay to appear blurry. However, if the spectral images and the reference images are acquired with timestamps, they can be temporally aligned, enabling a spatial correction of the sequence image. The spectral images can then be sequentially assigned to the sequence image according to their temporal alignment, thus reducing the spatial offset.This enables the creation of a spatially precise overlay representation, thereby improving its use as an information source for the user.

[0015] The imaging device can be microscopic, macroscopic, and / or exoscopic. The imaging device can be configured as a microscope, macroscope, and / or exoscope and / or comprise such a microscope. In some embodiments, the imaging device can be an endoscopic imaging device. The imaging device can be an endoscope device. It can comprise an endoscope and / or an endoscope system and / or be configured as such and / or form at least a part, and preferably at least a major part and / or principal component, of an endoscope and / or an endoscope system. "At least a major part" can mean at least 55%, preferably at least 65%, more preferably at least 75%, particularly preferably at least 85%, and most preferably at least 95%, particularly with respect to a volume and / or mass of an object.

[0016] In some embodiments, the imaging device is configured to be insertable into a cavity for inspection and / or observation, for example, into an artificial and / or natural cavity, such as the interior of a body, a body organ, tissue, or the like. The imaging device may also be configured to be insertable into a housing, casing, shaft, pipe, or other structure, particularly artificial ones, for inspection and / or observation.

[0017] The imaging device, and in particular the image acquisition unit, can be configured for multispectral and / or hyperspectral imaging, specifically for acquiring and / or generating multispectral and / or hyperspectral image data. Multispectral imaging or multispectral image data can refer in particular to imaging in which at least two spectral bands can be acquired and / or are acquired independently of one another. Hyperspectral imaging or hyperspectral image data can refer in particular to imaging in which at least 20, at least 50, or even at least 100 spectral bands can be acquired and / or are acquired independently of one another.

[0018] The imaging device, in particular the endoscopic imaging device, may have at least one proximal section, one distal section, and / or one intermediate section. The distal section is specifically designed to be inserted into and / or located within a cavity to be examined during operation, such as during diagnostic and / or therapeutic procedures. The proximal section is specifically designed to be positioned outside the cavity to be examined during operation, such as during diagnostic and / or therapeutic procedures.

[0019] The term "distal" is understood to mean, in particular, when used facing a patient and / or away from a user. The term "proximal" is understood to mean, in particular, when used facing away from a patient and / or towards a user. Specifically, proximal is the opposite of distal. The medical imaging device has, in particular, at least one rigid or flexible shaft. The shaft may be an elongated object. Furthermore, the shaft may at least partially, and preferably at least predominantly, form the distal section.

[0020] The image acquisition unit can be located at least partially, and in particular at least to a large extent, in and / or form part of the proximal section. In other embodiments, the image acquisition unit can be located at least partially, and preferably at least to a large extent, in and / or form part of the distal section. Furthermore, the image acquisition unit can be distributed at least partially between the proximal and distal sections.

[0021] The first acquisition line can include at least one image sensor, and the second acquisition line can include at least one image sensor. The first and second image sensors can be of different designs and, in particular, have different spectral sensitivities. The second image sensor can be sensitive to light in the infrared, especially near-infrared, spectral range, or be configured to detect infrared light, especially near-infrared light. The first and / or the second image sensor can be, for example, a CCD sensor and / or a CMOS sensor.

[0022] The first acquisition line can be configured for white light imaging. Accordingly, the reference images can be white light images. These can have two, particularly pronounced, spatial dimensions. In other words, the first acquisition line can be configured to generate two-dimensional spatial image data. The first acquisition line can be spatially resolved such that it provides a resolution of at least 100 pixels, preferably at least 200 pixels, preferably at least 300 pixels, and advantageously at least 400 pixels in at least two different spatial directions. In some embodiments, the first acquisition line is configured to generate continuously updated reference images.The first acquisition line can, for example, be set up to acquire the reference images essentially in real time, for example with a repetition rate of at least 10 Hz, in particular at least 20 Hz, preferably at least 30 Hz.

[0023] The second acquisition path can be configured for multispectral and / or hyperspectral imaging, specifically for acquiring and / or generating multispectral and / or hyperspectral image data. The spectral images can be based on multispectral and / or hyperspectral image data. In particular, the spectral images can be hyperspectral images. The second acquisition path can be configured to separately acquire spectral information for a given location within the object area, or to separately acquire light with different wavelengths. For example, the second acquisition path can include an optical element, in particular a dispersive element such as a prism and / or the like, configured to disperse light, especially object light. This allows the object light at the location point to be spectrally dispersed, and the spectral data for this location point to be acquired separately by the second image sensor.The spectral image comprises, in particular, a spectral dimension and, more specifically, a spatial dimension. The spectral image can have a resolution of at least 100 pixels, preferably at least 200 pixels, preferably at least 300 pixels, and advantageously at least 400 pixels in one spatial direction. In a second spatial direction, the spectral image can have a resolution of one pixel.

[0024] In contrast, the reference image can encompass two spatial dimensions and, in particular, no spectral dimension. The spectral image can, for example, be a spectral line image that, especially in one spatial direction, has only one pixel. Figuratively speaking, the spectral image can have a spatial axis, which can be assigned to the object area and / or the reference images, and a spectral axis, which is based on spectral information. A plane is defined by the two axes in such a way that a light spectrum can be read for each point on the spatial axis. The spatial and spectral information of the spectral images can be structured in such a way that a corresponding spectrum can be derived for each of several spatial pixels.

[0025] According to some embodiments, the image acquisition unit comprises optics, in particular comprising an objective lens, which is configured to image the object area and in particular to collect object light from the object area and / or couple it into the image acquisition unit.

[0026] The image acquisition unit can include a beam splitter configured to separate object light from the object area into a first part and a second part. The image acquisition unit can be configured to supply the first part to the first acquisition line and the second part to the second acquisition line. The beam splitter can be configured to separate the object light in a ratio that is at least substantially equal. However, other ratios are also conceivable. The beam splitter can be configured to separate object light coupled through the objective lens in such a way that it can be supplied to the first and second acquisition lines. In other words, the image acquisition unit can include a single objective lens for both the first and second acquisition lines.

[0027] The control unit can comprise multiple computing units, processors, and / or the like, arranged at different sections of the imaging device. For example, the control unit can be located partially in the distal section near the image sensors and / or within or integrally with the image acquisition unit, and in particular, partially outside of an endoscope. The imaging device can comprise, for example, a computing unit and an endoscope incorporating the image acquisition unit, with the computing unit enabling control of a function of the endoscope.

[0028] The control unit can be configured to acquire the image stream and the spectral images separately and / or independently. This can mean that the control unit is configured to perform the acquisition of the image stream and the acquisition of spectral images at different acquisition rates or repetition rates. In particular, this can mean that the spectral images are not co-registered with the reference images. Both acquisition steps can occur concurrently and / or independently. The image stream and the spectral images can be acquired asynchronously.

[0029] The term "image stream" can refer to a continuous series of reference images captured in rapid succession. This allows for the creation of a video representation of the object area. The image stream can be generated in such a way that a new reference image is continuously displayed, with each new reference image replacing the previous one. As a result, there is essentially no perceptible spatial offset between the individual reference images, and the transitions between them are seamless.

[0030] In image stream display, a new reference image can be continuously and continuously shown. This display can be shown to the user via a display device. Thus, a new reference image can be continuously shown to the user. Image stream display can refer to a serial presentation of the reference images. Individual reference images can be skipped. The image stream display can be an endoscopic video.

[0031] The sequence image can include a parametric representation of spectral data. This parametric representation might include, for example, a false-color representation of spectral information, such as the intensity ratio of different spectral ranges, and / or similar representations. Furthermore, the sequence image can consist of multiple spectral representation segments. These segments can be added to the sequence image step by step, thus building it up incrementally. A spectral representation segment can be based on a single spectral image. It is also conceivable that a single spectral representation segment can be based on multiple spectral images. Consequently, the sequence image can be constructed in a mosaic-like manner. The spectral representation segments can be joined and / or stitched together.

[0032] For example, a spectral image can be converted into a spectral representation, such as by converting spectral information into color values. A spectral representation can accordingly have the spatial dimensions of an original spectral image, specifically multiple pixels in one spatial direction and one pixel in a second spatial direction. Each pixel can be assigned a color value corresponding to the spectral information. If necessary, the spatial dimension in the second spatial direction can be extended by widening the spectral representation. For example, the control unit can be configured to widen the spectral representation such that the sequence image can be extended by at least two pixels along one extension direction using the spectral representation.The extension direction can refer to a direction in which the sequence image is extended by adding the spectral representation parts.

[0033] According to some embodiments, the control unit is configured to remove an old spectral representation portion of the sequence image if a spectral representation portion to be added overlaps the old spectral representation portion. The old spectral representation portion can refer to a spectral representation portion that has already been added to the sequence image. If the image acquisition unit or the image sensors are stationary and / or only moving slightly, it can happen that a spectral image encompassing a single location point of the object area is generated multiple times. In such a case, removing the old spectral representation portion can prevent the overlapping of multiple spectral representation portions.

[0034] The sequence image makes spectral information available to the user. This information can be tailored to a specific application. For example, the user can examine the oxygen saturation of a tissue. The sequence image can then be generated from specific spectral information, allowing the oxygen saturation to be estimated.

[0035] The term "overlay" can refer to a crossfade. The sequence image can be superimposed on the endoscopic video. According to some embodiments, the sequence image can be superimposed on the image stream in a semi-transparent manner.

[0036] Without the spatial mapping of the spectral images to the image stream, a spatial offset could occur between the spectral representation components and the currently displayed reference image. Spatial mapping means that a spatial coordinate of the spectral image is assigned to a spatial coordinate of a reference image, particularly within a reference coordinate system. This reference coordinate system can be defined by the reference image. The sequential addition of spectral representation components to the sequence image can be understood as a continuous and / or ongoing update of the sequence image. The previously existing sequence image can remain, at least in sections, and be gradually extended with further spectral representation components. Thus, the sequence image can be built up through continuous spectral and reference image acquisition. The updated sequence image can then be superimposed on the image stream.This allows it to grow continuously and progressively. For the user, this can create the impression that the sequence image sweeps over the display of the video stream. New spectral representation segments can be continuously superimposed on the video stream, thus covering a larger area of ​​the video stream with the sequence image. In other words, "sweeping over" can mean that the video stream display gradually and dynamically fills with a spectral parameter representation. In this way, spectral information is made available to the user step by step and / or section by section, with the spectral information being spatially assigned to the display, in particular an endoscopic video.

[0037] According to some embodiments, the display of the image stream is generated together with the sequence image, particularly at a frame rate. This can give the user the impression that a multi- and / or hyperspectral video is being generated. A spectral display component added to the sequence image can be based on one or more spectral images.

[0038] A timestamp can be understood as a temporal recording of a recording point in time. This can be defined relative to a starting point. The starting point could be, for example, the beginning of use or similar events. The timestamp can be recorded with a resolution in the millisecond range and / or even higher. This allows a recording point to be assigned to each spectral image and reference image. Each reference image can therefore have its own first timestamp, and each spectral image can have its own second timestamp.

[0039] In general, temporal correlation allows the acquisition time of a spectral image to be placed in a temporal relationship with the acquisition time of a reference image. Through temporal correlation, a spatial correlation can also be determined and / or ascertained. In particular, a spectral image whose acquisition time is temporally closest to a reference image can also be spatially closest to the reference image.

[0040] Using timestamps, the acquisition time of a spectral image can be compared with the acquisition time of a reference image. This allows the temporal correlation to be determined. For example, from several spectral images, one can be identified whose acquisition time is closest to the acquisition time of a corresponding reference image. Similarly, a sequence of spectral images can be created based on the second timestamp of the spectral images compared to the first timestamp of a reference image. Thus, the sequence of spectral images can be determined by the control unit, with the sequence temporally assigned to the image stream and / or the reference images.

[0041] Accordingly, a sequence image can be generated by determining the temporal assignment, whereby the spectral representation components are also assigned to spatially corresponding reference images. The sequence image can therefore be built up step by step.

[0042] Spectral data can be made intuitively perceptible to a user, especially when the sequence image is superimposed on the display of the image stream at a frame rate sufficient for real-time display. The sequence image display can appear video-like to the user. In particular, the control unit can be configured to superimpose the sequence image on the display at least substantially every update of the image stream. It is understood that the display of individual reference frames can be omitted and / or that individual reference frames may be of such poor quality that an update of the sequence image is omitted.

[0043] According to some embodiments, the control unit is further configured to correct the timestamps of the reference images and / or the spectral images by a time offset. This allows for precise temporal and / or spatial correlation. Due to system limitations, the time offset for spectral image acquisition and reference image acquisition may differ. This can be caused by different transmission speeds, different image sensor configurations, and / or similar factors. This time offset can, for example, be determined once for both spectral image acquisition and reference image acquisition. Subsequently, the time offset can be assumed to be constant for the first and second timestamps and used for correction.

[0044] Spectral data can be made particularly intuitive for a user if the control unit is configured to acquire the spectral images as a separate image stream. The image stream and the separate image stream can be based on different image acquisition rates. Preferably, the image acquisition rate of the separate image stream is higher than that of the image stream. Since the spectral images can each only have one spatial dimension, the higher image acquisition rate of the separate image stream allows the sequence image to build up more smoothly. For example, from reference image display to reference image display, several spectral images can be assigned temporally and / or spatially according to the image stream display and added together as a spectral representation segment, and / or added individually, with several spectral representation segments being added to the sequence image step by step.This allows a second spatial dimension of the sequence image to be captured faster and / or with higher resolution.

[0045] According to a further training, the control unit is configured to sequentially align and add the spectral representation components based on the spectral images to the sequence image of the object area when generating the sequence image. This alignment is based on spatial information from the reference image assigned to the respective sequence image. This allows for the creation of a high-quality sequence image. From reference image to reference image, changes such as rotation of the first image sensor, a change in the distance between the objective lens and the object area, and / or a lateral movement of the image sensor can occur. The spatial information can include information about the rotation, movement, and / or change in distance. This alignment compensates for the rotation, movement, and / or change in distance. Accordingly, the spectral images or spectral representation components can be aligned so that they spatially match the sequence image.

[0046] In principle, alignment can refer to image registration. Registering an image can describe a process in which individual images are adjusted so that they align themselves with respect to their position, scaling, rotation, and / or distortion. Accordingly, the spectral image, or spectral line image, can be registered in such a way that it corresponds spatially to the reference image, particularly with respect to points within the object area.

[0047] Furthermore, the control unit can be configured to sequentially align the spectral representation elements based on spatial information from the image stream and add them to the sequence image when generating the sequence image of the object area. This allows the sequence image to be built up step by step, taking into account movements of the first image sensor. The spectral representation elements can be aligned so that they correspond spatially and / or spatially to the sequence image.

[0048] According to some embodiments, the control unit is further configured to perform the step of selecting a first spectral image, a first reference image, a second spectral image, and a second reference image, and the step of assigning a first alignment rule based on the first reference image to the first spectral image and a second alignment rule based on the second reference image to the second spectral image. This allows an alignment rule suitable for the spectral image to be selected, thereby achieving a small spatial offset between the sequence image and the reference image. The selection of the spectral images can be based on temporal assignment, whereby, in particular, temporally compatible spectral images and reference images are selected. The reference images can be temporally successive reference images of the image stream.Spectral images matching the temporal sequence of the image stream can therefore be selected, particularly from multiple spectral images. Especially when spectral images are generated with a higher repetition rate than the reference images, selection allows for the selection of spectral images that are temporally appropriate to the image stream. The associated alignment rule does not necessarily have to be used to align the spectral representation components based on the first or second spectral image. An estimated alignment rule can also be used for this purpose. This can be determined analogously to an estimated alignment rule for a spectral image that lies temporally between the two spectral images. The alignment rule can be a reference-image-specific alignment rule. That is, a specific alignment rule can be determined for at least substantially every reference image. Based on this alignment rule, the spectral image, or...The spectral representation component based on the spectral image must be alignable and / or registered. The alignment rule can be a projective transformation between two images or image regions, represented in particular by a 3x3 matrix. This alignment rule can describe the geometric relationships between pixels of both images. Specifically, the alignment rule can be a homography. Homography can be used to perform image registration and / or align the spectral images and / or the spectral representation components.

[0049] Furthermore, the control unit can be configured to perform the step of determining an estimated alignment rule based on the first and second alignment rules for a third spectral image occurring temporally between the first and second spectral images, and adding the third spectral image to the sequence image according to its temporal position, with the third spectral image being aligned according to the estimated alignment rule. These steps allow multiple spectral images to be registered between two still images of the image stream, thus accelerating the generation of the sequence image. This improves usability and reliability for the user, as the spectral information is available more quickly. Additionally, the spatial resolution of the sequence image can be enhanced.

[0050] Adding the third spectral image can be understood as adding a spectral representation component based on the third spectral image to the sequence image. Similarly, aligning the third spectral image can be understood as aligning and / or registering the spectral representation component based on the third spectral image. The estimated alignment rule may be an interpolated homography.

[0051] If the subsequent image stream comprising the spectral images has a higher frame rate than the initial image stream, homography based on the first reference image can be used to register the first spectral image. Similarly, homography based on the second reference image can be used to register the second spectral image. However, for the third spectral image, using alignment rules or homographies based on the first or second reference image may not result in sufficiently good image registration, thus impairing the quality of the sequence image. Image registration can be improved by using interpolated homography. Interpolated homography is a homography generated by interpolating between two or more known homographies.

[0052] When the first acquisition stream is moved during image acquisition, the relative position of the object's location points to the first image sensor changes. Consequently, the pixels in the image stream display move. This shifts the relative position of the pixels in the image stream display to their corresponding pixels in the sequence image. The inventors recognized the advantage of adjusting the sequence image according to the movement of the pixels in the image stream display. This ensures that the sequence image is spatially well superimposed on the image stream display. To achieve this, the control unit can be configured to perform an additional step of adjusting the sequence image to the image stream based on spatial information from the image stream. Thus, the sequence image can be aligned according to the image stream display.This can be done, for example, in preferably every instance of generating a still image based on a new reference image during the generation of the image stream. Thus, when a new still image of the image stream is generated, the sequence image can be aligned and adjusted accordingly. This allows the entire sequence image to be aligned in such a way that a spatial relationship between the sequence image and the image stream is maintained.

[0053] According to some embodiments, the control unit is further configured to perform the step of determining a reference image-specific orientation rule, wherein the orientation rule is determined based on landmark detection. Landmark detection allows for the reliable determination of a high-quality orientation rule, resulting in high-quality sequence image generation. Landmark detection can be performed for each reference image. By comparing the position of the landmarks, movement and / or similar behavior between two reference images can be inferred. The orientation rule can then be determined accordingly, for example, by calculating a homography for a second reference image that describes movement and / or similar behavior relative to a first reference image. In this case, the acquisition time of the second reference image is later than the acquisition time of the first reference image.Sequentially, a reference image-specific alignment rule, in particular homography, can be determined from reference image to reference image in temporal sequence.

[0054] The computational effort required to determine the alignment rule can be reduced if the control unit is configured to determine the reference image-specific alignment rule in a multi-stage, particularly two-stage, alignment process. The alignment process can be understood as a process during which the alignment rule is determined and / or the spectral display components are aligned according to the spatial information of the reference images. In other words, the spectral image can be registered during the alignment process.

[0055] The control unit can also be configured to detect low-quality reference images during the alignment process. A reference image might be considered low-quality, for example, if it detects a small number of landmarks, particularly a number below the reference number above which the quality is sufficient. Sequence image generation based on a low-quality reference image can then be omitted. Corresponding spectral images can be excluded from being added. Similarly, the control unit can be configured to detect movement from reference image to reference image during the alignment process, suspending the addition of a spectral representation component to the sequence image if movement below a certain reference movement threshold is detected. Thus, if only a small amount of movement is detected, sequence image generation can be suspended, thereby saving processing power.

[0056] When comparing landmarks of sequentially acquired reference images and determining an alignment rule based on this comparison, the quality of the alignment rule can progressively decrease with ongoing image acquisition. For example, an alignment rule for the later reference image could be determined based on the alignment rule of the earlier reference image in order to determine an alignment rule for the new reference image with respect to the entire sequence image. This chaining of the alignment rules can lead to error propagation and accumulation.To avoid this, landmarks can be selected in the first stage of the alignment process based on landmark recognition of the reference image compared to a selected previous reference image. In the second stage, the alignment rule is determined based on a comparison of the landmarks with global landmarks based on several previous reference images. The second stage can be a so-called "global registration," while the first stage can be a so-called "local registration." This multi-stage alignment process can be understood as a "hybrid registration."

[0057] In the first stage, landmarks can be determined by comparing two consecutive reference images. This can be a so-called frame-to-frame registration. From several landmarks, those with particularly robust recognition can be selected. This avoids the need to compare a large number of landmarks with global landmarks. Instead, only robustly recognized, high-quality landmarks are used for global registration. Local registration thus improves landmark quality, filters out low-quality reference images, and enables the detection of camera movement.

[0058] In the second stage, the landmarks are compared with global landmarks. These are derived from landmark recognition of previous reference images. This can be a so-called frame-to-scene registration. A reference image (frame) of the image stream can be registered with a reference of the entire scene, representing the visible object area. The reference can be determined, for example, at the beginning of the sequence image generation and can be defined, in particular, by the first reference image assigned to a spectral image. By determining the alignment rule according to the second stage, error propagation and accumulation can be avoided. The global landmarks can be stored in a database. Furthermore, the control unit can be configured to update the stored global landmarks based on the alignment process.This ensures that the database doesn't become too large and contains only high-quality global landmarks. New landmarks can be added, and existing ones updated and / or removed, depending on the results of the alignment process. The database can be empty at the start of creating a sequence image and populated based on the alignment process.

[0059] Furthermore, the control unit can be configured to perform a step of truncating the sequence image. This involves detecting the size of the sequence image and, once a certain size limit is reached, sequentially deleting an older portion of the sequence image as a new portion is added. As the size of the sequence image increases, the computational effort required to align it with the image stream also increases. Truncation prevents uncontrolled size growth and ensures that sufficient processing power is available to align the sequence image. Truncation can be understood as the removal of image portions. The size of the sequence image can remain the same even when new image portions are added. Furthermore, truncation allows the user to identify which sections of the sequence image are old or outdated.which area of ​​the object area has not been displayed for a longer period of time.

[0060] To ensure that the sequence image is of good quality and can be spatially overlaid with the image stream, the control unit can be configured to restart the acquisition of the image stream from reference images and / or the generation of the sequence image in an error mode. During the alignment process, a plausibility check and / or a sanitization check of detected landmarks and / or specific alignment rules can be performed. After an accumulation of errors in the plausibility check and / or the sanitization check, the database can be cleared and the image registration of the spectral images restarted.

[0061] A spectral image, especially a spectral line image, can have a width of only one pixel in one spatial direction. However, if the image acquisition unit, particularly the first image sensor, moves quickly relative to the object area, stripes can appear in the sequence image for which no spectral information has been captured. These might, for example, be displayed as black. To avoid these black stripes, the line width of the spectral image could be increased as described above. However, this reduces the resolution of the sequence image. Alternatively, the frame rate of the subsequent image stream encompassing the spectral images could be increased, thereby capturing the spectral information at a higher frequency. However, this could lead to excessive computing power requirements.It is also conceivable that, to avoid black bars in the sequence image, the control unit is configured to perform the following steps: determining the relative movement between the image acquisition unit and the object area, and then adjusting the width of the spectral display components based on this determined relative movement. The width of the spectral display components can thus be dynamically adjusted according to the relative movement. If the image acquisition unit moves faster, the width can be increased; if it moves slower, the width can be decreased.

[0062] To determine relative motion, the imaging device can further include a motion sensor. The motion sensor allows the speed of the relative motion to be inferred. The motion sensor can be located near the objective lens. Alternatively, the motion sensor can be positioned such that, based on knowledge of its geometric position relative to the objective lens, movement of the objective lens relative to the object area can be inferred. The motion sensor can include an accelerometer.

[0063] The image acquisition unit can further comprise an optical element configured to mask a portion of the object area from the image. This optical element is arranged in the second acquisition line such that light passing through it falls onto an image sensor of the second acquisition line. This ensures that only the object light from a portion of the object area falls onto the second image sensor to generate the spectral image. In this way, a specific spatial dimension can be selectively masked. Preferably, the optical element is a slit-shaped aperture. This can be designed analogously to a slit-shaped aperture in a hyperspectral camera for the pushbroom technique.

[0064] Furthermore, the image acquisition unit can also include a motor configured to move the optical element and the image sensor of the second acquisition line in such a way that spectral images of different sub-areas of the object can be generated using the second acquisition line. The imaging device can thus be configured to perform multispectral or hyperspectral imaging using the pushbroom method. For example, the imaging device can be operated in different modes, with a first mode generating multispectral or hyperspectral images using the pushbroom method and a second mode generating multispectral or hyperspectral images using the features described above.

[0065] Furthermore, the image acquisition unit can be configured for stereo imaging, with the control unit configured to correct the sequenced image based on stereo image data. Stereo image data can be available, particularly with respect to the reference images. From this stereo image data, the curvature of the object area can be determined, and this curvature can be taken into account when aligning the spectral display components. The image acquisition unit can, for example, include a second objective lens, another first acquisition string, and another first image sensor. The objective lens can be configured to couple object light from the object area into the image acquisition unit at a different viewing angle than the first objective lens. Furthermore, the image acquisition unit can be configured to be held by a user and moved relative to the object area.The image acquisition unit is moved manually by the user, allowing spectral images of different sections of the object to be generated. The sequence image is thus created based on the user's movement of the image acquisition unit, specifically the endoscope. The user can move and / or align the endoscope within the cavity while spectral images are being generated. A motor for partial movement of the second acquisition strand is no longer necessary.

[0066] The present invention further provides a system comprising an imaging device according to the invention and a robotic arm configured to hold the image acquisition unit and move it relative to the object area. The movement of the image acquisition unit can be controlled and / or predetermined.

[0067] Furthermore, the control unit can be configured to perform a step of correcting a sequence image, whereby the intensity profile of the sequence image is shifted based on a reference intensity profile derived from a reference image. The sequence image may still exhibit a spatial offset relative to the reference image. To correct this, the correction step can be performed. This can be particularly advantageous for a still image representing the image stream. An existing reference image can thus be used to correct the sequence image. The correction can be performed similarly to the dynamic time warping (DTW) of an audio signal. The entire sequence image can be shifted line by line such that the superimposed representation of the sequence image on the reference image is spatially corrected.

[0068] Furthermore, the invention provides for a method for imaging, in particular an endoscopic imaging method. The method is carried out, in particular, with an imaging device according to the invention. The method comprises the following steps:

[0069] - Capturing a stream of reference images;

[0070] - Acquisition of spectral images simultaneously with the acquisition of the image stream of reference images; and

[0071] - Generating a representation of the image stream, wherein a sequence image of the object area is superimposed on the representation, the sequence image being formed based on the spectral images by spatially assigning the spectral images to the representation of the image stream and sequentially adding spectral representation parts based on the spectral images to the sequence image in such a way that the sequence image sweeps over the representation of the image stream.

[0072] Furthermore, the invention provides for a method for imaging, in particular an endoscopic imaging method. The method is carried out, in particular, with an imaging device according to the invention. The method comprises the following steps:

[0073] - Acquisition of a stream of reference images; - Acquisition of spectral images simultaneously with the acquisition of the stream of reference images;

[0074] - Determining a temporal assignment of at least one spectral image to a reference image of the image stream by comparing a corresponding first timestamp of the spectral image in question with a corresponding second timestamp of the reference image in question; and

[0075] - Generating a sequence image of the object area based on spectral images by sequentially adding spectral representation parts based on the spectral images to the sequence image according to the temporal assignment.

[0076] Furthermore, the invention provides for the provision of program code comprising instructions which, when executed by a processor, effect the execution of a method according to the invention.

[0077] Furthermore, the invention provides for the provision of a computer program product that includes a machine-readable medium on which program code according to the invention is stored.

[0078] The present invention is described below by way of example with reference to the accompanying figures. The drawing, the description, and the claims contain numerous features in combination. A person skilled in the art will expediently consider the features individually and use them meaningfully in combination within the scope of the claims.

[0079] If more than one instance of a particular object exists, only one of them may be identified with a reference symbol in the figures and description. The description of this instance can then be applied to the other instances of the object. If objects are named using numerical terms, such as first, second, third object, etc., these serve to identify and / or classify objects. Thus, for example, a first object and a third object, but not a second object, may be included. However, numerical terms could also indicate a number and / or sequence of objects.

[0080] They show:

[0081] Fig. 1 shows a schematic representation of an imaging device;

[0082] Fig. 2 is a schematic representation of a sequence image;

[0083] Fig. 3 shows another schematic representation of a sequence image;

[0084] Fig. 4 shows a timeline, on which events are listed in chronological order from left to right.

[0085] The recording times, spectral images and reference images are plotted;

[0086] Fig. 5 shows a schematic representation of a representation of an image stream;

[0087] Fig. 6 shows another schematic representation of the image stream; Fig. 7 shows a schematic representation of the image stream with a sequence image superimposed on it;

[0088] Fig. 8 shows a schematic representation of a system; and

[0089] Fig. 9 shows a schematic flowchart of an imaging procedure.

[0090] Figure 1 shows a schematic representation of an imaging device 100, 200. In the example shown, the imaging device 100, 200 is an endoscopic imaging device, specifically an endoscope. Alternatively, the imaging device 100, 200 could be an exoscopic, microscopic, or macroscopic imaging device. The imaging device 100, 200 is intended for examining a cavity, particularly a patient. The imaging device 100, 200 has the features of both an imaging device 100 and an imaging device 200. The imaging devices 100, 200 are not shown individually.

[0091] The imaging device 100, 200 comprises an image acquisition unit 102, which is configured as an endoscope 160. The image acquisition unit 102 has a handle 148 and a shaft 105. The shaft 105 is inserted into the cavity for examination of the patient, while the handle 148 remains outside the cavity. A user can hold the image acquisition unit 102, or the endoscope 160, by the handle 148 and guide the shaft 105 within the cavity. The shaft 105 has an objective lens 107 at a distal section 150, by means of which an object area is imaged and object light from the object area is coupled into the shaft 105. The coupled object light is guided along the shaft 105 on an optical path 146 to the handle 148. The image acquisition device 102 comprises optical elements not shown in detail, for example rod lenses and / or the like.Within the handle 148, a first detection strand 104 and a second detection strand 108 are arranged. It is also conceivable that at least one of the detection strands 104, 108 is arranged outside the endoscope or within the distal section 150 (neither case is shown). The object light is split equally into two parts by means of a beam splitter 144, which is also arranged in the handle 148. A first part is directed to the first detection strand 104 and a second part to the second detection strand 108.

[0092] The first acquisition line 104 has an image sensor 142 configured for white light imaging. The object light supplied to the first acquisition line 104 is imaged onto the image sensor 142 by means of further optical elements (not shown). This allows reference images 106 to be acquired by the first acquisition line 104. These are conventional RGB images or white light images and have two spatial dimensions. In the illustrated case, the reference image has 960 pixels in one spatial direction (width) and 540 pixels in another spatial direction (height). The second acquisition line 108 is configured for hyperspectral imaging. For this purpose, the image acquisition unit 102 includes an optical element 138 designed as a slit-shaped aperture. The aperture 138 is configured to block out a portion of the object area from being imaged. Object light from only this slit-shaped portion is transmitted.The optical element 138, or aperture 138, is arranged in the second detection line 108 such that the object light passing through it falls onto an image sensor 140 of the second detection line 108. A dispersive element 139, designed as a prism, is also arranged in front of the image sensor 140. This prism is configured to disperse the object light. As a result, dispersed object light, which has previously passed through the slit-shaped aperture, falls onto the image sensor 140. This allows the image sensor 140 to capture a spectral image that has both a spatial and a spectral dimension. The aperture 138 selects a slit-shaped portion of the object area, so that the spectral image has only one pixel in one spatial direction (width). In another spatial direction, which runs along the slit of the aperture 138, the spectral image has 540 pixels.The spectral image has 100 pixels in the spectral dimension. Accordingly, the spectral images are spectral line images. Such a device is known from hyperspectral cameras that operate according to the pushbroom method. Furthermore, the second acquisition line 108 and, in particular, the image sensor 140 are configured to acquire spectral information up to the infrared range, more precisely up to a wavelength of 1100 nm.

[0093] The image acquisition unit 102 also includes a motor (not shown) configured to move the optical element 138 and the image sensor 140 of the second acquisition strand 108 such that spectral images of different sub-areas of the object can be generated by the second acquisition strand 108. The imaging device 100, 200 can therefore operate in one mode according to the pushbroom method. In another operating mode, the aperture 138 remains in a constant relative position to the image sensor 140. In this mode, the user can manually move the image acquisition unit 102 so that the aperture 138 is moved to different relative positions to the object. This allows spectral images of different linear sub-areas of the object to be generated.

[0094] A motion sensor 136, designed as an inertial sensor, is also arranged on the handle 148. The motion sensor 136 can determine relative movement between the image acquisition unit 102 and the object area. From this, a relative velocity can be deduced.

[0095] The imaging device 100, 200 further comprises a light source 103 by means of which illumination light is generated. The illumination light is guided to the endoscope 160 by means of a light guide 161 and coupled into the endoscope 160. Inside the endoscope 160, the illumination light is guided along the shaft 105 by means of an optical element (not shown), for example, a light guide, to the distal section 150 and coupled out of the endoscope 160. This illuminates the object area. The imaging device 100, 200 further comprises a display device 101 on which a display 112 of an image stream of reference images 106 is shown. This display 112 is an endoscopic video of the object area. Thus, when the user moves the endoscope 160 in the cavity, they can view anatomical structures within the cavity at least substantially in real time on the display device 101.A sequence image 114 is superimposed on the display 112. The sequence image 114 is generated by a control unit 110, which is also part of the imaging device 100, 200. The control unit 110 is configured to execute the procedure described below. As will be described below, the sequence image 114 is sequentially constructed from spectral representation components based on the spectral images. These components exhibit a false-color representation of the spectral information of the spectral images. Since the sequence image 114 is superimposed on the display again at each still image (frame) of the endoscopic video in which a new reference image 106 is displayed, with new spectral representation components being added to the sequence image 114, it appears to the user as if a spectral video is being superimposed on the display 112. In this sense, sequence image 114 builds up in real time, line by line, and thus covers representation 112.In other words, sequence image 114 is superimposed on representation 112 of the image stream with a frame rate sufficient for real-time display.

[0096] Since pixels move in the endoscopic video, sequence image 114 is superimposed frame by frame onto the display 112 of the image stream. To do this, sequence image 114 is adjusted according to the image stream, in particular the currently displayed reference image 106. It is thus shifted so that the spatial assignment is maintained. To achieve this, the control unit 110 is configured to perform an image adjustment of sequence image 114 to the image stream based on spatial information from the image stream.

[0097] In this case, oxygen saturation is to be determined using spectral imaging. For this purpose, two spectral ranges of the sequence image are correlated. One parameter is determined by dividing the measured intensity at 860 nm by the average intensity between 950 nm and 980 nm. This parameter is color-coded. The spectral images are converted into spectral representation segments, whereby a parameter is determined for each pixel of the spectral image and assigned to the corresponding spectral representation segment. The height of the spectral image (540 pixels) corresponds to the height of the spectral representation segment. The width of the spectral representation segment is extended to allow the sequence image 114 to build up more quickly. For example, the spectral representation segment can have a width of 3 pixels.

[0098] Sequence image 114 is displayed in more vibrant colors than display 112. Furthermore, sequence image 114 is spatially superimposed on the currently displayed reference image 106 of display 112. For this purpose, sequence image 114 is registered based on the reference image 106, as will be described in more detail below. In other words, the spectral images are spatially assigned to display 112 of the image stream so that the spectral representation components spatially match display 112 well. A location point of the object area is thus preferably represented by a pixel on the display device, wherein an image of the location point based on the first acquisition line and an image of the same location point based on the second acquisition line are superimposed on the pixel.

[0099] The control unit 110 is located outside the endoscope 102 and is configured to receive image data from the image sensors 140 and 142. For this purpose, the control unit is connected to the endoscope 160 via a data cable. Furthermore, the control unit 110 can control a function of the image sensors 140 and 142. For example, an image acquisition process can be started and / or stopped. It is also conceivable that the control unit 110 additionally includes a processor located within the handle 148. The control unit 110 can also be located entirely within the handle 148 (not shown). It can also be seen that the control unit 110 has a database 126. Global landmarks, which are determined during the generation of the sequence image 114, are stored in the database 126.

[0100] It is also conceivable that the imaging device 100, 200 is configured as the image acquisition unit 102 for stereo imaging (not shown). For this purpose, the image acquisition unit 102 can include an additional objective lens and an additional image sensor, by means of which further reference images can be acquired. These can be acquired from a different viewing angle than those of the first acquisition stream 104. This allows the curvature of an anatomical structure of the object area to be determined. The control unit 110 can then be configured to correct the sequence image 114 based on the specific correction determined from the stereo image data. The sequence image 114 can thus be positioned more spatially in relation to the representation 112 of the image stream.

[0101] Figure 2 shows a schematic representation of a sequence image 114. The sequence image 114 is composed of six spectral representation parts 116, which were added sequentially to the sequence image 114 in the order of their numbering. During sequence image generation, the endoscope 160 was moved approximately in the direction of arrow 137. This allowed spectral line images of different areas of the object to be acquired by means of the second acquisition line 108. The spectral representation parts 116 are added to the sequence image 114 such that the spatial relationship of the pixels in the sequence image 114 corresponds to the spatial relationship of the points in the object. This allows the sequence image 114 to be superimposed on the representation 112 in a spatially appropriate manner.

[0102] Each of the spectral representation parts 116 has a width 134 of 3 pixels and a height 135 of 540 pixels. Since the endoscope 160 was moved at a relatively high speed relative to the object area, a spectral image was not generated of the entire object area swept by the endoscope. This results in gaps 141 between the spectral representation parts 116, which are filled by black areas in the sequence image 114. Furthermore, it can be seen that the first spectral representation part 116 is located at an outer edge 151 of the sequence image 114. This spectral representation part 116 can define an origin of a coordinate system of the sequence image 114. Subsequently added spectral representation parts 116 are aligned according to this coordinate system and added to the sequence image 114.It can be seen that the sequence image 114 is built up in a mosaic-like manner and that the spectral representation elements 116 are gradually assigned to a specific area. This area can be visualized as a canvas that is filled sequentially and line by line. Furthermore, it can be seen that the endoscope 160 was rotated during its movement along the arrow 137. The spectral representation elements 116 were aligned accordingly.

[0103] To improve the quality of the sequence image 114, the width 134 of the spectral display elements 116 is dynamically adjusted to a movement of the endoscope 160 relative to the object area. Fig. 3 shows a schematic representation of a sequence image 114 similar to the sequence image 114 of Fig. 2. The control unit 110 is configured to determine the relative movement between the image acquisition unit 102 and the object area. For this purpose, the motion sensor 136 is used. The width 134 of the spectral display elements 116 is adjusted based on the determined relative movement. A relatively large relative movement was determined between spectral display element 116 numbered 1 and spectral display element 3. The width 134 was increased from 3 pixels of the spectral representation part 116 with the number 1 to a width 134 with 6 pixels (spectral representation part 116 with the number 3).Since a smaller relative motion was subsequently detected, the width 134 of spectral representation section 116, numbered 5, was reduced to 2 pixels. This reduces the gaps 141 and largely prevents the spectral representation sections 116 from overlapping. However, it is understood that the spectral representation sections 116 can still overlap. If a newer spectral representation section largely (not shown) overlaps an older one, the newer spectral representation section can replace the older one.

[0104] The reference images 106 are captured with a first timestamp, and the spectral images with a second timestamp. The timestamps mark the acquisition times of the corresponding reference images 106 and spectral images. Fig. 4 shows a timeline on which, in chronological order from left to right, the acquisition times 162 of each captured spectral image and the acquisition times 163 of each captured reference image 163 are plotted. The acquisition times 162 and 163 each correspond to a temporal position 120. For clarity, not all acquisition times are labeled with a reference symbol. It can be seen that the reference images 106 and the spectral images are each acquired as an image stream. Both image streams are based on different frame rates. In this case, the spectral images are acquired at a frame rate of 70 Hz, and the reference images at a frame rate of 10 Hz.Furthermore, the image streams are asynchronous and run parallel to each other. However, by capturing the spectral images with timestamps, the spectral images can be temporally assigned to the reference images by determining a temporal correlation. Since a time offset occurs between the mapping of the object area and image acquisition and / or the reception of image data by the control unit, and this offset differs between the reference images and spectral images, the timestamps are corrected accordingly. The magnitude of this time offset is system-dependent and is determined before the sequence image generation. In this case, the timestamps of the spectral images are corrected by 35 ms, while the timestamp of the reference images remains unchanged. The time offset is assumed to be constant for all spectral images.

[0105] The control unit 110 is configured to determine a temporal assignment 118 of at least one spectral image to each reference image 106 of the image stream by comparing a corresponding first timestamp of the respective spectral image with a corresponding second timestamp of the respective reference image 106. This assigns each reference image 106 to the next corresponding spectral image. Several spectral images lie between the reference images 106. A sequence of these spectral images is formed based on the timestamps of these spectral images. Thus, each spectral image is temporally assigned to the reference images, and the spectral images are placed in a temporal relationship with the reference images. The inventors recognized that this eliminates the need to co-register the reference and spectral images. Accordingly, more spectral images than reference images can be acquired, which allows the sequence image 114 to be built up more quickly.

[0106] The present invention further exploits the fact that the spatial relationship between a spectral image and a reference image can be inferred from their temporal relationship. A spectral image that is temporally closest to a reference image also exhibits the greatest spatial correspondence, since the first and second acquisition strands 104, 108 use the same objective lens 107. Therefore, if a later reference image is spatially related to an earlier reference image, and, for example, a homography is determined that assigns pixels of the two reference images to each other, this can be applied to align spectral images, or the spectral representation components based on them, that are temporally assigned to the two reference images. It is understood that the homography is adapted to be applicable to one-dimensional image data (line images) or a spectral representation component 116.From reference image 106 to reference image 106 according to arrows 164, an alignment rule, in particular homography, is determined in each case. This rule describes the movement of pixels between the two images. However, the alignment rule is not used to display and / or align reference image 106. As already described, this is displayed as a single frame of an endoscopic video. Instead, the alignment rule is used to align the spectral representation components so that they are added to sequence image 114 in such a way that sequence image 114 spatially corresponds to the representation 112 of the image stream of reference images 106. The spectral representation components 116 are thus added to sequence image 114 sequentially according to the temporal assignment 118.

[0107] For the spectral images located between the reference images, an estimated alignment rule is determined, presented here as an interpolated homography. The homographies of sequentially arranged reference images are first determined, as described in more detail below. Since the sequence of spectral images is known through the temporal assignment 118, it is possible to estimate, based on the current sequence, how the spectral representation components 116 of the corresponding spectral images are to be aligned. For this purpose, a first spectral image, a first reference image, a second spectral image, and a second reference image are selected based on the temporal assignment. A first alignment rule based on the first reference image is assigned to the first spectral image, and a second alignment rule based on the second reference image is assigned to the second spectral image. These alignment rules can be used to align the corresponding spectral images.Spectral representation parts 116 are used. Alternatively, an estimated alignment rule can be used for the first and second spectral images. For each spectral image occurring between the reference images, a separate interpolated homography is determined, transformed so that it can be applied to a line image, and applied to the corresponding spectral representation part 116 to align it. The aligned part 116 is then added to the sequence image 114 according to its temporal position 120. The control unit 110 is thus configured to sequentially align the spectral representation parts 116 based on spatial information from the image stream and add them to the sequence image 114 when the sequence image 114 of the object area is generated.

[0108] Figures 5 and 6 each show a schematic representation of the representation 112. Figure 5 shows a representation 112 based on the first reference image 106', and Figure 6 shows a representation based on the second reference image 106". The reference images 106' and 106" are temporally successive reference images 106. The sequence image 114 is superimposed on each representation 112. In Figure 6, several spectral representation components 116 can be seen, which have been added to the sequence image 114 compared to the sequence image 114 in Figure 5. By adding these components, a previously unsuperimposed area of ​​the representation 112 of the object region is overlaid by the sequence image 114. This process is carried out from reference image 106 to reference image 106, whereby the sequence image 114 sweeps over the representation 112. Furthermore, as previously described, it can be seen that the sequence image 114 moves along with the reference image 106. Since the endoscope 160 moves in the direction of arrow 165 (Fig.5) When the object was moved, the object area shifted relative to the endoscope 160, which is why the pixels in display 112 were shifted. Sequence image 114 was shifted according to the shift of the pixels, so that the spatial relationship between sequence image 114 and display 112 of the image stream remains unchanged.

[0109] The alignment rule is determined for each reference image. Individual reference images can be omitted, for example, if the image quality is too poor. This could be due to an insufficient number of landmarks being detected. Furthermore, a reference image is omitted if it is detected that the movement from one reference image to the next was below a certain threshold. Fundamentally, the alignment rule is determined based on landmark detection. More precisely, the control unit 110 is configured to determine the reference image-specific alignment rule in a two-stage alignment process. This is a hybrid registration process comprising local and global registration. First, local registration is performed. For this purpose, landmarks 122 are detected in the first reference image 106'.If reference image 106' is the first reference image 106 used in the creation of sequence image 114, landmarks 122 are stored as global landmarks in database 126 (not displayed). Landmarks 122 are also detected in the second reference image 106". These are compared with the landmarks 122 of the first reference image 106'. Landmarks 169 were detected in both images 106' and 106" and marked as matches. Since the endoscope 160 moved along the arrow 165, landmark 166 was no longer detected in the second reference image 106". Furthermore, landmark 167 was not detected, as indicated by the empty dashed box in Fig. 6. In addition, landmarks 168 were detected in Fig. 6. If too few landmarks are detected, in this case fewer than 70, reference image 106 is not used to determine an alignment rule.

[0110] The global registration is then performed. In this process, the detected landmarks 169 are compared with global landmarks stored in database 126. Based on the detected landmarks 169, in this case 500 landmarks (not shown), although the number could be smaller or larger, the alignment rule is calculated. This is a global homography and describes the movement of pixels from the reference image 106" to the entire scene. This is defined by an initialization. Thus, the global homography does not (directly) describe the movement of pixels to the previous reference image 106', but rather to a reference that is defined at the beginning of the generation of the sequence image 114. The global homography is then subjected to a plausibility test, and any outliers (individual matches) identified are discarded. Furthermore, the plausibility test may reveal that the determined homography is implausible.The homography is then calculated using other landmarks or a new reference image is used. If too many homographies are implausible, in this case more than 20, an error mode is activated and sequence image generation is restarted. As previously described, the global homography is used to align and register the spectral display components 116. Furthermore, the sequence image 114 is aligned based on the global homography and superimposed on the display 112.

[0111] If too few matches are found among several reference images (in this case, 20), sequence image 114 and the global database 126 are reset. The creation of sequence image 114 is restarted, and database 126 is also repopulated with global landmarks. The control unit 110 is therefore configured to restart the acquisition of the image stream from reference images 106 and / or the generation of sequence image 114 in error mode.

[0112] As previously described, database 126 is populated with recognized landmarks 122 at the beginning of the creation of sequence image 114. Database 126 is continuously updated to ensure that it contains a manageable number of global landmarks and primarily recent ones. Unrecognized landmarks, such as landmark 166 or landmark 167, are removed from database 126 after failing to detect them in 80 reference images. Furthermore, the landmarks 122 are categorized according to their detection quality, and the existing global landmarks are updated based on their detection in a new reference image 106. Newly detected landmarks, such as landmark 168, are also added to database 126. Only the 50 highest-quality newly detected landmarks 168 are added to database 126. In error mode, database 126 is reset.

[0113] The sequence image 114 is continuously extended until a limiting size of the sequence image 114 is reached. This limiting size describes the extent of the sequence image 114 in the extension direction. Once the limiting size is reached, it is maintained. To nevertheless add new spectral representation parts 116 to the sequence image 114, older image parts of the sequence image 114 are excluded from the display or deleted. Figure 7 schematically shows the display 112 of the image stream superimposed on the sequence image 114. The sequence image 114 has a size 128 in the extension direction (see previous figures). This corresponds to the limiting size. If a new image part 130 is added to the sequence image 114, an older image part 132 must be deleted. The image parts 130 and 132 comprise several spectral representation parts (not shown in detail).The control unit 110 is therefore set up to shorten the sequence image 114, whereby the size 128 of the sequence image 114 is recognized and from the point at which a limit size of the sequence image 114 is reached, the older image part 132 of the sequence image 114 is deleted sequentially by adding the new image part 130.

[0114] Fig. 8 shows a schematic representation of a system 300 comprising the imaging device 100, 200 and a robot arm 302. The robot arm 302 is configured to hold the image acquisition unit 102 and move it relative to the object area. The system is a medical system. The robot arm 302 is part of a medical robot 304, in particular a surgical robot.

[0115] Fig. 9 shows a schematic flowchart of an imaging method. The method can be performed with the imaging device 100 and / or the imaging device 200. The control unit 110 of the imaging device 100, 200 is configured to carry out the steps of this method. The individual steps have features described in connection with Figs. 1 to 9. Furthermore, the steps can be performed in a different sequence than shown.

[0116] The procedure comprises step 400, which involves acquiring the image stream of reference images 106, and step 402, which involves acquiring spectral images simultaneously with the image stream of reference images 106. The spectral images are also acquired as an image stream. Furthermore, the spectral images and the reference images are captured with a timestamp. The procedure also includes steps 401 and 403, which correct the timestamps of the reference images 106 (step 401) and the spectral images (step 403) by the specified time offset. The procedure also includes step 404 of selecting the first spectral image, the first reference image 106', the second spectral image and the second reference image 106" and step 406 of assigning the first alignment rule based on the first reference image 106 to the first spectral image and the second alignment rule based on the second reference image 106 to the second spectral image.The alignment rules are determined in step 405 of the procedure by defining a reference image-specific alignment rule, whereby the alignment rule is determined based on landmark detection. Furthermore, the procedure includes step 407, which determines the temporal assignment 118 of at least one spectral image to each reference image 106 of the image stream by comparing the corresponding first timestamp of the respective spectral image with the corresponding second timestamp of the respective reference image.The procedure also includes step 408, which determines the estimated alignment rule based on the first and second alignment rules, each for a third spectral image located temporally between the first and second spectral images, and step 409, which adds the third spectral image to the sequence image 114 according to its temporal position 120, whereby the third spectral image is aligned according to the estimated alignment rule. Step 409 is executed together with step 410 of the procedure. Step 410 comprises generating the sequence image 114 of the object domain from spectral images by sequentially adding spectral representation parts 116 based on the spectral images to the sequence image 114 according to the temporal assignment 118.The method further comprises step 411 of determining the relative motion between the image acquisition unit 102 and the object area, and step 412 of adjusting the width 134 of the spectral representation parts 116 based on the determined relative motion. The method also comprises step 413 of adapting the sequence image 114 to the image stream based on spatial information of the image stream, and step 414 of shortening the sequence image 114, whereby the size 128 of the sequence image 114 is detected, and once the limiting size of the sequence image 114 is reached, the older image part 132 of the sequence image 114 is sequentially deleted by adding the new image part 130.Furthermore, the method comprises step 415 of generating the representation 112 of the image stream, wherein the sequence image 114 of the object domain is superimposed on the representation 112 (step 416), wherein the sequence image 114 is formed from the spectral images by spatially assigning the spectral images to the representation 112 of the image stream and sequentially adding spectral representation parts 116 based on the spectral images to the sequence image 114 such that the sequence image 114 sweeps over the representation 112 of the image stream, and step 416 of superimposing the representation 112 with the sequence image 114, wherein the sequence image 114 is formed from the spectral images by spatially assigning the spectral images to the representation 112 of the image stream and adding spectral representation parts 116 based on the spectral images to the sequence image 114 are added sequentially in such a way that the sequence image 114 sweeps over the representation 112 of the image stream.

[0117] Reference symbol list

[0118] 100 Imaging device

[0119] 101 Display device

[0120] 102 image capture units

[0121] 103 Light source

[0122] 104 first recording strand

[0123] 105 shaft

[0124] 106 Reference image

[0125] 107 lens

[0126] 108 second recording strand

[0127] 110 Control unit

[0128] 112 illustration

[0129] 114 sequence image

[0130] 116 Spectral representation section

[0131] 118 temporal allocation

[0132] 120 temporal position

[0133] 122 Landmark

[0134] 124 previous reference image

[0135] 126 database

[0136] Size 128

[0137] 130 new image section

[0138] 132 older image section

[0139] 134 width

[0140] 135 Height

[0141] 136 motion sensor

[0142] 137 Arrow

[0143] 138 optical element

[0144] 139 dispersive element

[0145] 140 image sensor

[0146] 141 Gap

[0147] 142 image sensor

[0148] 144 beam splitters

[0149] 146 optical path

[0150] 148 Handle

[0151] 150 distal section outer edge

[0152] endoscope

[0153] optical fibers

[0154] Recording time

[0155] Recording time

[0156] Arrow

[0157] Arrow

[0158] country brand

[0159] country brand

[0160] country brand

[0161] country brand

[0162] Imaging device

[0163] system

[0164] Medical robotic arm

[0165] Step

[0166] Step

[0167] Step

[0168] Step

[0169] Step

[0170] Step

[0171] Step

[0172] Step

[0173] Step

[0174] Step

[0175] Step

[0176] Step

[0177] Step

[0178] Step

[0179] Step

[0180] Step

[0181] Step

Claims

Claims 1. Imaging device (100), in particular an endoscopic imaging device, comprising: an image acquisition unit (102) configured to image an object area, comprising: a first acquisition line (104) configured to acquire reference images (106) of the object area, and a second acquisition line (108) configured to acquire spectral images of the object area; a control unit (110) configured to perform the following steps: Acquisition of an image stream of reference images (106) by means of the first acquisition strand; and acquisition of spectral images by means of the second acquisition strand (108) simultaneously with the acquisition of the image stream of reference images (106), characterized in that the control unit (110) is further configured to perform the following step: Generating a representation (112) of the image stream, wherein a sequence image (114) of the object area is superimposed on the representation (112), wherein the sequence image (114) is formed on the basis of the spectral images by spatially assigning the spectral images to the representation (112) of the image stream and sequentially adding spectral representation parts (116) based on the spectral images to the sequence image (114) such that the sequence image (114) sweeps over the representation (112) of the image stream.

2. Imaging device (100) according to claim 1, wherein the sequence image (114) is superimposed on the representation (112) of the image stream with a frame rate sufficient for real-time display.

3. Imaging device (200) at least according to the preamble of claim 1 and in particular according to one of the preceding claims, characterized in that the image acquisition unit (102) is configured to acquire the reference images (106) with a first timestamp and to acquire the spectral images with a second timestamp, and the control unit (110) is configured to further perform the following steps: Determining a temporal assignment (118) of at least one spectral image to each reference image (106) of the image stream by comparing a corresponding first timestamp of the relevant spectral image with a corresponding second timestamp of the relevant reference image (106); and Generating a sequence image (114) of the object area based on spectral images by adding spectral representation parts (116) based on the spectral images sequentially to the sequence image (114) according to the temporal assignment (118).

4. Imaging device (200) according to claim 3, wherein the control unit (110) is configured to further perform the following step: Correcting the timestamps of the reference images (106) and / or the spectral images by a time offset.

5. Imaging device (200) according to claim 3 or 4, wherein the control unit (110) is configured to acquire the spectral images in the form of a further image stream, and wherein the image stream and the further image stream are based on different image acquisition rates.

6. Imaging device (200) according to one of claims 3 to 5, wherein the control unit (110) is configured to sequentially align and add to the sequence image (114) the spectral representation parts (116) based on the spectral images when generating the sequence image (114) of the object area, based on local information of the reference image (106) assigned to the sequence image (114) in question.

7. Imaging device (100, 200) according to one of the preceding claims, wherein the control unit (110) is configured to align the spectral representation parts (116) sequentially based on local information of the image stream and add them to the sequence image (114) when generating the sequence image (114) of the object area.

8. Imaging device (100, 200) according to claim 7, wherein the control unit (110) is configured to further perform the following steps: selecting a first spectral image, a first reference image (106'), a second spectral image and a second reference image (106"); and Assigning a first alignment rule based on the first reference image (106) to the first spectral image and a second alignment rule based on the second reference image (106) to the second spectral image.

9. Imaging device (100, 200) according to claim 8, wherein the control unit (110) is configured to further perform the following steps: determining an estimated alignment rule based on the first alignment rule and the second alignment rule, each for a third spectral image occurring in time between the first spectral image and the second spectral image; and Adding the third spectral image to the sequence image (114) according to its temporal position (120), wherein the third spectral image is aligned according to the estimated alignment rule.

10. Imaging device (100, 200) according to one of the preceding claims, wherein the control unit (110) is configured to further perform the following step: Performing an image adaptation of the sequence image (114) to the image stream based on local information of the image stream.

11. Imaging device (100, 200) according to one of the preceding claims, wherein the control unit (110) is configured to further perform the following step: Determining a reference image-specific alignment rule, where the alignment rule is determined based on landmark detection.

12. Imaging device (100, 200) according to claim 11, wherein the control unit (110) is configured to determine the reference image-specific alignment rule in a multi-stage, in particular two-stage, alignment process.

13. Imaging device (100, 200) according to claim 12, wherein in a first stage of the alignment process landmarks (122) are selected based on landmark recognition of the reference image (106) in comparison to a selected previous reference image (124); and wherein in a second stage of the alignment process the alignment rule is determined based on a comparison of the landmarks (122) with global landmarks based on several previous reference images (106).

14. Imaging device (100, 200) according to claim 13, wherein the global landmarks are stored in a database (126), and wherein the control unit (110) is configured to update the stored global landmarks based on the alignment process.

15. Imaging device (100, 200) according to one of the preceding claims, wherein the control unit (110) is configured to further perform the following step: Shortening the sequence image (114), whereby a size (128) of the sequence image (114) is recognized and, from the point of reaching a limit size of the sequence image (114), an older image part (132) of the sequence image (114) is sequentially deleted by adding a new image part (130).

16. Imaging device (100, 200) according to one of the preceding claims, wherein the control unit (110) is configured to restart the acquisition of the image stream of reference images (106) and / or the generation of the sequence image (114) in an error mode.

17. Imaging device (100, 200) according to one of the preceding claims, wherein the control unit (110) is configured to further perform the following steps: Determining a relative motion between the image acquisition unit (102) and the object area; and Adjusting a width (134) of the spectral representation parts (116) based on the determined relative motion.

18. Imaging device (100, 200) according to one of the preceding claims, further comprising a motion sensor (136).

19. Imaging device (100, 200) according to one of the preceding claims, wherein the image acquisition unit (102) comprises an optical element (138) which is configured to block out a partial area of ​​an image of the object area, wherein the optical element (138) is arranged in the second acquisition strand (108) such that light passing through the optical element (138) falls onto an image sensor (140) of the second acquisition strand (108).

20. Imaging device (100, 200) according to claim 19, wherein the image acquisition unit (102) further comprises a motor which is configured to move the optical element (138) and the image sensor (140) of the second acquisition string (108) in such a way that spectral images of different sub-areas of the object area can be generated by means of the second acquisition string (108).

21. Imaging device (100, 200) according to one of the preceding claims, wherein the image acquisition unit (102) is configured for stereo imaging, wherein the control unit (110) is configured to correct the sequence image (114) on the basis of stereo image data.

22. Imaging device (100, 200) according to one of the preceding claims, wherein the image acquisition unit (102) is configured to be held by a user and moved relative to the object area.

23. System (300) comprising: an imaging device (100, 200) according to one of the preceding claims, and a robot arm (302) configured to hold the image acquisition unit (102) and move it relative to the object area.

24. Imaging method, in particular endoscopic imaging method, in particular carried out with an imaging device (100, 200) according to one of the preceding claims, comprising the steps: Acquiring a stream of reference images (106); and Acquisition of spectral images simultaneously with the acquisition of the image stream of reference images (106); characterized in that the method further comprises the following step: Generating a representation (112) of the image stream, wherein a sequence image (114) of the object area is superimposed on the representation (112), wherein the sequence image (114) is formed on the basis of the spectral images by spatially assigning the spectral images to the representation (112) of the image stream and sequentially adding spectral representation parts (116) based on the spectral images to the sequence image (114) such that the sequence image (114) sweeps over the representation (112) of the image stream.

25. Imaging method, in particular endoscopic imaging method, in particular carried out with an imaging device (100, 200) according to one of the preceding claims, comprising the steps: Acquiring a stream of reference images (106); and Acquisition of spectral images simultaneously with the acquisition of the image stream of reference images (106); characterized in that the method further comprises the following steps: Determining a temporal assignment (118) of at least one spectral image to each reference image (106) of the image stream by comparing a corresponding first timestamp of the relevant spectral image with a corresponding second timestamp of the relevant reference image (106); and Generating a sequence image (114) of the object area based on spectral images by adding spectral representation parts (116) based on the spectral images sequentially to the sequence image (114) according to the temporal assignment (118).

26. Program code comprising instructions which, when executed by a processor, cause the execution of a method according to claim 24 and / or 25.

27. Computer program product comprising a machine-readable medium on which program code according to claim 26 is stored.

Images