Motion compensated laser speckle contrast imaging

Abstract

Methods and systems for motion-corrected and motion-compensated laser speckle contrast imaging are disclosed. The method comprises: exposing a target region to coherent first light of a first wavelength, the target region comprising living tissue; and capturing at least one sequence of images. The at least one image sequence includes a first speckle image that is captured during exposure with the first light. The method further includes determining one or more registration parameters of an image registration algorithm for registering the first speckle images to each other. The method may further include determining a registered first speckle image by registering the first speckle image based on one or more registration parameters and an image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or determining a first speckle contrast image based on the first speckle image, determining a registered speckle contrast image by registering the first speckle contrast image based on one or more registration parameters and an image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast image.

Description

Technical Field

[0001] The present disclosure relates to motion compensated laser speckle contrast imaging, and in particular, but not exclusively, to methods and systems for motion compensated laser speckle contrast imaging, modules for motion compensation in laser speckle contrast imaging systems and methods, and computer program products enabling a computer system to perform such methods. Background Art

[0002] Laser speckle contrast imaging (LSCI) provides a rapid, full-field, in vivo imaging method for determining the two-dimensional (2D) perfusion map of living biological tissue. Perfusion can be an indicator of tissue viability and can therefore provide valuable information for diagnosis and surgery. For example, during intestinal surgery, the selection of a highly perfused intervention site can reduce anastomotic leaks.

[0003] LSCI is based on the principle that backscattered light from tissue illuminated with coherent laser light forms a random interference pattern at the detector due to differences in optical path lengths. The resulting interference pattern is called a speckle pattern and it can be imaged in real time using a digital camera. The motion of particles within the tissue causes fluctuations in this speckle pattern, resulting in blurring of the speckle in those parts of the image where perfusion occurs.

[0004] For example, if the fluctuations are caused by the movement of red blood cells, the blurring may be related to blood flow. In this way, blood perfusion can be imaged in living tissue in a relatively simple way. An example of a prior art clinical perfusion imaging scheme by LSCI is described in the review article "Clinical applications of laser speckle contrast imaging: a review" by W. Heeman et al., J. Biomed. Opt. 24: 8 (2019). The perfusion of other body fluids (e.g., lymphatic perfusion) can be imaged in a similar manner.

[0005] However, LSCI is extremely sensitive to any type of motion. Blurring can be caused not only by the movement of blood flow, but also by any other type of motion, such as movement of tissue due to breathing, heartbeat, muscle contraction, or movement of the camera (especially in hand-held cameras).

[0006] In many medical applications such as diagnosis and surgery, it is expected that LSCI systems can generate accurate, high-resolution blood flow images, especially microcirculation images, in real time, in which motion artifacts are significantly reduced. Therefore, during the processing of the raw speckle images, measures are needed to minimize motion artifacts so that accurate, high-resolution perfusion images can be acquired. This can improve the identification of well-perfused and poorly perfused areas, thereby increasing diagnostic and treatment outcomes.

[0007] Various schemes for reducing motion artifacts in speckle images are known in the prior art. For example, WO2020 / 045015A1 discloses a laser speckle contrast imaging system that can capture near-infrared speckle images and white light images of an imaging target. Simple motion detection schemes can include using reference markers on the image target, tracking feature points in a visible light image, or changes in speckle shape in a speckle image to determine a global motion vector indicating the amount of movement of the image target between two subsequent images. A speckle contrast image can be generated based on the speckle image, and correction can be made for the amount of motion based on the motion vector.

[0008] Motion will be even more prominent in a handheld LSCI system than in a tripod-supported system, for example. For example, Lertsakdadet et al., in their article "Correcting for motion artefact in handheld laserspeckle images", Journal of biomedicaloptics 23(2), March 2018, describe a motion compensation scheme for laser speckle imaging that uses fiducial markers attached to the tissue to be imaged. In many applications it is not possible to use markers.

[0009] Similarly, P. Miao et al., in their article “High resolution cerebral blood flow imaging by registered laser speckle contrast analysis”, IEEE transactions on bio-medical engineering 57(5):1152–1157, describe a method for producing high-resolution LSCI images by registering raw speckle images based on convolution filtering and correlation and interpolation schemes. The registered images are then retrospectively analyzed using temporal laser speckle contrast analysis. However, this registration of raw speckle images requires a lot of computational resources and is therefore not suitable for accurate real-time imaging applications.

[0010] Therefore, it can be seen from the above that there is a need in the art for improved motion compensated laser speckle contrast imaging schemes. In particular, there is a need in the art for improved methods and systems for laser speckle contrast imaging that allow for real-time, robust, label-free, high-resolution perfusion imaging, in particular microcirculation imaging, in which motion artifacts are substantially eliminated or at least reduced. Summary of the invention

[0011] As will be appreciated by those skilled in the art, aspects of the present disclosure may be embodied as systems, methods or computer program products. Therefore, aspects of the present disclosure may take the form of complete hardware embodiments, complete software embodiments (including firmware, resident software, microcode, etc.), or embodiments of combined software and hardware aspects, which may generally be referred to herein as "circuits", "modules" or "systems". The functions described in the present disclosure may be implemented as algorithms executed by a microprocessor of a computer. In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media, which has a computer-readable program code embodied (e.g., stored) thereon.

[0012] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0013] A computer readable signal medium may include a propagated data signal having computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and is capable of conveying, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0014] The program code embodied in the computer readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. The computer program code for performing the operation of various aspects of the present disclosure can be written in any combination of one or more programming languages, including functions such as Java (TM), Scala, C++, Python, etc. or object-oriented programming languages ​​and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer, server, or virtualized server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet).

[0015] Various aspects of the present disclosure are described below with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments. It should be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, particularly a microprocessor or a central processing unit (CPU) or a graphics processing unit (GPU), to produce a machine so that instructions executed by a processor of a computer, other programmable data processing device or other device create a device for implementing the function / action specified in one or more frames of a flowchart and / or block diagram.

[0016] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing device, or other apparatus to act in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0017] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operational steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0018] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments. In this regard, each frame in the flow chart or block diagram can represent a module, a fragment or a part of the code, which includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative embodiments, the function indicated in the frame may not occur in the order indicated in the figure. For example, depending on the function involved, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order. It will also be noted that each frame illustrated in the block diagram and / or flow chart and the combination of frames in the block diagram and / or flow chart can be realized by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action.

[0019] It is an object of embodiments of the present disclosure to reduce or eliminate at least one disadvantage known in the prior art.

[0020] In a first aspect, an embodiment may be directed to a method of motion compensated laser speckle contrast imaging. The method comprises: exposing a target area to a coherent first light of a first wavelength, the target area comprising living tissue; and capturing at least one image sequence. The at least one image sequence comprises a first speckle image, the first speckle image being captured during the exposure with the first light. The method further comprises determining one or more registration parameters of an image registration algorithm for registering the first speckle images with each other. The method may further comprise: determining a registered first speckle image by registering the first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or determining a first speckle contrast image based on the first speckle image, determining a registered speckle contrast image by registering the first speckle contrast image based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast image.

[0021] Thus, a sequence of speckle images or speckle contrast images may be registered or aligned without the use of markers. The registered images may then be combined into a combined speckle contrast image with high resolution and accuracy. The combined image may include information from a plurality of speckle images, such as pixel averages, preferably weighted averages, or the combined image can be, for example, the single most reliable image in a moving window of images, such as the image with the smallest transformation (i.e., closest to the identity transformation using a suitable metric) relative to subsequent images in the sequence of images.

[0022] In this way, the image is less sensitive to noise due to motion. Motion may be caused by, for example, movement of the camera, particularly in handheld systems, or may be caused by movement of the patient, for example due to muscle contractions. Thus, embodiments of the present disclosure may enable or improve speckle contrast imaging in a wide range of applications, including, for example, perfusion imaging of the large intestine, which requires movement of the camera along the entire surface to be imaged, or perfusion imaging of skin burns, where the patient may not be able to remain still due to pain.

[0023] An advantage of the method is that it can be performed in (substantially) real time using commonly available hardware. In some embodiments, there may be a small delay based on, for example, a number of frames (e.g., 20 frames, or 20 frames of sufficient quality), a fixed amount of time (e.g., 1 second), or a time based on a physiological characteristic (e.g., the time of one or two heartbeats or one or two breaths). Such delays are generally not detrimental for clinical use. The physiological characteristic may be based on image analysis of the speckle image, the speckle contrast image, and / or an image from a plurality of images, based on a predetermined constant based on knowledge about physiological phenomena, or may be based on an external input, for example, from a heart rate monitor.

[0024] Another advantage of this method is that it does not require the placement of fiducial markers in the field of view. This is particularly relevant for areas where it is undesirable to place fiducial markers, for example, when imaging internal organs, burned tissue, or brain tissue, or for relatively large image objects that would otherwise require multiple fiducial markers or repeated replacement of markers.

[0025] Because the method corrects and / or compensates for motion of the target region relative to the camera, a more reliable perfusion image can be created. In some cases, reliable images can be acquired more quickly or more easily, such as when imaging a target with irregular motion, and the operator would otherwise have to wait for a period of time with little motion to acquire an image. For example, motion correction can include transforming one or more images based on detected apparent motion. Motion compensation can include combining multiple images, thereby increasing the signal-to-noise ratio.

[0026] By registering the first speckle images or the first speckle contrast images, a plurality of first speckle images from a sequence of first speckle images may be used to determine a combined speckle contrast image having increased contrast and / or spatial resolution compared to individual first speckle contrast images. The number of registered speckle images to be combined may be based on clinical requirements and / or quality parameters of the first speckle images.

[0027] A speckle contrast image may be calculated based on unregistered and therefore untransformed speckle images. Subsequently, the speckle contrast image may be transformed and then combined. The transformation may distort the speckle pattern or parts thereof, for example, the speckle may be enlarged, shrunk or deformed, particularly for transformations more general than just translation or rotation. Therefore, calculating the speckle contrast image based on the untransformed speckle image may prevent the introduction of noise due to the transformation into the speckle contrast image.

[0028] Alternatively, the speckle images may first be registered, and thus transformed, and subsequently a speckle contrast image may be calculated. In this way, a temporal or spatiotemporal speckle contrast may be calculated based on two or more registered speckle images. Using temporal or spatiotemporal speckle contrast may result in a higher spatial resolution. In this embodiment, the images are preferably registered with sub-pixel accuracy.

[0029] Another advantage of this embodiment is that only a single image sequence is required: the same image sequence can be used for image registration (determining the registration parameters), for determining the average weights, and for determining the combined laser speckle contrast image or perfusion image. However, in other embodiments, a second image sequence can be used to determine the registration parameters for image registration and for determining the average weights.

[0030] In an embodiment, the determination of one or more registration parameters is based on a plurality of images, preferably images of the plurality of images are selected from the first speckle image and / or from images associated with the first speckle image and / or from images derived from the first speckle image or images associated with the first speckle image.

[0031] For example, an image in the plurality of images can be an image obtained or derived from a corresponding first speckle image using image processing. For example, the image derived from the first speckle image can represent a texture of the first speckle image, a normalized version of the first speckle image, a filtered (e.g., blurred or sharpened) first speckle image, or any other suitable image derived from the first speckle image. In other embodiments described below, a separate image is captured and associated with the first speckle image. An image in the plurality of images can also be an image derived from an image associated with the first speckle image. As another example, an image in the plurality of images can be an image obtained by transforming the first speckle image, an image derived from the first speckle image, an image associated with the first speckle image, and / or an image derived from an image associated with the first speckle image.

[0032] In an embodiment, the registration parameters are based on a similarity measure of pixel values ​​in one or more pixel groups in each image of a plurality of images of at least one image sequence, the images of the plurality of images being selected from the first speckle image and / or from images associated with the first speckle image.

[0033] One or more pixel groups may cover the entire image or only a portion thereof. Pixel groups in different images may have the same or different sizes. For example, where one pixel group corresponds to a feature, groups corresponding to the same or similar features typically have the same or similar sizes. In a different example, where pixel groups correspond to locations in an image, pixel groups corresponding to the same or similar locations in different images may have different sizes. Matching one or more position-based pixel groups with typically different sizes may also be referred to as template matching.

[0034] An advantage of using groups of pixels is that groups of pixels can often be matched more reliably than entire images, particularly when the groups are relatively small compared to the image size. In addition, if the two images are related by a non-homogeneous transformation, then using one or more groups of pixels can improve the reliability of the image registration, where the transformation of one group of pixels can correspond to a local transformation of the image.

[0035] In an embodiment, the method further comprises: determining a transformed image by transforming the first speckle image and / or an image associated with the first speckle image. The transformation may comprise one or more of: a transformation to the frequency domain, such as a Fourier transform, a Mellin transform, a Laplace transform, a Radon transform, and a coordinate transformation, such as a log-polar transform. In such an embodiment, the registration parameters may be based on a comparison of the transformed images.

[0036] Typically, feature-based, intensity-based and / or frequency-based methods can be used to determine the registration parameters. The advantage of using a feature-based method is that it involves real-world features and is therefore less sensitive to image artifacts. In addition, features can be selected so that only high-quality data is used to determine the registration parameters, thereby potentially increasing accuracy and / or robustness. In addition, feature-based methods are typically able to handle both translation and rotation of the image and scaling, particularly when the changes are relatively small compared to the image size. In principle, feature-based methods can be used together with intensity-based or frequency-based methods to determine features. The advantage of intensity-based methods is that they are typically simple to implement and computationally inexpensive to perform. The advantage of frequency-based methods is that information from the entire image is used, making the method less sensitive to local interference and image artifacts.

[0037] In an embodiment, the transform is a transform to the frequency domain, preferably a Fourier transform. In such an embodiment, the comparison of the transformed images may comprise: determining a cross-correlation of the transformed images; determining a transform of the cross-correlation to the spatial domain, preferably using an inverse Fourier transform; and determining a peak in the cross-correlation in the spatial domain. The position of the determined peak corresponds to the translation between the two images.

[0038] In an embodiment, the transformation is a log-polar transformation; wherein the comparison of the transformed images comprises: determining a shift of the transformed images relative to each other; and wherein determining the registration parameters comprises determining a rotation and / or scaling based on the determined shift. An advantage of using a log-polar transformation is that a horizontal shift of the transformed image corresponds to a rotation of the untransformed image, and a vertical shift of the transformed image corresponds to a scaling of the untransformed image. Thus, rotation and scaling, in particular global scaling and rotation, can be determined in a relatively straightforward manner. The log-polar transformation is often applied in combination with different methods to determine the translation of the image.

[0039] The combination of the Fourier transform and the logpolar transform is sometimes called the Fourier–Mellin transform.

[0040] For example, the determined registration parameters may include a translation vector, a rotation angle, and a scale factor. Based on these registration parameters, the first speckle images may be aligned (transformed) with each other. These registration parameters can also be used to determine the weights of a weighted average with which the first speckle image or the first speckle contrast image may be combined. Generally, there is no need to explicitly determine the alignment vectors of the plurality of image regions, neither for the image registration itself nor for the weight determination.

[0041] An advantage of using a large image region to determine the registration parameters, as is typical in, for example, template matching or frequency-based methods, is that noise can be suppressed compared to, for example, methods using multiple small regions. This is particularly relevant for registration parameters such as rotation and scale when determined using a log-polar transform or a Fourier-Mellin transform, as rotation and scale may be difficult to determine accurately in the untransformed spatial domain.

[0042] In an embodiment, the method further comprises determining a plurality of masks for the first speckle image. Each mask in the plurality of masks is associated with a corresponding first speckle image. Each mask in the plurality of masks may associate a reliability score with one or more pixels in the associated first speckle image. In such an embodiment, the determination of the combined speckle contrast may be directly or indirectly based on the plurality of masks.

[0043] The reliability score may be binary. Alternatively, the reliability score may have more than two potential values, such as 255 (byte value) or a floating point value (typically selected between 0 and 1). Different image types (e.g., speckle images, speckle contrast images, and combined speckle contrast images) may use different mask types, such as a binary mask for speckle images and a byte value mask for speckle contrast images. The mask may be determined for the entire image or only for one or more regions of interest in the image. For example, a speckle contrast value may have a reliability score based on the number of unreliable input pixel values ​​used to determine the speckle contrast. The reliability score can also be based on a registration parameter (e.g., based on the amount of motion determined using, for example, optical flow). Depending on whether the registration parameter is determined globally (for the entire image), regionally (for image blocks or groups of pixels), or locally (for individual pixels), the reliability score can similarly be determined globally, regionally, or locally. Reliability scores from different sources may be combined into a single reliability score.

[0044] In principle, the mask may be applied before or after image registration (and application of the corresponding transformation) and before or after computing the speckle contrast.The information encoded in the mask may also be used several times during the computation and may be changed during the computation.

[0045] In an embodiment, the method further comprises determining a plurality of registration masks by registering the plurality of masks based on the one or more registration parameters and the image registration algorithm. In such an embodiment, the determination of the combined speckle contrast may be based on the plurality of registered masks.

[0046] In an embodiment, at least one of the plurality of masks is based on an artifact, preferably a specular artifact, identified in the corresponding speckle image. Artifacts may be due to several sources. For example, specular artifacts may occur (particularly in endoscopic / laparoscopic settings) where pixels or groups of pixels are fully saturated and thus no contrast may be calculated. Artifacts may also be due to, for example, malfunctioning pixel elements in the camera, stains on the lens that gathers the light, etc.

[0047] Artifacts may be identified by identifying deviated input pixel values ​​and / or deviated speckle contrast values. For example, pixel values ​​above a predetermined absolute or relative upper threshold or values ​​below a predetermined absolute or relative lower threshold may be marked as deviated pixel values. Additionally or alternatively, pixel contrast values ​​below or above a corresponding absolute or relative lower or upper threshold may be identified. For example, the relative threshold may be based on an analysis of the environment of the pixel, such as a mean or median or other statistical representation.

[0048] In an embodiment, at least one of the plurality of masks is based on pixels identified as not representing living tissue. For example, an image may include pixels representing surgical instruments (particularly in an open surgical setting), clamps, sutures, gauze, etc. It can be beneficial not to calculate speckle contrast values ​​or perfusion values ​​for these pixels, or at least not to display them.

[0049] In such embodiments, identifying pixels that are not representative of living tissue may be based on an image recognition algorithm. Such image recognition algorithms are known in the art. The image recognition algorithm may use a speckle image and / or a speckle contrast image as input. In embodiments where at least one sequence of images further comprises other images (e.g., white light images) in addition to the speckle image, additionally or alternatively, pixels that are not representative of living tissue may be identified in the other images. The identified pixels that are not representative of living tissue may be used to improve perfusion calculations and / or determine overall apparent motion in the living tissue, for example, by determining the inverse of a perfusion value determined for the pixels that are not representative of living tissue and using the inverse to determine a motion correction value.

[0050] In an embodiment, the method further comprises exposing the target area to a second light of one or more second wavelengths, preferably coherent light of the second wavelength or light of a plurality of second wavelengths comprising the visible spectrum, wherein the exposure to the second light is performed alternately with the exposure to the first light or simultaneously with the exposure to the first light. In this embodiment, the at least one image sequence comprises a second image sequence, the second images being captured during the exposure to the second light; and the plurality of images are selected from the second image sequence, each of the second images or images derived therefrom being associated with the first speckle image.

[0051] Instead of the original second image, an image derived from the second image may be associated with the first speckle image, such as a texture image, a normalized image or an image pre-processed in another way.

[0052] Therefore, in an embodiment, a method of laser speckle contrast imaging comprises exposing a target area to coherent first light of a first wavelength and second light of one or more second wavelengths, preferably coherent light of the second wavelength or light of a plurality of second wavelengths comprising a visible spectrum, alternately or simultaneously, the target area preferably comprising living tissue. The method may further comprise: capturing a first speckle image sequence during the exposure with the first light; and capturing a second image sequence during the exposure with the second light, the speckle images of the first speckle image sequence being associated with images of the second image sequence. One or more registration parameters of a registration algorithm for registering at least a portion of the first speckle image sequence may be determined based on a similarity measure of pixel values ​​of pixel groups in at least a portion of the second image sequence associated with the first speckle image. The method may further comprise determining a registered first speckle image by registering at least a portion of the first speckle image sequence based on the one or more registration parameters and the registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image. Alternatively, the method may include determining a first speckle contrast image based on at least a portion of the sequence of first speckle images, determining a registered speckle contrast image by registering the first speckle contrast images based on one or more registration parameters and a registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast images.

[0053] In the first embodiment defined above, the first light and the second light are the same light with the same wavelength, and the first speckle image sequence and the second image sequence are the same image sequence. However, in an advantageous embodiment, at least one of the one or more second wavelengths is different from the first wavelength, and therefore, the second image is different from the first speckle image. The first speckle image and the associated second image can also be different parts of a single image. The images in the image sequence may be frames in a video stream or in a multi-frame snapshot. In the present disclosure, the second image may also be referred to as a correction image. Typically, at least one pixel group in each of at least a part of the second image sequence is used to determine one or more registration parameters.

[0054] Thus, the capture of a first wavelength and a first speckle image sequence may be optimized for speckle contrast imaging (e.g., optimized exposure time) according to the quantity to be measured; while the capture of one or more second wavelengths and a second image sequence may be optimized to obtain images that can be easily and accurately registered, for example by ensuring high contrast between anatomical features such as blood vessels and normal tissue. Typical combinations may be a speckle image acquired using infrared light and a second image acquired using white light, or a speckle image acquired using red light and a second image acquired using green or blue light, but of course, other combinations are possible.

[0055] In an embodiment, the pixel group may represent a predetermined feature in the plurality of images, the predetermined feature preferably being associated with an object in the target region, preferably associated with an anatomical structure in the target region.The pixel group may be selected by a feature detection algorithm.

[0056] Features may be features associated with physical objects in the target area, such as features associated with blood vessels, rather than, for example, image features not directly associated with the object, such as overexposed image portions or speckle. Predetermined features may be determined, for example, by belonging to a class of features, such as corners or areas with large intensity differences. They may be further determined, for example, by quality metrics, restrictions on mutual distances between features, etc. In some embodiments, a neighborhood of one or more determined features may be used to determine the alignment vector and / or the transformation.

[0057] Determining the displacement based on a relatively small number of features compared to the total number of pixels can substantially reduce the computation time while still giving accurate results. This is particularly true for relatively simple motions, where for example the entire target area is displaced due to the motion of the camera.

[0058] In an embodiment, the method may further include filtering the plurality of images using a filter adapted to increase the probability that a group of pixels represents a feature corresponding to an anatomical feature. For example, the filter may determine overexposed and / or underexposed regions and / or other image artifacts, and a mask may be created based on these regions or artifacts. Thus, features associated with these regions or artifacts may be prevented from being determined.

[0059] In an embodiment, determining the registration parameters based on the similarity of pixel values ​​of the pixel groups may include: for each pixel group in an image from the plurality of images, determining a convolution or cross-correlation with at least a portion of a different image from the plurality of images. The method may also include determining a polynomial expansion to model pixel values ​​in a neighborhood of pixels in the plurality of images, comparing expansion coefficients, and determining an alignment vector based on the comparison.

[0060] In an embodiment, determining one or more alignment parameters may include: determining a plurality of associated pixel groups based on a similarity measure, each pixel group belonging to a different image from a plurality of images; determining a plurality of alignment vectors based on positions of the pixel groups relative to corresponding images from the plurality of images, the alignment vectors representing movement of a target region relative to an image sensor; and determining the alignment parameters based on the plurality of alignment vectors.

[0061] By determining alignment vectors for a plurality of features or corresponding feature pairs, any movement of the camera relative to the imaging target can be determined and corrected. The alignment vectors can be used to determine, for example, an affine transformation, a projective transformation, or a homography, which can correct, for example, translation, rotation, scaling, and shearing of the image based solely on the image data. Thus, the method does not require information about, for example, the distance between the camera and the target, the angle of incidence, etc.

[0062] The determination of the alignment vector and / or the determination of the one or more transformations may be based on optical flow parameters determined using a suitable optical flow algorithm (eg, a dense optical flow algorithm or a sparse optical flow algorithm).

[0063] In an embodiment, determining the combined speckle contrast image may further comprise: calculating an average of the registered first speckle image and the registered first speckle contrast image, respectively, the average being preferably a weighted average, the weight of the image preferably being based on a registration parameter or on a relative magnitude of the speckle contrast. Alternatively, combining the speckle image or the speckle contrast image may comprise: filtering at least a portion of the first speckle image (respectively the first speckle contrast image) using, for example, a median filter or a minimum or maximum filter. The weight for the weighted average may be based on, for example, a quantity derived from the speckle contrast or derived from the alignment vector.

[0064] In an embodiment, calculating the combined laser speckle contrast image may comprise calculating an average, preferably a weighted average, of the sequence of registered speckle images.

[0065] In an embodiment, the average is a weighted average, and masked pixels have a weight equal to zero. If the mask defines a multi-valued reliability score, the weights may be based on the reliability score, where pixels with higher reliability scores have higher weights than pixels with lower reliability scores.

[0066] In an embodiment, the method further comprises determining a combined speckle contrast image mask. The combined speckle contrast image mask may indicate whether at most a predetermined percentage of input pixels are masked. For example, the weighted average is calculated only if less than the predetermined percentage of input pixels are masked, and wherein if more than the predetermined percentage of input pixels are masked, the pixel is marked as having an invalid pixel value.

[0067] Alternatively, the combined speckle contrast image mask may indicate a reliability score for pixels in the combined speckle contrast image mask. The reliability score may be based on the fraction of masked pixels in the weighted average, and / or based on the reliability score of the corresponding mask.

[0068] Speckle contrast or perfusion values ​​are typically displayed as an overlay on the captured image. There are various options for handling pixels in the overlay for which valid perfusion values ​​cannot be calculated. For example, pixels marked as having invalid pixel values ​​are removed or rendered as transparent, assigned error values, or assigned interpolated values ​​based on surrounding pixel values. Two or more of these options can be combined, for example, based on the cause of the invalid pixel value and / or based on the size of the area of ​​connected invalid pixel values. For example, an invalid pixel area due to the presence of non-tissue objects (e.g., surgical instruments) can be shown as transparent so that the instruments or other non-tissue objects can be clearly seen. As another example, small invalid pixel areas can be filled using interpolation to provide a clean image with the most likely correct values, while large invalid pixel areas can be assigned error values, indicating that valid perfusion data was not obtained.

[0069] If each pixel, or at least each pixel in a relevant area, is assigned a reliability score, the reliability score may be rendered with varying transparency (e.g., using an alpha channel), with reliable pixels having low transparency (high opacity) and unreliable pixels having high transparency (low opacity).

[0070] In an embodiment, the method may further comprise: for each first speckle image or each first speckle contrast image associated with an image from the plurality of images, determining a transformation size associated with the corresponding first speckle image or first speckle contrast image based on the plurality of alignment vectors, preferably based on the lengths of the plurality of alignment vectors, and / or based on parameters defining the determined transformation. The weighted average may be determined using a weight, the weight being based on the determined transformation size associated with the corresponding first speckle contrast image, preferably the weight being inversely related to the determined transformation size.

[0071] A weight based on the size or amount of displacement or based on the size or amount of transformation can be quickly determined for each image independently of the other images. The transformation size can be based on, for example, the norm of a matrix representing the transformation, or the norm of a matrix representing the difference between a transformation and an identity transformation. The transformation size can also be based on, for example, a statistically representative measure of the alignment vectors, such as the mean, median, nth percentile, or maximum alignment vector length. Images with a large amount of displacement are typically noisier and can therefore be assigned a lower weight, thereby improving the quality of the combined image.

[0072] In an embodiment, the method may further comprise: for each first speckle contrast image, determining a normalized amount of speckle contrast or a change in speckle contrast relative to one or more previous and / or subsequent images in the sequence of first speckle contrast images. The weighted average may be determined using a weight based on the determined normalized amount of speckle contrast or the determined change in speckle contrast associated with the corresponding first speckle contrast image. Alternatively or additionally, the weight may be determined based on the normalized amount of speckle contrast or the change in speckle contrast of the second speckle contrast image.

[0073] Weights based on differences or changes (especially sudden changes) in speckle contrast can be indicative of image quality. In general, speckle contrast, and therefore these weights, can be affected by various factors in the overall system, such as motion of the camera relative to the target area, movement of the optical fiber, or other factors affecting optical path length or fluctuating lighting conditions. Therefore, using weights based on speckle contrast, a higher quality combined image can be obtained. In general, speckle contrast is determined in arbitrary units, so the weights can be determined by analyzing a sequence of speckle images. Since speckle contrast is negatively correlated with perfusion, perfusion units based on speckle contrast can be similarly used.

[0074] In an embodiment, the algorithm may be applied to a predefined region of interest in the field of view of the camera. Such a region of interest may be determined by a user, or may be predetermined. For example, the outer border of the image may be ignored, and / or hidden from view, for example to prevent the transformed image border from being visible. Applying the algorithm or a portion of the algorithm to only a portion of the image may be faster. The region of interest may be transformed based on the determined transformation. In various embodiments, the algorithm may be applied to the entire image.

[0075] In an embodiment, the plurality of images may be a sequence of first speckle images. In such an embodiment, the first wavelength is preferably a wavelength in the green or blue part of the electromagnetic spectrum. In this way, a balance may be achieved between a good speckle signal and good visual distinctiveness (i.e. high contrast of anatomical features, which is distinguishable from high speckle contrast), which is advantageous for determining features. Thus, no pre-processing step is required to increase the contrast of the speckle image.

[0076] Alternatively, a first wavelength in the red portion of the electromagnetic spectrum, preferably in the range of 600-700 nm, more preferably in the range of 620-660 nm, or in the infrared portion of the electromagnetic spectrum, preferably in the range of 700-1200 nm, may be used. Depending on the tissue type and imaging parameters such as exposure time, the visual distinctiveness may be sufficient to adequately determine the features. Since red and infrared light are mostly reflected by red blood cells, these wavelengths result in speckle with relatively high intensity and are therefore well suited for speckle contrast imaging of blood flow.

[0077] As in these embodiments, the first speckle image and the image from the plurality of images are the same image, the image can be acquired with a relatively simple system, requiring only a single light source and a single camera.

[0078] In an embodiment, the at least second wavelength of light may be light of at least a second wavelength different from the first wavelength, preferably coherent light of a predetermined second wavelength, preferably in the green or blue part of the electromagnetic spectrum, preferably in the range of 380-590 nm, more preferably in the range of 470-570 nm, even more preferably in the range of 520-560 nm. Blue light or in particular green light may result in high contrast or visual distinctiveness because it is more strongly absorbed in blood vessels than in normal tissue. Therefore, features related to blood vessels, such as edges or corners, may be used to determine the registration parameters.

[0079] Since the first speckle image itself is inherently noisy (in terms of imaging of anatomical features), it is preferable to use a second image based on a different wavelength to determine the alignment vector. In this way, the first speckle image can be acquired based on light selected to optimize the speckle contrast signal, while the second image can be acquired based on light selected to optimize visual distinctiveness. Such a system is particularly advantageous for imaging relatively deeply perfused tissue (e.g., skin). In such tissue, most green or blue light will not penetrate deep enough to interact with blood cells, thereby producing a relatively noise-free image.

[0080] In an embodiment, the first wavelength is preferably a wavelength in the red part of the electromagnetic spectrum, preferably in the range of 600-700 nm, more preferably in the range of 620-660 nm, or a wavelength in the infrared part of the electromagnetic spectrum, preferably in the range of 700-1200 nm. Red light and infrared light are mostly reflected by red blood cells and are therefore very suitable for speckle contrast imaging of blood flow. Infrared light has a greater penetration depth than red light. Red light can be more easily integrated into existing systems, using for example the red channel of an RGB camera to acquire a speckle image.

[0081] Preferably, the first wavelength is selected to be scattered or reflected by the fluid of interest; for example, red light or near infrared light can be used to image blood in a vessel. Preferably, the first wavelength can be selected based on the desired penetration depth. Light with a relatively high penetration depth can allow light scattered by the body fluid of interest to be detected with a sufficient signal-to-noise ratio even at a certain depth in the imaged tissue.

[0082] Preferably, the second wavelength is selected to provide an image with high visual distinctiveness, thereby producing consistent features on the tissue surface in the image. For example, green light can be used to image blood vessels in internal organs because green light is generally absorbed more strongly by blood than by tissue. The light of the second wavelength can be coherent light or incoherent light. The second image can also be based on multiple wavelengths, for example white light can be used. The light of the second wavelength can be generated by, for example, a second coherent light source. Alternatively, the light of the first wavelength and the light of the second wavelength can be generated by a single coherent light source, which is configured to generate coherent light of multiple wavelengths.

[0083] In an embodiment, the second image sequence may be a second speckle image sequence, and the method may further include: determining a second speckle contrast image based on the second speckle image sequence; and adjusting or correcting the first speckle contrast image based on a change in speckle contrast amplitude in the second speckle contrast image sequence.

[0084] Multispectral coherence correction (also known as dual laser correction) can remove or reduce noise in a first speckle contrast image by adjusting the speckle contrast determined in the first speckle image based on changes in the speckle contrast determined in a second image sequence. The adjustment can be based on a predetermined correlation between the speckle contrast of the first speckle contrast image and the speckle contrast of the second speckle contrast image.

[0085] By using the second image based on the second wavelength for both multispectral coherence correction and for image registration, multispectral coherence correction can be advantageously combined with image registration using the second image. In such an embodiment, the second wavelength preferably has a relatively small penetration depth. In this way, the second image can include information mainly related to the surface of the target area. This is especially true for tissues that have little perfusion close to the surface (such as skin, scar tissue, and some tumor types).

[0086] In an embodiment, the method may further comprise: dividing each image in at least a portion of the sequence of the first speckle image, respectively first speckle contrast images, and each image in the plurality of images into a plurality of regions, preferably into disjoint regions. Preferably, the regions in an image from the plurality of images correspond to regions in the associated first speckle image, respectively first speckle contrast image. Determining the registration parameter may comprise determining the registration parameter for each region, and determining the sequence of registered first speckle images, respectively first speckle contrast images may comprise registering each region of the first speckle image, respectively first speckle contrast image based on a transformation according to the corresponding region in an image from the plurality of images.

[0087] Regions may be determined based on, for example, the geometry of the image (eg, a grid of rectangular or triangular regions) or based on image characteristics (eg, light intensities or groups of pixels that appear to belong to anatomical structures).

[0088] In this way, local motion in portions of the images can be corrected, resulting in a higher quality combined image. Local motion is typically caused by motion in the object, for example due to person movement, breathing, heartbeat, or muscle contraction, such as peristaltic motion in the lower abdomen. Regions can be as small as a single pixel. If a weighted average is used to combine two or more images, weights can be assigned to each region individually and / or to the images as a whole. The combined image can also be region-based, or done on an image-by-image basis.

[0089] In an embodiment, the target area may include a perfused organ, preferably perfused by body fluid, more preferably perfused by blood and / or lymph, and / or may include one or more blood vessels and / or lymphatic vessels. The method may further include calculating a perfusion intensity, preferably a blood perfusion intensity or a lymphatic perfusion intensity based on the combined speckle image.

[0090] The method may also include post-processing the image, such as thresholding, pseudo-colorizing, overlaying on other images, such as a white light image, and / or displaying a combined image or an image derived therefrom.

[0091] In a second aspect, an embodiment may be directed to a hardware module for an imaging device, preferably a medical imaging device. The hardware module may include a first light source for exposing a target area to a coherent first light of a first wavelength, the target area preferably including living tissue. The hardware module may also include an image sensor system having one or more image sensors for capturing at least one image sequence, the at least one image sequence including a first speckle image, the first speckle image being captured during exposure to the first light. The hardware module may further include a computer-readable storage medium having a computer-readable program code embedded therein, and a processor, preferably a microprocessor, more preferably a graphics processing unit, coupled to the computer-readable storage medium, wherein, in response to executing the computer-readable program code, the processor is configured to: determine one or more registration parameters of an image registration algorithm for registering the first speckle images with each other, the registration parameters being based on a similarity measure of pixel values ​​of pixel groups in a plurality of images, the images in the plurality of images being selected from the first laser speckle image or associated with the first speckle image, the registration parameters preferably defining one of: a homography, a projective transformation, or an affine transformation; and determine a registered first speckle image by registering the first speckle images based on the one or more registration parameters and the image registration algorithm, and determine a combined speckle contrast image based on the registered first speckle image; or determine the first speckle contrast image based on the first speckle image, determine a registered speckle contrast image by registering the first speckle contrast images based on the one or more registration parameters and the image registration algorithm, and determine a combined speckle contrast image based on the registered first speckle contrast image.

[0092] In an embodiment, the hardware module may include a second light source for illuminating the target area with light of at least a second wavelength different from the first wavelength simultaneously or alternately with the first light source. The at least one image sensor may be configured to capture a second sequence of images, the second images being captured during exposure with the second light. The plurality of images may be selected from the second sequence of images, each of the second images being associated with the first speckle image.

[0093] The image sensor system may include a first image sensor for capturing the first sequence of images and a second image sensor for capturing the second sequence of images, or a single image sensor for capturing both the first sequence of images and the second sequence of images.

[0094] In an embodiment, the hardware module may further comprise a display for displaying the combined speckle image and / or images derived therefrom, preferably a perfusion intensity image.Alternatively or additionally, the hardware module may comprise a video output for outputting the combined speckle image and / or images derived therefrom.

[0095] The image sensor system may include a first image sensor for capturing images at a first wavelength and a second image sensor for capturing images at least at a second wavelength. The first image sensor and the second image sensor may be the same image sensor, different parts of a single image sensor (e.g., red and green channels from an RGB camera), or different image sensors. The module may also include optics to direct light from the first light source and from an optional second light source to a target area and / or to direct light from a target area to the first image sensor and the second image sensor.

[0096] The present disclosure also relates to a medical imaging device comprising such a hardware module, preferably an endoscope, a laparoscope, a surgical robot, a handheld laser speckle contrast imaging device or an open surgical laser speckle contrast imaging system.

[0097] In another aspect, the present disclosure relates to a computing module for a laser speckle imaging system, the computing module comprising a computer-readable storage medium and a processor, the computer-readable storage medium having at least a portion of a program embedded therein, the processor coupled to the computer-readable storage medium, the processor preferably being a microprocessor, more preferably being a graphics processing unit, wherein, in response to executing the computer-readable storage code, the processor is configured to perform executable operations. The executable operations may include:

[0098] receiving at least one image sequence, the at least one image sequence comprising a first speckle image, the first speckle image being captured during exposure of a target area to coherent first light of a first wavelength, the target area comprising living tissue; determining one or more registration parameters of an image registration algorithm for registering the first speckle images with each other, the registration parameters being based on a similarity measure of pixel values ​​of groups of pixels in a plurality of images of the at least one image sequence, the images of the plurality of images being selected from or associated with the first speckle image, the registration parameters preferably defining one of: a homography, a projective transformation or an affine transformation; and determining a registered first speckle image by registering the first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or determining the first speckle contrast image based on the first speckle image, determining a registered speckle contrast image by registering the first speckle contrast image based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast image.

[0099] Such a computational module may, for example, be added to an existing or new medical imaging device, such as a laparoscope or an endoscope, in order to improve laser speckle contrast imaging, in particular perfusion imaging. In an embodiment, the method steps described in the present disclosure may be performed by a processor in a device for coupling coherent light into an endoscopic system. Such a device may be coupled between a light source and a video processor of an endoscopic system and an endoscope (e.g., a laparoscope) of the endoscopic system. Thus, the coupling device may add laser speckle imaging capabilities to the endoscopic system. Such a coupling device is described in more detail in Netherlands patent application NL2026240, which is incorporated herein by reference.

[0100] In an embodiment, at least one sequence of images comprises a sequence of second images, the second images being captured during exposure with a second light, the second light having one or more second wavelengths, preferably the second light being coherent light of the second wavelength or the second light comprising a plurality of second wavelengths of the visible spectrum, wherein the exposure to the second light is performed alternately with the exposure to the first light or simultaneously with the exposure to the first light. The plurality of images are selected from the sequence of second images, each of the second images being associated with a first speckle image.

[0101] The present disclosure may also relate to a computer program or a computer program suite, comprising at least one software code portion or a computer program product storing at least one software code portion, which software code portion is configured to perform any of the above method steps when running on a computer system.

[0102] The present disclosure may also relate to a non-transitory computer-readable storage medium storing at least one software code portion, which when executed or processed by a computer is configured to perform any of the method steps as described above.

[0103] The embodiments will be further described with reference to the attached schematic drawings.It should be understood that the present disclosure is not limited in any way to these specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] The following description of the drawings of the specific embodiments is merely exemplary in nature and is not intended to limit the present teachings, their application, or uses.

[0105] Figure 1A schematically depicts a system for motion compensated laser speckle contrast imaging according to an embodiment, and Figure 1B-Figure 1E depicts a flow chart for laser speckle contrast imaging according to an embodiment;

[0106] Figure 2A-2C The original speckle image, the laser speckle contrast image based on the original speckle image, and the perfusion image based on the laser speckle contrast image are depicted;

[0107] Figure 3A-3D depicts a flow chart for motion compensated laser speckle contrast imaging according to an embodiment;

[0108] Figure 4A and Figure 4B depicts a flow chart for motion compensated laser speckle contrast imaging according to an embodiment;

[0109] Figure 5A-5F A method for determining registration parameters according to an embodiment is depicted;

[0110] Figure 6A-6D A method for determining registration parameters according to an embodiment is depicted;

[0111] Fig. 7A and Figure 7B A flow chart depicting laser speckle contrast imaging for motion compensation combining more than two raw speckle images according to an embodiment is depicted;

[0112] Figure 8 depicts a flow chart for calculating a corrected laser speckle contrast image according to an embodiment;

[0113] Fig. 9A and Fig. 9B schematically depicts determining a motion compensated speckle contrast image based on a weighted average according to an embodiment; and

[0114] Fig.10 is a block diagram illustrating an exemplary data processing system that can be used to execute the methods and software products described in this application. DETAILED DESCRIPTION

[0115] Laser speckle contrast images can be based on spatial contrast, temporal contrast, or a combination. Typically, using spatial contrast results in high temporal resolution but relatively low spatial resolution. In addition, individual images may suffer from quality losses, e.g. due to motion or illumination artifacts, resulting in image quality that may vary from image to image. On the other hand, using temporal contrast is associated with relatively high spatial resolution and relatively low temporal resolution. However, the quality of temporal contrast may be strongly affected by the motion of the target relative to the camera, which may cause pixels to be incorrectly combined. Hybrid methods may share some advantages and disadvantages of both methods.

[0116] In the present disclosure, a speckle image may also be referred to as a raw speckle image to better distinguish between a (raw) speckle image and a speckle contrast image. Thus, the term "raw speckle image" may refer to an image representing a speckle pattern, wherein pixels have pixel values ​​representing light intensity. A raw speckle image may be an unprocessed image or a (pre)processed image. The term "speckle contrast image" may be used to refer to a processed speckle image having pixels having pixel values ​​representing speckle contrast magnitude, typically a relative standard deviation over a predefined neighborhood of pixels.

[0117] Figure 1A A system 100 for motion compensated laser speckle contrast imaging according to an embodiment is schematically depicted. The system may include a first light source 104 for generating coherent light (e.g., laser) of a first wavelength for illuminating a target area 102. The target is preferably living tissue, such as skin, intestine, or brain tissue. The first wavelength may be selected to interact with a body fluid (e.g., blood or lymph) that may move through the target. The first wavelength may be in the red or (near) infrared part of the electromagnetic spectrum, for example in the range of 600-700 nm, preferably in the range of 620-660 nm, or in the range of 700-1200 nm. The first wavelength may be selected based on the body fluid of interest and / or the tissue being imaged. The first wavelength may also be selected based on the characteristics of the imaging sensor. Different quantities of interest may be imaged, for example, flow in individual large or small vessels, or microvascular perfusion of a target area, depending in particular on the exposure time, the size of the imaging area, the speckle size, and the image resolution.

[0118] In embodiments, the system may further include a second light source 106 for generating light of at least a second wavelength, preferably including light in the green portion of the electromagnetic spectrum, for illuminating the target area 102. At least the second wavelength may be selected to include a wavelength that creates an image with high visual distinctiveness (i.e., high contrast of anatomical features). The second wavelength may be selected based on the tissue in the target area. Typically, the light of at least the second wavelength may be coherent light or incoherent light, and may be monochromatic, such as blue or green narrow-band imaging light, or polychromatic light, such as white light. In other embodiments, only the first light source is used. In Figure 1C In the illustrated embodiment, at least the light of the second wavelength is monochromatic coherent light. The second wavelength can be generated by, for example, a second coherent light source. Alternatively, the light of the first wavelength and the light of the second wavelength can be generated by a single coherent light source that is configured to generate coherent light of multiple wavelengths.

[0119] The system may also include one or more image sensors 108 for capturing images associated with light of a first wavelength, and where applicable, light of at least a second wavelength, the first wavelength and the at least second wavelength having interacted with the target in the target area. In various embodiments, the system may include multiple cameras, such as a first camera for acquiring a first raw speckle image associated with the first wavelength and a second camera for acquiring a corrected image associated with the second wavelength. The system may also include additional optics, such as optical fibers, lenses, or beam splitters, to direct light from the one or more light sources to the target area and from the target area to the one or more image sensors.

[0120] When a target is illuminated with coherent light, a laser speckle pattern may be formed by self-interference. The image may be received and processed by the processing unit 110. Figure 1B-Figure 1D The examples are described in more detail. The processing unit may output the processed image to, for example, a display or computer 112 substantially in real time. The processing unit may be a separate unit or may be part of a computer. The processed image may be displayed by a display or computer.

[0121] In an embodiment, an endoscope or laparoscope may be used to direct light to a target area and acquire an image.The one or more light sources, the one or more image sensors, the processing unit and the display may, for example, be part of an endoscope system.

[0122] Figure 1B-Figure 1E A flow chart of laser speckle contrast imaging for motion compensation according to an embodiment is depicted.Alternatives mentioned with respect to one of the embodiments can also be applied to the other embodiments, unless such combination would result in a contradictory description.

[0123] Figure 1B A flow chart of laser speckle contrast imaging for motion compensation according to an embodiment is depicted. In this embodiment, only coherent light of a first wavelength generated by a first light source 104 is used, which can be, for example, green light or preferably red light. The one or more image sensors 108 are single image sensors, typically monochrome image sensors optimized for or at least suitable for the wavelength used. As described above and will be described below Figure 1C and Figure 1D As shown in the examples, other embodiments may use different configurations.

[0124] In a first step 120, a sequence of raw speckle images, for example captured or received from an external source, is obtained. In this example, these speckle images are also used as correction images. Based on each first raw speckle image, a first speckle contrast image may be calculated 126. The speckle contrast may be determined in any suitable manner, for example by convolution with a predetermined convolution kernel, or by determining a relative standard deviation of pixel value intensities in a sliding window. Since speckle contrast is related to perfusion, a perfusion unit weight may be determined based on the speckle contrast value of the first speckle contrast image.

[0125] In a single wavelength embodiment, the original speckle image or the speckle contrast image may be used as the correction image. In optional step 129, the correction image may be transformed. Based on the optionally transformed correction image, the registration parameters may be calculated. For example, in each correction image, the position of a predetermined object feature may be determined. Based on the position of the predetermined object feature in two or more correction images, an alignment vector identifying the motion of the target area may be determined, for example using an optical flow algorithm. A (sparse) optical flow algorithm may be used to determine the alignment vector based on the selected features. Other embodiments may use, for example, a dense optical flow algorithm; in that case, no features need to be determined.

[0126] Based on the alignment vector, a transformation for registering the images in the second image sequence may be determined. Preferably, the transformation is selected from the class of homographies, projective transformations or affine transformations. Optionally, the optical flow weights may be determined based on the alignment vector or parameters defining the transformation.

[0127] Alternatively, the registration parameters can be calculated in any other suitable manner, such as using template matching, phase matching, matching in a log-polar coordinate system, etc. Figure 5A-5D Various examples of suitable methods for determining registration parameters are described in more detail in FIG. 6 .

[0128] The determined registration parameters can then be used to register 132 or geometrically align the speckle contrast images to each other, resulting in a registered first speckle contrast image. In an alternative embodiment, the raw speckle images may be registered before speckle contrast is calculated. However, such an embodiment is less preferred as image registration may affect pixel values ​​and therefore contrast.

[0129] Subsequently, the registered first speckle contrast images may be combined 134, for example using a temporal filter. The temporal filter may include averaging a plurality of first speckle contrast images. The averaging may be a weighted average, wherein the weights are based on optical flow weights and / or on perfusion unit weights. In an alternative embodiment, the registered raw speckle images may be combined using a temporal filter, and a speckle contrast image may be determined based on the combined rendered speckle images. This results in a motion compensated speckle contrast image 136.

[0130] The combined first raw speckle image may then be post-processed, for example, converted to perfusion values, thresholded to indicate areas with high or low perfusion, overlaid on a white light image of the target area, etc. The post-processing may be performed by, for example, the processing unit 110 or the computer 112 . Figure 1C A flow chart for motion compensated laser speckle contrast imaging is depicted in accordance with an embodiment. In the depicted embodiment, the light of the first wavelength generated by the first light source 104 is red light, and the light of at least the second wavelength generated by the second light source 106 is coherent green light. The one or more image sensors 108 are a single sensor including red, green, and blue channels. The first wavelength and the second wavelength are selected to minimize crosstalk. The signal in the red channel caused by the green light is substantially smaller than the signal caused by the red light; and the signal in the green channel caused by the red light is substantially smaller than the signal caused by the green light. As described above, other embodiments may use different configurations.

[0131] In a first step 140, a sequence of RGB images is received. A first sequence of first raw speckle images may be extracted 142 from a red channel of the sequence of RGB images, and a second sequence of corrected images may be extracted 144 from a green channel of the RGB images. The sequence of corrected images may be a second sequence of second raw speckle images. Each first raw speckle image may be associated with a corrected image extracted from the same RGB image. In other embodiments, the first raw speckle images and the corrected images may be acquired by different cameras, by different sensors of a multi-sensor camera (e.g., a 3CCD camera), by other (color) channels of a single camera (e.g., a YUV camera), or, if the target is illuminated alternately with light of a first wavelength and light of at least a second wavelength, by a single monochrome camera.

[0132] Based on each first raw speckle image, a first speckle contrast image may be calculated 146. Optionally, based on each second raw speckle image, a second speckle contrast image may be calculated 148. The speckle contrast may be determined in any suitable manner, such as by convolution with a predetermined convolution kernel, or by determining a relative standard deviation of pixel value intensities in a sliding window. Since speckle contrast is related to perfusion, a perfusion unit weight may be determined based on the speckle contrast value of the first speckle contrast image. The second speckle contrast image may optionally be used to correct 152 the first speckle contrast image, such as by using a reference Figure 8 In this case, the corrected speckle contrast values ​​may be used to determine the perfusion unit weights.

[0133] Based on the correction image (i.e., in the present embodiment, the second original speckle image), registration parameters may be calculated 150. For example, the position of a predetermined object feature may be determined. Based on the position of the predetermined object feature in two or more correction images, an alignment vector for identifying the motion of the target area may be determined, for example, using an optical flow algorithm. A (sparse) optical flow algorithm may be used to determine the alignment vector based on the selected features. Other embodiments may use, for example, a dense optical flow algorithm; in that case, no features need to be determined. Alternatively, the registration parameters may be calculated in any other suitable manner, for example using template matching, phase matching, matching in a log polar coordinate system, etc. Some of these examples may include determining a transformed image based on the correction image. Reference will be made below to Figure 5A-5D Various examples of suitable methods for determining registration parameters are described in more detail in FIG. 6 .

[0134] Based on the registration parameters alignment vector, a transformation for registering the images in the second image sequence may be determined. Preferably, the transformation is selected from the class of homographies, projective transformations or affine transformations. Optionally, the optical flow weights may be determined based on the alignment vector or parameters defining the transformation.

[0135] The determined registration parameters may then be used to register 154 or geometrically align the first speckle contrast image associated with the correction image.

[0136] Subsequently, the registered first speckle contrast images may be combined 156, for example using a temporal filter. The temporal filter may include averaging a plurality of first speckle contrast images. The averaging may be a weighted averaging, wherein the weights are based on optical flow weights and / or on perfusion unit weights.

[0137] In a final step 158 , the combined first raw speckle image may be post-processed, e.g., converted to perfusion values, thresholded to indicate areas with high or low perfusion, overlaid on a white light image of the target area, etc. The post-processing may be performed by, e.g., the processing unit 110 or the computer 112 .

[0138] Figure 1D A flow chart for motion compensated laser speckle contrast imaging according to an embodiment is depicted. In the depicted embodiment, the light of the first wavelength generated by the first light source 104 is infrared light, but other colors such as red, green or blue are also possible. At least the second wavelength of light generated by the second light source 106 is (incoherent) white light. An advantage of using infrared light and white light is that both can be used simultaneously without substantially affecting each other.

[0139] As shown, the one or more image sensors 108 are two image sensors in two cameras (an infrared camera and a color camera). This can be practical if laser speckle imaging is added to a system that already includes a color camera, such as in an open surgical setting. In addition, having a dedicated image sensor can allow for separate optimization of hardware and / or device parameters, such as exposure time. As described above, other embodiments may use different configurations. For example, in some embodiments, a single camera may be used to capture both an infrared image and a (white light) color image. The advantage of using a single camera is that the infrared image and the color image are automatically aligned and associated with each other.

[0140] In a first step 160, a first sequence of first images is captured by an infrared camera, which may be stored as a speckle image sequence 162. The first sequence may be stored by an image processor as a raw speckle image sequence. In a second step 161, a second sequence of second images is captured by a color camera. The second sequence may be stored 164 as a corrected image sequence. Each raw speckle image may be associated with one or more corrected images. Preferably, each raw speckle image is associated with at least a corrected image which was captured closest in time to the raw speckle image, preferably simultaneously with the raw speckle image. Preferably, the frame rates of the first camera and the second camera are selected so as to allow direct association, for example by selecting one frame rate to be an integer multiple of the other frame rate.

[0141] Based on each raw speckle image in the speckle image sequence, a speckle contrast image may be calculated 166. The speckle contrast may be determined in any suitable manner, such as by convolution with a predetermined convolution kernel, or by determining a relative standard deviation of pixel value intensities in a sliding window. Since speckle contrast is related to perfusion, a perfusion unit weight may be determined based on the speckle contrast value of the first speckle contrast image.

[0142] Based on the correction image (i.e., in the present embodiment, the white light image), registration parameters may be calculated 170. For example, the position of a predetermined object feature may be determined. Based on the position of the predetermined object feature in two or more correction images, an alignment vector for identifying the movement of the target area may be determined, for example, using an optical flow algorithm. A (sparse) optical flow algorithm may be used to determine the alignment vector based on the selected features. Other embodiments may use, for example, a dense optical flow algorithm; in that case, the features do not need to be determined. Alternatively, the registration parameters may be calculated in any other suitable manner, for example using template matching, phase matching, matching in a log polar coordinate system, etc. Some of these examples may include determining a transformed image based on the correction image. Reference will be made below to Figure 5A-5D Various examples of suitable methods for determining registration parameters are described in more detail in FIG. 6 .

[0143] Based on the alignment vector, a transformation for registering the images in the image sequence may be determined. Preferably, the transformation is selected from the class of homographies, projective transformations or affine transformations. Optionally, the optical flow weights may be determined based on the alignment vector or parameters defining the transformation.

[0144] The determined registration parameters may then be used to register 174 or geometrically align the first speckle contrast image associated with the correction image. Since two cameras are used in this embodiment, the two cameras do not share a field of view and a reference system. Therefore, registering the speckle contrast images based on the registration parameters determined using the correction image may include applying a transformation to the registration parameters to account for this change in the reference system. If the cameras are positioned at fixed positions relative to each other, the transformation may be predetermined. Otherwise, the transformation can be determined based on image processing of, for example, calibration images or markers. The markers may be natural or artificial.

[0145] Subsequently, the registered first speckle contrast images may be combined 176, for example using a temporal filter. The temporal filter may include averaging a plurality of first speckle contrast images. The averaging may be a weighted averaging, wherein the weights are based on optical flow weights and / or on perfusion unit weights.

[0146] In a final step 178 , the combined first raw speckle image may be post-processed, e.g., converted to perfusion values, thresholded to indicate areas with high or low perfusion, overlaid on a white light image of the target area, etc. The post-processing may be performed by, e.g., the processing unit 110 or the computer 112 .

[0147] Figure 1E A flow chart of laser speckle contrast imaging for motion compensation according to an embodiment is depicted. Step 180 comprises obtaining an image sequence, for example, as described above with reference to Figure 1B As described. Furthermore, in step 182, artifacts are detected in the obtained image. In the present context, "artifact" should be understood broadly and may refer to any pixel value that may adversely affect the tissue perfusion calculation. Artifacts may be attributed to several sources. For example, specular reflection artifacts may occur (particularly in endoscopic / laparoscopic settings), where pixels or groups of pixels are completely saturated and therefore no contrast may be calculated. Artifacts may also be due to, for example, malfunctioning pixel elements in the camera, stains on the lens that gathers the light, etc.

[0148] Artifacts may be identified by identifying deviating input pixel values ​​and / or deviating speckle contrast values. For example, pixel values ​​above a predetermined absolute or relative upper threshold, or values ​​below a predetermined absolute or relative lower threshold may be flagged as deviating pixel values. For example, fully saturated pixels (i.e., pixels with a maximum pixel value) may be excluded, but it may be beneficial to also exclude nearly saturated pixels (e.g., pixels with a pixel value equal to or above 99% of the maximum pixel value).

[0149] The relative threshold may be based, for example, on an analysis of the pixel's environment, such as a mean or median value or other statistical representation. For example, pixels that deviate from the regional median by more than a specified amount, or pixels that deviate from the regional mean by more than a predetermined number of (regional) standard deviations may be identified as artifacts. Alternatively, a (Gaussian) blur may be applied to the image in order to determine a background value relative to which the relative threshold may be applied. The region used to determine outliers is typically chosen to be substantially larger than the window used to calculate the spatial speckle contrast.

[0150] Additionally, a border region around these pixels may be included in the mask, for example by growing a region with the identified pixels. The size of the border region may be based on the size of the region over which the spatial contrast is calculated. For example, if the size of the border region is at least equal to the radius of the window for spatial contrast, then the spatial contrast may be calculated without taking the mask into account and all speckle contrast values ​​based on the identified pixels will be masked.

[0151] Additionally or alternatively, pixel contrast values ​​below or above a corresponding absolute or relative lower or upper threshold may be identified. Certain artifacts may result in very low or very high speckle contrast values. Furthermore, the size and / or shape of regions with very high or low calculated speckle contrast values ​​may be used to identify artifacts.

[0152] In the present context, image artifacts may also refer to pixels that do not represent living tissue. For example, an image may include pixels that represent surgical instruments (particularly in an open surgical setting), clamps, sutures, gauze, etc. Since these objects are not living tissue, they are (usually) not perfused. Therefore, in addition to or instead of the speckle image, the speckle contrast image may be used to identify these kinds of artifacts. In embodiments where at least one image sequence also includes other images (e.g., white light images) in addition to the speckle image, pixels that do not represent living tissue may be identified in the other images in addition or alternatively.

[0153] Image recognition algorithms can also be used to identify artifacts. This is particularly useful for artifacts with known shapes or other known or learnable visual attributes.

[0154] In some embodiments, the detection of artifacts is limited to one or more regions of interest in the image. In this case, the subsequent steps may also be limited to such regions of interest.

[0155] In step 184, a mask is created for each image in the sequence of input images based on the artifacts detected for that image. The mask associates a reliability score with one or more pixels in the corresponding input image. This is typically a binary mask indicating which pixels are considered unreliable. However, multi-valued (i.e., non-binary) values ​​may also be used; for example, pixels identified as unreliable may be surrounded by a region of increasingly reliable pixels.

[0156] In some cases, multiple masks may be determined for each input image, for example, one mask representing deviating pixel values ​​and one mask representing non-tissue objects. Using multiple masks allows for different downstream processing of masked pixels. Alternatively, a multi-valued mask may be used, where different values ​​indicate different sources of uncertainty or error.

[0157] In step 185, registration parameters are calculated as described above. It may be beneficial to calculate image registration parameters based on a masked image to prevent fitting image artifacts. For example, some feature-based image registration algorithms may attempt to register overexposed speckle or specular artifacts; using a mask that excludes these artifacts may force the registration algorithm to instead register on live tissue features.

[0158] In step 186, speckle contrast is calculated. In some cases, the speckle contrast calculation takes into account the calculated mask. According to an embodiment, when the input for the speckle contrast calculation includes one or more masked pixels, these masked pixels may be ignored, or the calculation or assignment of an error value may be skipped. The treatment of masked pixels may be based on the absolute or relative number of masked pixels in the input.

[0159] In some cases, additional unreliable pixels may be determined based on calculated speckle contrast (eg, based on deviating speckle contrast values).

[0160] Therefore, in optional step 188, the mask is updated (or a new mask is created) based on the calculated speckle contrast.

[0161] In principle, the mask can be applied directly to the speckle image or the speckle contrast image, i.e., the pixel values ​​in the speckle (contrast) image are replaced by the mask values. However, it is often beneficial to keep the mask in a separate image (or separate image layer). In that case, however, care must be taken that the mask matches the image. Therefore, in step 192, the mask is transformed using the same transformation algorithm and parameters as used for the image registration of the speckle contrast image in step 190.

[0162] In step 194, a temporal filter is applied to the speckle contrast image as described above. For example, a pixel-wise weighted average may be calculated, wherein the weight of each pixel may be according to, for example, image registration parameters and / or local or global speckle contrast in the image. Additionally (or even alternatively), the weights may be according to a mask. In particular, masked pixels may have a weight equal to zero. If the mask defines a multi-valued reliability score, the weights may be according to the reliability score, wherein pixels with a higher reliability score have a higher weight than pixels with a lower reliability score. Thus, a motion compensated speckle contrast image sequence may be determined 196 based on the speckle contrast image sequence and the associated mask.

[0163] In optional step 198, a combined speckle contrast image mask is determined. The combined speckle contrast image mask may indicate whether at most a predetermined percentage of input pixels are masked. For example, a weighted average is calculated only when less than a predetermined percentage of input pixels are masked, and wherein if more than a predetermined percentage of input pixels are masked, the pixel is marked as having an invalid pixel value.

[0164] Alternatively, the combined speckle contrast image mask may indicate a reliability score for pixels in the combined speckle contrast image mask. The reliability score may be based on the fraction of masked pixels in the weighted average, and / or on the reliability score of the corresponding mask. The mask may also be based on other weighting factors into the temporal filter, such as local or global registration parameters and / or local or global speckle contrast values.

[0165] Speckle contrast or perfusion values ​​are typically displayed as an overlay on the captured image. There are various options for handling pixels in the overlay for which valid perfusion values ​​cannot be calculated. For example, pixels marked as having invalid pixel values ​​are removed or rendered as transparent, assigned error values, or assigned interpolated values ​​based on surrounding pixel values. For example, two or more of these options can be combined based on the cause of the invalid pixel value and / or based on the size of the connected area of ​​invalid pixel values. For example, an invalid pixel area due to the presence of non-tissue objects (e.g., surgical instruments) can be shown as transparent so that the instruments or other non-tissue objects can be clearly seen. As another example, small invalid pixel areas can be filled using interpolation to provide a clean image with the most likely correct values, while large invalid pixel areas can be assigned error values, indicating that valid perfusion data was not obtained.

[0166] If each pixel, or at least each pixel in a relevant area, is assigned a reliability score, the reliability score may be rendered with varying transparency (e.g., using an alpha channel), with reliable pixels having low transparency (high opacity) and unreliable pixels having high transparency (low opacity).

[0167] In an embodiment, the method steps described in the present disclosure may be performed by a processor in a device for coupling coherent light into an endoscope system. Such a device may be coupled between a light source and a video processor of an endoscope system and an endoscope (e.g., a laparoscope) of the endoscope system. Thus, the coupling device may add laser speckle imaging capabilities to the endoscope system. Such a coupling device is described in more detail in Netherlands patent application NL2026240, which is incorporated herein by reference.

[0168] In alternative embodiments, the method steps described in the present disclosure may be applied in an open surgical setting, possibly in combination with a pre-existing imaging system.

[0169] Figure 2A Depicted are raw speckle images of a target with low perfusion and a target with high perfusion and laser speckle contrast images based on the raw speckle images. Images 202 and 204 are raw speckle images of the tip of a human finger (including the nail bed). When image 202 was obtained, blood flow through the finger was restricted, resulting in low blood perfusion (artificial ischemia) of the finger. When image 204 was obtained, blood flow was not restricted, resulting in much higher blood perfusion than in the previous case. For a human observer, it is difficult to see the difference in speckle patterns associated with the perfusion difference. An enlarged portion 210 of image 204 is also shown, showing the speckle structure in more detail.

[0170] Images 206 and 208 are speckle contrast images based on images 202 and 204, respectively. Light colors represent low contrast and therefore high perfusion, while dark colors represent high contrast and therefore low perfusion. In these images, the difference in perfusion is immediately clear, especially in the nail bed where blood flow exists relatively close to the surface.

[0171] Figure 2B Depicted is a series of laser speckle contrast images of a hypoperfused target exhibiting motion before and after motion correction, in accordance with an embodiment. Image 220 1-5 is a speckle contrast image of the tip of a human finger (including the nail bed). 2-4 During the acquisition of the finger, the finger moves, resulting in a loss of contrast due to the finger movement. If the user is interested in blood flow, the image 220 represented by the light color 2-4 The low speckle contrast in can be considered a motion artifact. Image 222 1-5 Based on the image 220 1-5 The same original speckle image, but having been corrected by a motion correction algorithm according to an embodiment.

[0172] Figure 2CDepicted are speckle contrast images from a series of speckle contrast images and graphs representing perfusion levels based on the series of speckle contrast images. Graph 230 depicts perfusion measurements of the nail bed, showing a first curve 232 representing perfusion determined based on uncorrected measurements, and a second curve 234 representing perfusion determined based on measurements corrected using a motion correction algorithm as described in the present disclosure. Around the 10 second time mark, blood flow to the finger is artificially restricted, and approximately 30 seconds later, the restriction is removed. Particularly between the 23 to 38 second time range, the uncorrected perfusion measurements show many motion artifacts, where perfusion again appears to rise and fall dramatically.

[0173] The figure further shows that before processing using a single wavelength based motion correction and compensation algorithm (image 236 1-3 ) and after (Image 238 1-3 ). In these images, light colors represent low speckle contrast and, therefore, high perfusion, while dark colors represent high speckle contrast and, therefore, low perfusion. The three uncorrected images look more or less the same, making it difficult for the user to identify times (or, in other applications, areas) with low or high perfusion. In contrast, the motion-corrected image shows a clear difference between the (middle) image acquired during blood flow restriction and the other images with unrestricted blood flow, allowing the user to select times or locations with high perfusion.

[0174] Additionally, anatomical structures such as the edges of fingers and nail beds can be more easily identified in motion compensated images, whereas details are difficult to discern in uncorrected images due to their grainy nature. This further facilitates the user's interpretation of the image.

[0175] Figure 3A A flow chart for motion compensated laser speckle contrast imaging according to an embodiment is depicted. In a first step 302, a first raw speckle image may be obtained at a first time t=t1. A light source may illuminate a target area with coherent light of a predetermined wavelength, and an image sensor may capture a first raw speckle image based on the predetermined wavelength. Based on the first raw speckle image, a first laser speckle contrast image may be calculated 304. For example, the laser speckle contrast image may be determined by determining a relative standard deviation of pixel values ​​in a sliding window (e.g., a 3×3 window, a 5×5 window, or a 7×7 window). In general, a (2n+1)×(2n+1) window may be selected for a natural number n according to the speckle size. Alternatively, a convolution with a kernel may be used, wherein the size of the kernel may be selected based on the speckle size. The relative standard deviation may be determined by calculating the standard deviation of pixel intensity values ​​in a region divided by an average pixel intensity value in the region. Alternatively, the laser speckle contrast value may be determined in any other suitable manner.

[0176] Steps 306 and 308, respectively performed at the second time t=t2, are similar to steps 302 and 304. Therefore, a second raw speckle image may be obtained 306 at the second time t=t2. The light source may illuminate the target area with coherent light of a predetermined wavelength, and the image sensor may capture the second raw speckle image based on the predetermined wavelength. Based on the second raw speckle image, a second laser speckle contrast image may be calculated 308.

[0177] In a next step 310, the processor may determine a registration parameter based on the first speckle image and the second speckle image, based on an image associated with the first speckle image and the second speckle image, and / or based on a transformation of the first speckle image and the second speckle image. The registration parameter describes a geometric relationship between the first speckle image and the second speckle image, allowing the first speckle image and the second speckle image to be aligned with each other. For example, the registration parameter may include an alignment vector describing a displacement of one or more pixels in a real or transformed space.

[0178] In a next step 312, the processor may determine an alignment transformation, such as an affine transformation or more generally a homography, based on the registration parameters for registering or aligning the first speckle image and the second speckle image with each other. Based on the alignment transformation, the processor may register or align the first laser speckle contrast image and the second laser speckle contrast image with each other by transforming the first laser speckle contrast image and / or the second laser speckle contrast image 314. In general, an older image may be transformed to register or align with a newer image.

[0179] The processor may then calculate 316 a combined (e.g., averaged) laser speckle contrast image based on the registered first laser speckle contrast image and the second laser speckle contrast image. Calculating the combined image may include calculating a weighted average, with the weights preferably being based on a normalized amount of speckle contrast, based on a relative change in speckle contrast, and / or based on the determined registration parameters. Calculating the combined image may also include applying one or more filters, such as a median filter or an outlier filter.

[0180] In other embodiments, the steps may be performed in a different order. For example, the laser speckle contrast image may be calculated after the raw speckle images have been registered. In this way, a temporal speckle contrast image or a spatiotemporal speckle contrast image may be calculated. However, if the alignment transformation is more general than a translation (e.g., including rotation, scaling, and / or shearing), the alignment transformation may distort the speckle pattern, thereby introducing a source of noise. In some embodiments, a single laser contrast raw speckle image may be calculated based on the combined (e.g., averaged) raw speckle images. In this case, the images are preferably registered with sub-pixel accuracy.

[0181] Figure 3BA flow chart of laser speckle contrast imaging for motion compensation according to an embodiment is depicted. In a first step 322 , a first raw speckle image may be obtained at a first time instant t=t1 , and based on the first raw speckle image, a first laser speckle contrast image may be calculated 324 .

[0182] In a next step 325, the processor may determine a first plurality of first features in the first raw speckle image. Alternatively, the first plurality of first features may be determined in the first speckle contrast image. Preferably, the image with the most clearly defined anatomical features is used; whether the anatomical structure has a higher visual distinctiveness in the raw speckle image or in the speckle contrast image may depend on imaging parameters such as wavelength and exposure time. Figure 3A In the remainder of the description, the term "speckle image" may refer to the original speckle image or the speckle contrast image. Figure 5A and Figure 6A-6D The steps related to feature detection are described in more detail.

[0183] Steps 326-329 performed at the second time t=t2 are similar to steps 322-325, respectively. Therefore, a second raw speckle image can be obtained 326 at the second time t=t2. The light source can illuminate the target area with coherent light of a predetermined wavelength, and the image sensor can capture the second raw speckle image based on the predetermined wavelength. Based on the second raw speckle image, a second laser speckle contrast image can be calculated 328.

[0184] In a next step 329, the processor may determine a second plurality of second features in the second speckle image. At least a portion of the second plurality of second features should correspond to at least a portion of the first plurality of first features. Typically, the second speckle image is similar to the first speckle image because, in typical applications, the target area does not change much between t=t1 and t=t2. Therefore, when a deterministic algorithm is used to detect features, most of the features detected in the second speckle image will typically correspond to features detected in the first speckle image, in the sense that the features detected in the images represent the same or substantially the same anatomical features in the imaged target. Thus, a plurality of second features may be associated with a plurality of first features.

[0185] In a next step 330, the processor may determine a plurality of alignment vectors based on the first features and the corresponding second features, the alignment vectors describing the displacement of the features relative to the image. For example, the processor may determine a feature pair including a first feature and a second feature, determine a first position of the first feature relative to the first speckle image, determine a second position of the second feature relative to the second speckle image, and determine a difference between the first position and the second position. Typically, the pair of corresponding features may be a pair of a first feature and an associated second feature representing the same anatomical feature.

[0186] In a next step 332, the processor may determine a transformation, such as an affine transformation or a more general homography, for registering the corresponding features to each other based on the plurality of alignment vectors. The transformation may be found, for example, by selecting a transformation from a class of transformations that minimizes the distance between pairs of corresponding features. Based on the transformation, the processor may register or align the first laser speckle contrast image and the second laser speckle contrast image to each other by transforming the first laser speckle contrast image and / or the second laser speckle contrast image 334. In general, an older image may be transformed to be registered with a newer image.

[0187] In other embodiments, steps 325 and 329 may be omitted, and the alignment vector may be determined based on the first speckle image and the second speckle image. For example, a dense optical flow algorithm (such as a pyramid Lucas-Kanade algorithm or a Farneback algorithm) may be used to determine the alignment vector. Such an algorithm typically performs a convolution of a neighborhood of pixels from the first speckle image with a portion or all of the second speckle image, so that the neighborhood of each pixel in the first speckle image matches the neighborhood in the second speckle image. Such a method may also include determining a polynomial expansion to model pixel values ​​in the neighborhood of pixels in the first speckle image and the second speckle image, comparing the expansion coefficients, and determining the alignment vector based on the comparison.

[0188] In this way, an alignment vector may be determined for, for example, individual pixels or groups of pixels based on pixel values ​​in groups of pixels in the speckle image.

[0189] In some embodiments, step 330 may be omitted and the transformation may be determined based on the pixel values ​​of a pixel group in the first speckle image and the pixel values ​​of an associated pixel group in the second speckle image, for example using a trained neural network that receives the first image and the second image as input and provides the transformation as output to register the first image with the second image, or alternatively, to register the second image with the first image.

[0190] The processor may then calculate 336 a combined (e.g., averaged) laser speckle contrast image based on the registered first laser speckle contrast image and the second laser speckle contrast image. Calculating the combined image may include calculating a weighted average, with the weights preferably being based on a normalized amount of speckle contrast, on a relative change in speckle contrast, on a determined alignment vector, and / or on a parameter associated with the determined transformation. Calculating the combined image may also include applying one or more filters, such as a median filter or an outlier filter.

[0191] Figure 3CA flow chart for motion compensated laser speckle contrast imaging according to an embodiment is depicted. In a first step 342, a first raw speckle image may be obtained at a first time t=t1. A first light source may illuminate a target area with coherent light of a first wavelength, and a first image sensor may capture a first raw speckle image based on the first wavelength. Based on the first raw speckle image, a first laser speckle contrast image may be calculated 344. The laser speckle contrast image may be determined as described above with reference to step 304.

[0192] In a next step 343, a first corrected image associated with the first speckle image may be obtained at a first time instant t=t1. The first corrected image may include pixels having pixel values ​​and pixel coordinates, the pixel coordinates identifying the location of the pixels in the image. The first corrected image is preferably obtained simultaneously with the first raw speckle image, but in alternative embodiments, the first corrected image may be obtained, for example, before or after the first raw speckle image.

[0193] The second light source can illuminate the target area with light of at least a second wavelength different from the first wavelength, and the second image sensor can capture the first corrected image based on at least the second wavelength. The second light source can use coherent light or incoherent light. The second light source can generate monochromatic light or polychromatic light, such as white light. The second image sensor can be the same sensor as the first image sensor, or a different sensor.

[0194] Reference above Figure 3A An embodiment is described in which the second wavelength is the same as the first wavelength and the first corrected image is the same image as the first original speckle image.

[0195] In a next step 345, the processor may determine a first plurality of first features in the first corrected image. The steps related to feature detection will be described in more detail with reference to Figs. 5-6.

[0196] Steps 346-349 performed at the second time t=t2 are similar to steps 342-345, respectively. Therefore, a second raw speckle image can be obtained 346 at the second time t=t2. The first light source can illuminate the target area with coherent light of the first wavelength, and the first image sensor can capture the second raw speckle image based on the first wavelength. Based on the second raw speckle image, a second laser speckle contrast image can be calculated 348.

[0197] In the next step 345, a second corrected image associated with the second raw speckle image may be obtained at a second time t=t2. The second light source may illuminate the target area with light of at least a second wavelength, and the second image sensor may capture the second corrected image based on at least the second wavelength.

[0198] In the next step 349, the processor may determine a second plurality of second features in the second corrected image. At least a portion of the second plurality of second features should correspond to at least a portion of the first plurality of first features. Typically, the second corrected image is similar to the first corrected image because, in typical applications, the target area does not change much between t=t1 and t=t2. Therefore, when a deterministic algorithm is used to detect features, most of the features detected in the second corrected image will typically correspond to the features detected in the first corrected image in the sense that the features detected in the image represent the same or virtually the same anatomical features in the imaging target. Therefore, multiple second features may be associated with multiple first features.

[0199] In a next step 350, the processor can determine a plurality of alignment vectors based on the first features and the corresponding second features, the alignment vectors describing the displacement of the features relative to the image. For example, the processor can determine a feature pair including a first feature and a second feature, determine a first position of the first feature relative to the first corrected image, determine a second position of the second feature relative to the second corrected image, and determine a difference between the first position and the second position. Typically, the pair of corresponding features can be a pair of a first feature and an associated second feature representing the same anatomical feature.

[0200] In a next step 352, the processor may determine an alignment transformation, such as an affine transformation or more generally a homography, based on the plurality of alignment vectors for registering the corresponding features with one another. The alignment transformation may be found, for example, by selecting a transformation from a class of transformations that minimizes the distance between pairs of corresponding features. Based on the alignment transformation, the processor may register or align the first laser speckle contrast image and the second laser speckle contrast image with one another 354 by transforming the first laser speckle contrast image and / or the second laser speckle contrast image. Typically, an older image may be transformed to register with a newer image. In embodiments with more than one image sensor, the registration parameters determined based on the correction image may be adjusted to account for differences between the fields of view of the more than one image sensor.

[0201] In other embodiments, steps 345 and 349 may be omitted, and the alignment vector may be determined based on the first corrected image and the second corrected image. For example, a dense optical flow algorithm (such as a pyramid Lucas-Kanade algorithm or a Farneback algorithm) may be used to determine the alignment vector. Such an algorithm typically performs a convolution of a neighborhood of pixels from the first corrected image with a portion or all of the second corrected image, thereby matching the neighborhood of each pixel in the first corrected image with the neighborhood in the second corrected image. Such methods may also include determining a polynomial expansion to model pixel values ​​in the neighborhood of pixels in the first corrected image and the second corrected image, comparing the expansion coefficients, and determining the alignment vector based on the comparison.

[0202] In this way, for example, alignment vectors of individual pixels or groups of pixels may be determined based on pixel values ​​in the pixel groups in the first corrected image and pixel values ​​in the associated pixel groups in the second corrected image.

[0203] In some embodiments, step 350 may be omitted and the alignment transformation may be determined based on pixel values ​​of a pixel group in the first corrected image and pixel values ​​of an associated pixel group in the second corrected image, for example using a trained neural network that receives the first image and the second image as input and provides a transformation as output to align the first image with the second image or alternatively to align the second image with the first image.

[0204] The processor may then calculate 356 a combined (eg, averaged) laser speckle contrast image based on the registered first laser speckle contrast image and the second laser speckle contrast image, as described above with reference to step 316 .

[0205] In other embodiments, the steps may be performed in a different order. For example, the laser speckle contrast image may be calculated after the raw speckle images have been registered. In this way, a temporal speckle contrast image or a spatiotemporal speckle contrast image may be calculated. However, if the transformation is more general than translation and rotation (e.g., includes scaling or shearing), the transformation may distort the speckle pattern, thereby introducing a source of noise. This is particularly true for embodiments with more than one camera. In some embodiments, a single laser contrast raw speckle image may be calculated based on the combined (e.g., averaged) raw speckle images. In this case, the images are preferably registered with sub-pixel accuracy.

[0206] Figure 3D A flow chart of laser speckle contrast imaging for motion compensation according to an embodiment is depicted. In a first step 362, a first raw speckle image may be obtained at a first time instant t=t1, and based on the first raw speckle image, a first laser speckle contrast image may be calculated 364.

[0207] In a next step 365, the processor may determine a first transformed image based on the first original speckle image or based on the first speckle contrast image, such as a Fourier transform, a Mellin transform, or a log-polar transform. Figure 5C-5F The steps associated with image transformation are described in more detail.

[0208] Steps 366-369 performed at the second time t=t2 are similar to steps 362-365, respectively. Therefore, a second raw speckle image can be obtained 366 at the second time t=t2. The light source can illuminate the target area with coherent light of a predetermined wavelength, and the image sensor can capture the second raw speckle image based on the predetermined wavelength. Based on the second raw speckle image, a second laser speckle contrast image can be calculated 368. A second transformed image can be determined 369 based on the second speckle image using the same transformation as in step 365.

[0209] In a next step 370, the processor may determine registration parameters based on the first speckle image and the second speckle image, based on an image associated with the first speckle image and the second speckle image, and / or based on a transformation of the first speckle image and the second speckle image. The registration parameters describe a geometric relationship between the first speckle image and the second speckle image, allowing the first speckle image and the second speckle image to be aligned with each other. For example, the registration parameters may include an alignment vector describing a displacement of one or more pixels in a real or transformed space.

[0210] In a next step 372, the processor may determine an alignment transformation, such as an affine transformation or more generally a homography, based on the registration parameters for registering or aligning the first speckle image and the second speckle image with each other. Based on the alignment transformation, the processor may register or align the first laser speckle contrast image and the second laser speckle contrast image with each other by transforming the first laser speckle contrast image and / or the second laser speckle contrast image 374. In general, an older image may be transformed to register or align with a newer image.

[0211] The processor may then calculate 376 a combined (e.g., averaged) laser speckle contrast image based on the registered first laser speckle contrast image and the second laser speckle contrast image. Calculating the combined image may include calculating a weighted average, with the weights preferably being based on a normalized amount of speckle contrast, on a relative change in speckle contrast, on a determined alignment vector, and / or on a parameter associated with the determined transformation. Calculating the combined image may also include applying one or more filters, such as a median filter or an outlier filter.

[0212] You can combine the above references Figure 3A-3D For example, the registration parameters may include a translation parameter, a rotation parameter, and a scaling parameter, wherein the translation parameter is determined based on the speckle image, for example using template matching, and the rotation parameter and the scaling parameter are determined based on the transformed image, for example using cross-correlation of log polar coordinate images.

[0213] Figure 4AA flow chart of a motion compensated speckle contrast imaging method according to an embodiment is depicted. In a first step 402, the method may include exposing a target area to coherent light of a predetermined wavelength. Preferably, the target area includes living tissue, such as skin, burns, or internal organs, such as intestine or brain tissue. Preferably, the living tissue is perfused and / or includes blood vessels and / or lymphatic vessels. The predetermined wavelength may be a wavelength in the visible spectrum, such as a wavelength in the red, green or blue part of the visible spectrum, or the predetermined wavelength may be a wavelength in the infrared part of the spectrum, preferably a wavelength in the near infrared part.

[0214] In a next step 404, the method may comprise capturing, for example by means of an image sensor, at least one sequence of images comprising a (raw) speckle image, the (raw) speckle image being captured during exposure with the first light.

[0215] Each raw speckle image may include pixels, which are defined by pixel coordinates and have pixel values. The pixel coordinates may define the location of the pixel relative to the image and are typically associated with a sensor element of an image sensor. The pixel value may represent light intensity.

[0216] The image sensor may comprise a 2D image sensor, such as a CCD, such as a monochrome camera or a color camera.The images in the sequence of images can be frames in a video stream or in a multi-frame snapshot.

[0217] In a next step 406, the method may further comprise determining one or more registration parameters of an image registration algorithm for registering the speckle images with each other. In principle, the registration parameters may be determined in any suitable manner, for example as described below with reference to Figure 5A-5F and Figure 6A-6D described.

[0218] For example, the registration parameters may be based on a measure of similarity of pixel values ​​of groups of pixels in a plurality of images of at least one image sequence, the images in the plurality of images being selected from the speckle images. Alternatively, the images in the plurality of images may include other images associated with the original speckle image in addition to the original speckle image. The registration parameters are preferably defined from one or more transformations of the group of homographies, projective transformations or affine transformations.

[0219] The determination of the pixel group based registration parameters is described in more detail below with reference to step 416, it being understood that in this embodiment the (original) speckle image is used as the correction image.

[0220] The method may further include determining, using step 408 or step 410 , a combined laser speckle contrast image based on at least a portion of the first speckle image sequence and the one or more determined registration parameters.

[0221] In step 408, the method may further include determining a registered speckle image by registering the speckle images based on one or more registration parameters and an image registration algorithm, and determining a combined speckle contrast image based on the registered speckle image. In such an embodiment, the algorithm may first register the sequence of raw speckle images using the determined transformation, then calculate the sequence of speckle contrast images, and then combine the registered speckle contrast images.

[0222] In an alternative step 410, the method may further include determining a speckle contrast image based on the speckle image, determining a registered speckle contrast image by registering the speckle contrast images based on one or more registration parameters and an image registration algorithm, and determining a combined speckle contrast image based on the registered speckle contrast image. In other words, the algorithm may first compute a sequence of speckle contrast images, then register the speckle contrast images using the determined transformation, and then combine the registered speckle contrast images.

[0223] Further alternatives are discussed below with reference to steps 418 and 420 .

[0224] Figure 4B A flow chart of a motion compensated speckle contrast imaging method according to an embodiment is depicted. In a first step 412, the method may include exposing a target area to coherent first light of a first wavelength and second light of one or more second wavelengths, which are preferably at least partially different from the first wavelength, alternately or simultaneously. For example, the second light may be coherent light of a second wavelength, narrowband light, or light of a plurality of second wavelengths comprising a visible spectrum, such as white light. Preferably, the target area comprises living tissue, such as skin, burns, or internal organs, such as intestine or brain tissue. Preferably, the living tissue is perfused and / or comprises blood vessels and / or lymphatic vessels.

[0225] In a next step 414, the method may include capturing a first (raw) speckle image sequence during exposure with the first light, and a second (corrected) image sequence during exposure with the second light, for example by an image sensor system having one or more image sensors having a fixed relationship to each other. The speckle images of the first speckle image sequence may be associated with images of the second image sequence.

[0226] In the case of simultaneous exposure and acquisition, each second image may be associated with a first speckle image acquired simultaneously. In embodiments where the second image is the same image as the first speckle image, each image may be considered to be associated with itself, and the image may be referred to as a first speckle image or a second image, depending on its function in the algorithm (e.g., determining registration parameters or providing perfusion information).

[0227] In the case of alternating acquisition, a first speckle image may be associated with, for example, a second image acquired immediately before or after the first speckle image, or both. When the first speckle image is acquired at a higher rate than the second image, several first speckle images may be associated with a single second image.

[0228] Thus, a sequence of first raw speckle images of a target area and a sequence of corrected images of the target area may be acquired, each corrected image being associated with one or more first raw speckle images. Each corrected image may include a pixel, which is defined by a pixel coordinate and has a pixel value. The pixel coordinate may define the position of the pixel relative to the image and is typically associated with a sensor element of an image sensor. The pixel value may represent a light intensity.

[0229] The image sensor system may include one or more 2D image sensors, such as CCDs. The first raw speckle image and the corrected image may be acquired using one or more image sensors (e.g., using a grayscale camera or a color (RGB) camera). The images in the image sequence may be, for example, frames in a video stream or a multi-frame snapshot.

[0230] In a next step 416, the method may further comprise determining one or more registration parameters of a registration algorithm for registering at least a portion of the first speckle image sequence based on a similarity measure of pixel values ​​of a pixel group in at least a portion of the second image sequence associated with the first speckle image. The registration parameters preferably define one or more transformations from the group of homography, projective transformation or affine transformation.

[0231] Determining the registration parameters may include selecting a first corrected image from at least a portion of the sequence of corrected images, and determining a plurality of first pixel groups in the first corrected image. The first corrected image may be a reference corrected image. In an embodiment, the first corrected image may be a first image, for example, when a single output image is generated based on an input of a user. In a different embodiment, the first corrected image may be a most recent corrected image, for example, when a continuous stream of output images is being generated.

[0232] The first pixel group may be associated with a feature (e.g., an edge or corner) in the first corrected image. Preferably, the feature is associated with a physical or anatomical feature, such as a blood vessel, or more specifically, a sharp corner or bifurcation in a blood vessel. Image features that are not associated with physical features (such as overexposed image portions or edges of speckle) may show large frame-to-frame variations and may therefore be less useful for registering images. The feature may be a predetermined feature, such as a feature belonging to a class of features, such as a corner or area with a large intensity difference. The feature may also be determined by, for example, a quality metric, a restriction on mutual distances between features, and the like.

[0233] Alternatively, the pixel group may be associated with an area in the first corrected image, e.g., a neighborhood of a predetermined set of pixels. Thus, for example, the pixel group may include every pixel in the image, a contiguous region of pixels at a predetermined location (e.g., the center of the image), or a selection of pixels uniformly distributed across the image.

[0234] Determining the registration parameters may also include selecting one or more second corrected images that are different from the first corrected image. For each of the selected one or more second corrected images, a plurality of second pixel groups may be determined. The second pixel groups may be associated with features in the second corrected image. If feature-based (image) registration is used, such as using a sparse optical flow algorithm, then preferably the same algorithm is used to determine the first pixel group and the second pixel group.

[0235] If the first pixel group is determined based on pixel coordinates, the second pixel group may be determined by convolving or cross-correlating the first pixel group with the second corrected image and selecting the pixel group that is most similar to the first pixel group, for example based on a suitable similarity metric. The convolution may be restricted in space, for example by searching only for matching second pixel groups that are close to the position of the first pixel group. Alternatively or additionally, the second pixel group may be constrained to preserve the mutual orientation of the first pixel groups, for example to prevent anatomically impossible combinations.

[0236] The second pixel group may then be associated with the first pixel group based at least on the similarity of pixel values. In embodiments where the second pixel group is determined by matching or convolution with the first pixel group, such association may be performed as part of determining the second pixel group. If feature-based registration is used, the second pixel group may be associated with the first pixel group based on similarity of features associated with the second pixel group and features associated with the first pixel group.

[0237] A transformation for registering the second corrected image with the first corrected image, and hence for registering the associated first speckle image or derived first speckle contrast image, may be determined based on pixel coordinates of pixels in the associated first pixel group and second pixel group. Determining the transformation may include determining a 3D motion of the image sensor system relative to the target area, or may be informed by the effect that the 3D motion will have on the acquired image.

[0238] As an intermediate step, an alignment vector may be determined based on the positions of the first pixel group and the associated second pixel group (e.g., based on the positions of features in the first and second corrected images). The alignment vector may represent the motion of the target area relative to the image sensor or image capture device. In some embodiments, a neighborhood of one or more determined object features may be used to determine the alignment vector and / or transformation.

[0239] The determination of the alignment vector and / or the determination of the transformation may be based on optical flow parameters determined using any suitable sparse or dense optical flow algorithm. Determining the alignment vector may include determining a corresponding or matching feature pair, one feature in the feature pair being determined in a correction image associated with a first time instance and the other feature in the feature pair being determined in a subsequent correction image associated with a subsequent time instance in the sequence of correction images.

[0240] Methods of determining registration parameters and optional features are discussed in more detail below with reference to FIGS. 5 and 6 .

[0241] The method may further include determining, using step 418 or step 420 , a combined laser speckle contrast image based on at least a portion of the first speckle image sequence and the one or more determined registration parameters.

[0242] In step 418, the method may further include determining a registered first speckle image by registering at least a portion of the first speckle image sequence based on one or more registration parameters and a registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image. In such an embodiment, the algorithm may first register the original speckle image sequence using the determined transformation, then calculate the speckle contrast image sequence, and then combine the registered speckle contrast image.

[0243] In an alternative step 420, the method may further include determining a first speckle contrast image based on at least a portion of the sequence of first speckle images, determining a registered speckle contrast image by registering the first speckle contrast images based on one or more registration parameters and a registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast image. In other words, the algorithm may first calculate the sequence of speckle contrast images, then register the speckle contrast images using the determined transformation, and then combine the registered speckle contrast images.

[0244] In another alternative, determining the combined laser speckle contrast image may include determining a registered sequence of first raw speckle images based on the first raw speckle images and the determined transformation, determining a combined speckle contrast image based on two or more registered first raw speckle images of the registered sequence of first raw speckle images, and determining a combined speckle contrast image based on the combined raw speckle images. In such an embodiment, the algorithm may first register the sequence of raw speckle images using the determined transformation, then combine the registered speckle contrast images, and then calculate the speckle contrast image.

[0245] In a further alternative, the combining and calculation of the speckle contrast may be a single step, for example by calculating the temporal or spatiotemporal speckle contrast based on a sequence of registered first raw speckle images.

[0246] Combining the raw speckle images or the speckle contrast images may include averaging, weighted averaging, filtering with, for example, a median filter, etc. The weights used for weighted averaging may be based on, for example, quantities derived from speckle contrast, quantities derived from registration parameters, or quantities derived from alignment vectors. Fig. 9A Methods of combining raw speckle images or speckle contrast images are discussed in more detail.

[0247] Figure 4C A flow chart of a motion compensated speckle contrast imaging method according to an embodiment is depicted.Steps 422 and 424 may be similar to steps 402 and 404 or steps 412 and 414, respectively.

[0248] In the next step 425, the method may further include determining a transformed image by transforming the first speckle image and / or an image associated with the first speckle image (e.g., an image captured during exposure with the second light). For example, the transformed image may be obtained using Fourier transform, Mellin transform, and / or log-polar transform.

[0249] In a next step 426, the method may further include determining one or more registration parameters of an image registration algorithm for registering the speckle images with each other based on the transformed image. Figure 5D-5F and FIG. 6 describe examples.

[0250] The method may further include determining a combined laser speckle contrast image based on at least a portion of the first speckle image sequence and one or more determined registration parameters using step 428 or step 430. These steps are similar to steps 408 and 410, 418 and 420, respectively, discussed above.

[0251] Figure 5A A method for determining registration parameters according to an embodiment is depicted. A first plurality of first features 506 may be determined in a first corrected image 502 acquired at a first time instance t=t1 1-3 Each of the plurality of first features may be associated with a group of pixels in the first corrected image. Preferably, the first features relate to anatomical structures, such as blood vessels 504 1-2 , or other stable features that may be assumed not to move between subsequent frames. Therefore, it is preferred to use light that makes such anatomical structures clearly visible to obtain the first corrected image. For example, green light may be used, which is strongly absorbed by blood vessels but not absorbed by most other tissues. Therefore, blood vessels may appear as dark structures in the light environment, resulting in high visual distinctiveness. However, other embodiments may use one or more other wavelengths of light, such as blue light or white light.

[0252] Any suitable feature detection algorithm may be used to determine the features 506. 1-3, for example a feature detector based on a Harris detector or a Shi-Tomasi detector, such as goodFeaturesToTrack from the OpenCV library. Other examples of suitable feature detectors and descriptors include Speeded Up Robust Features (SURF), Features from Accelerated Segment Tests (FAST), Binary Robust Independent Elementary Features (BRIFER), and combinations thereof, such as ORB. Various suitable algorithms have been implemented in commonly available image processing libraries such as OpenCV. Preferably, the algorithm should be fast enough to allow real-time image processing, depending on the application.

[0253] Typically, sharp corners (such as feature 506 1,3 ) and bifurcations (e.g., feature 5062) are good features. Preferably, a deterministic feature detection algorithm is used, i.e., a feature detection algorithm that detects the same feature in the same image. Preferably, the features should be distributed over a large portion of the image area. A good distribution of feature points over the image can be obtained by requiring a minimum distance between selected feature points. In some embodiments, for example, based on an implementation of BRIEF or ORB, descriptors identifying the attributes of the feature can be assigned to the features, thereby facilitating feature differentiation and feature matching.

[0254] The minimum number of feature points depends on the type of transformation; for example, an affine transformation has six degrees of freedom, while a homography has eight degrees of freedom. Thus, the first plurality of first features may include at least 5 features, preferably at least 25, more preferably at least 250, and even more preferably at least 1000. The number of features may depend on the amount of pixels in the image, with a greater number of features for images with more pixels. In general, a higher number of features may result in a more accurate transformation because random errors may be averaged out.

[0255] However, there are various reasons why the number of features may be limited. For example, there may only be a limited number of features that meet a predetermined quality indicator (e.g., the magnitude of local contrast or the sharpness of corners). In addition, the computation time increases with the number of features, and therefore, the number of features may be limited to allow real-time image registration, e.g., for a 50fps video feed, the entire algorithm should preferably take less than 20ms per frame.

[0256] A second plurality of second features 516 associated with a second set of pixels may be determined in a second corrected image 512 acquired at a second time instance t=t2 1-3 Preferably, the field of view of the second corrected image substantially overlaps with the field of view of the first corrected image, preferably more than half overlaps. Preferably, the second feature relates to the same anatomical structure as the first feature, for example a blood vessel 514 1-2Preferably, the same feature detection algorithm is used to detect features in both the first corrected image and the second corrected image.

[0257] The determined first feature and the second feature may include position information relative to the first corrected image and the second corrected image, respectively. In this example, this is shown in the comparison image 522. 1-3 and a second plurality of second features 516 1-3 , multiple alignment vectors 524 can be determined 1-2 , the alignment vector describes the displacement of the feature relative to the image. In an intermediate step, corresponding feature pairs can be determined, for example, feature 5061 can be associated with feature 5161, feature 5062 can be associated with feature 5162, and feature 5063 can be associated with feature 5163. The corresponding feature pairs can be determined, for example, based on feature parameters (such as local contrast or sharpness of corners) or based on the distance between the features in the first corrected image and the second corrected image. In some embodiments, not all features in the first corrected image can be paired with features in the second corrected image. In some embodiments, the alignment vector 524 is determined. 1-3 and determining corresponding feature pairs can be performed in a single step.

[0258] For example, if a minimum distance between features is imposed and assuming that the displacement is less than the minimum distance, an algorithm that minimizes the distance between point clouds formed by first and second features, respectively, can implicitly determine pairs of corresponding features and alignment vectors for each pair of corresponding features. In a typical embodiment, a typical inter-frame displacement is only a few pixels. In an embodiment, the plurality of alignment vectors can be filtered to exclude potential outliers, e.g., alignment vectors that deviate by more than a predetermined amount from alignment vectors originating from nearby features.

[0259] In other embodiments, the alignment vector may be determined based on associated pixel groups, based on pixel values ​​in the first and second corrected images. As explained above with reference to FIG3 , in such embodiments, feature detection may be omitted. Instead, the alignment vector may be determined using, for example, a dense optical flow algorithm such as a pyramid Lucas-Kanade algorithm or a Farneback algorithm. In principle, any method for determining an alignment vector based on pixel values ​​of corresponding pixel groups may be used.

[0260] Based on multiple alignment vectors 524 1-3, a transformation can be determined. In an embodiment, the transformation can be defined by an average or median alignment vector, or by another alignment vector that is statistically representative of a plurality of alignment vectors. In different embodiments, the transformation can be an affine transformation, a projective transformation, or a homography, combining, for example, translation, rotation, scaling, and shearing transformations. Preferably, the transformation images features from the first corrected image onto corresponding features in the second corrected image. The transformation can then be applied to the first raw speckle image to register the first raw speckle image with the second raw speckle image.

[0261] In an embodiment, the first corrected image and the second corrected image may be pre-processed before determining the features. For example, overexposed and / or underexposed areas may be identified based on pixel values, for example, as described above with reference to Figure 1E As described. Subsequently, these areas may be masked so that no features may be detected in those areas. The mask may be slightly larger than the overexposed or underexposed areas, for example by growing the identified areas by a predetermined number of pixels. Masking the overexposed and / or underexposed areas may improve the quality of the features because it prevents features associated with, for example, edges or corners of overexposed areas. If a mask has been determined to identify artifacts, as described above with reference to Figure 1E As described, the same mask may then be applied prior to feature detection in order to prevent detection of unreliable features.

[0262] Figure 5B Another method for determining registration parameters according to an embodiment is depicted. This method is sometimes referred to as template matching. A first region 531 is selected in a first image 530 acquired at a first time instance t=t1. For example, the first image can be a raw speckle image, a speckle contrast image, or a corrected image. The first region has a predetermined size expressed in pixels and is selected at a predetermined position in pixel coordinates. Preferably, the first region is selected at or near the center of the first image.

[0263] A second region 533 is selected in a second image 532 acquired at a second time instance t=t2. The second image is typically an image of the same kind as the first image, for example, both may be raw speckle images. The second region has a predetermined size expressed in pixels and is selected at a predetermined position in pixel coordinates. The second region is typically smaller than the first region. The size difference can be selected based on an estimated or expected amount of motion. The center of the second region typically coincides with the center of the first region. Preferably, the second region is selected relative to the first region so that there is a high probability that the imaging (anatomical) region represented by the second region is contained in the imaging region represented by the first region. A single region in each image may be sufficient to determine the global translation registration parameters. As will be described below with reference to Fig.6DDiscussed in more detail, multiple first regions can be selected in the first image, and corresponding multiple second regions can be selected in the second image, for example, to determine non-global (i.e., regional or local) registration parameters. The first regions and second regions can be selected in an overlapping or non-overlapping manner.

[0264] Based on the first region and the second region, registration parameters may be determined. In this example, this is shown in comparison image 534. For example, a subregion of the first region that is most similar to the second region may be determined; this step is also referred to as matching. Registration parameters, such as alignment vectors 535, may be determined based on the relative positions of the subregions in the second region and the first region. For example, the matching method can be feature-based, intensity-based, or frequency-based. Reference has been made above to Figure 5A Examples of feature-based matching are discussed below and refer to Figure 5C An example of frequency-based matching is discussed. An example of intensity-based matching is the cross-correlation of the first region and the second region. The matching method may also include, for example, the following reference Figure 5D Coordinate transformation discussed. Depending on the matching method, the registration parameters can include translation parameters, rotation parameters and / or other parameters.

[0265] Figure 5C Another method for determining registration parameters according to an embodiment is depicted. Figure 5C A frequency-based registration algorithm, also known as phase correlation, is depicted. A first image 540 acquired at a first time instance t=t1 is transformed using a transform, resulting in a first transformed image 541. In this example, a discrete Fourier transform is used. Similarly, a second image 542 acquired at a second time instance t=t2 is transformed using a discrete Fourier transform, resulting in a second transformed image 543. Based on a comparison 544 of the first transformed image and the second transformed image, a registration parameter may be determined to register 556 the first speckle image and the second speckle image.

[0266] Image registration based on Fourier transforms is an example of frequency-based image registration. These methods are generally known in the art. Comparison of the transformed images may include calculation of the cross-power spectra of the transformed images. The calculation of the cross-power spectrum may include determining the complex conjugate of one of the transformed images and multiplying it element-wise with the other of the transformed images. The comparison may also include determining the (inverse) Fourier transform of the cross-power spectrum and determining the peak in the resulting image, for example by applying an argmax function. Using interpolation, the peak may be determined with sub-pixel accuracy. The location of the peak corresponds to the translation of the image to be aligned. In the depicted example, image 544 represents the inverse Fourier transform of the cross-correlation of the first transformed image and the second transformed image, and the vector between the upper right corner and the peak 545 represents the translation required to align the second image with the first image.

[0267] The comparison may include further operations to improve the results. For example, a two-dimensional Hanning window may be applied to the first image and the second image before applying the Fourier transform. As another example, a filter (e.g., a blur filter or an interpolation filter) may be applied to the inverse Fourier transform of the cross-correlation of the transformed images to improve peak detection. Various other operations for improving frequency-based image registration are known in the art. For example, to reduce the effects of noise, high frequencies may be filtered out. On the other hand, a high-pass filter may reduce image artifacts caused by image boundaries. Additionally, to prevent matching of speckle rather than anatomical structures, frequencies corresponding to speckle may be suppressed. Of course, care should be taken not to suppress all frequencies, and in particular not to suppress those frequencies where anatomical structures in the image region may give relatively strong signals. This may depend on the type of tissue being imaged.

[0268] Figure 5D Another method for determining registration parameters according to an embodiment is depicted. In this example, a so-called log-polar transformation is used. The log-polar transformation is usually applied in conjunction with a method for determining a translation, for which the method described above with reference to Figure 5A-Figure 5C Any of the methods discussed. Therefore, it can be considered as a pre-processing step of any of those methods. The first image 550 acquired at the first time instance t=t1 is transformed using a coordinate transformation, thereby producing a first transformed image 551. The horizontal axis of the transformed image represents the angle relative to the horizontal axis of the first image. and the vertical axis of the transformed image represents the logarithm of the (relative) radial distance from the center of the first image, log(r). Since r should be dimensionless, r is usually expressed in pixel units or relative to r max Indicates that, where r max represents the maximum distance from the origin (in this example, one of the diagonals). That is, if the pixel coordinates in the first image are expressed in terms of x and y (along the horizontal and vertical axes, respectively), with the origin chosen at the center of the image, then If x>0, then arctan2(y,x)=arctan(y / x), if x<0, then arctan2(y,x)= arctan(y / x)+sign(y)*π, and if x=0, then arctan2(y,x)=sign(y)*π / 2. Other embodiments may use different coordinate transformations.

[0269] Similarly, the second image 552 acquired at the second time instance t=t2 is transformed using the same coordinate transformation, thereby producing a second transformed image 553. In the depicted example, the entire first and second speckle images have been transformed. In other embodiments, only one or more regions of the first and second speckle images are transformed. Typically, at least one region includes the center of the image.

[0270] Based on the comparison 554 of the first transformed image and the second transformed image, a displacement or shift in the transformed (in this case, log polar) coordinate system may be determined. The displacement may be determined using any suitable method, for example, as described above with reference to Figure 5A-Figure 5C One of the methods described. Based on the determined displacement, registration parameters may be determined to register the first speckle image and / or the second speckle image to each other 556. In the depicted example, in addition to the rotation and scaling, the second image has also undergone a small translation, and therefore, for small r (bottom part of the image), the shift (and therefore the rotation) appears much larger than it actually is. For large r (top of the image), there is no data for all values ​​of r. Therefore, it can be advantageous to use only an extreme range of r values.

[0271] An advantage of using a log-polar transformation is that a horizontal shift 555 of the transformed image corresponds to a rotation 557 of the untransformed image, and a vertical shift of the transformed image corresponds to a scaling of the untransformed image. Thus, rotation and scaling, in particular global scaling and rotation, can be determined in a relatively straightforward manner. Since in this method a relatively large image area is used to determine the (global) scaling and rotation registration parameters, a relatively large amount of image data can be used, which can reduce the influence of noise on the determined registration parameters.

[0272] Using non-feature-based methods, e.g., as in ref. Figure 5B-5D As described above, it is usually not necessary to determine multiple alignment vectors, such as by using a reference Figure 5A Instead, for example, a single matrix may be determined based on registration parameters that, when applied to the first image, align the first image with the second image. This may reduce computational requirements.

[0273] Figure 5E and Fig. 5F Other methods for determining registration parameters according to embodiments are described. In these examples, reference is made to Figure 5A-5D Two or more of the described methods are combined. Figure 5B The template matching method described can be used to determine the translation, while the log-polar transform is used to determine the rotation and scale. When combining multiple methods, an intermediate registration step can be applied between the two registration parameter determinations. This tends to improve the results of the image registration.

[0274] Figure 5E An example is depicted, where translation parameters are first determined and applied to the first (speckle) image in an intermediate registration step. Subsequently, based on the results of the intermediate scaling step, rotation and scaling parameters are determined and applied. Fig. 5F An example is shown in which the rotation and scaling parameters are first determined and applied and then the translation parameters are determined and applied. In general, it is advantageous to determine and apply the largest transform first.

[0275] In particular, Figure 5E An example is depicted in which a first (corrected) image 560 and a second (corrected) image 562 are obtained. In step 563, a translation is determined based on the first image and the second image, and the translation is applied to the first image, thereby producing a translated first image 564. The translation may be determined using any suitable method, for example, as described above with reference to Figure 5A-Figure 5C For example, template matching may be used, where templates are matched using frequency-based phase correlation or intensity-based cross-correlation.

[0276] Subsequently, using a coordinate transformation (in this example a log-polar transformation (e.g., as described above with reference to Figure 5D The first image and the second image are transformed 565, 567 to produce a logarithmic shifted first image 566 and a logarithmic second image 568. In step 569, a shift or displacement is determined based on the logarithmic shifted first image and the logarithmic second image. The shift may be determined using any suitable method, for example, using one of the methods described above to determine the displacement. Figure 5A-Figure 5C The same method as in step 563 may be used, or a different method may be used. Based on the determined shift, rotation and / or scaling parameters are determined 571 and applied to the translated first image, thereby generating a rotated and / or scaled translated first image 572.

[0277] The rotated and / or scaled translated first image may then be combined with the second image to produce a combined image 574, for example using a weighted average. Fig. 9A As discussed in more detail, the weights may be based on registration parameters. In this example, the registration parameters include both translation parameters and rotation and / or scaling parameters. Thus, the weights may be based on only the translation parameters, only the rotation and / or scaling parameters, or both. For example, a pixel-by-pixel displacement vector may be determined based on the combined registration parameters, and the weights may be based on a statistical representation of the pixel-by-pixel displacement vectors for all pixels or a subset thereof (e.g., a region of interest), such as an average or maximum value of the norm of the displacement vectors.

[0278] Fig. 5F Depicted Figure 5E A first image 580 and a second (corrected) image 582 are obtained. The first and second images are transformed 583, 585 using a coordinate transformation (in this example a log-polar transformation) to produce a log-polar first image 584 and a log-polar second image 586. In step 587, a shift is determined based on the log-polar first image and the log-polar second image. Based on the determined shift, a rotation and / or scaling parameter is determined 589 and applied to the first image to produce a rotated and / or scaled first image 590.

[0279] In step 591, a translation is determined based on the rotated and / or scaled first image and the second image, and the determined translation is applied to the rotated and / or scaled first image, thereby generating a rotated and / or scaled translated first image 592. The rotated and / or scaled translated first image may then be combined with the second image, thereby generating a combined image 594.

[0280] In yet another example, both the determination of the rotation and / or scaling and the determination of the translation are based on the (untransformed) first and second images.An advantage is that in this case two (sets of) registration parameters can be determined in parallel.

[0281] Fig. 6A An example of determining registration parameters according to an embodiment is shown, wherein the determined transformation is a translation. Figure 5A As illustrated, the first plurality of first features 602 may be determined in the first corrected image acquired at t=t1. 1-3 , and a second plurality of second features 604 may be determined in a second corrected image acquired at t=t2 1-3 Based on the corresponding feature pairs, multiple alignment vectors 606 may be determined 1-3 For clarity, only features and alignment vectors are shown, but not the (anatomical) structures.

[0282] In a typical case, the determined alignment vector 606 1-3 will not all be exactly the same. In the depicted example, alignment vector 6061 is slightly shorter than the average, while alignment vector 6063 is slightly larger than the average. Similarly, the directions of the alignment vectors show some variation. Based on the alignment vectors, an average alignment vector 608 may be determined. The translation may be defined by a single vector. For example, all pixels of the first raw speckle image acquired at t=t1 may be shifted by an amount equal to the average alignment vector. In principle, the translation may be determined based on a single alignment vector. However, by determining multiple alignment vectors, the accuracy of the transformation may be improved.

[0283] In an embodiment, the average alignment vector 608 and the determined alignment vector 606 may be calculated, for example, based on the change in the alignment vector 1-3 In this way, an indication of the extent to which the transformation compensates for the detected displacement of each feature pair may be obtained. Alternatively, the average distance between features in the first corrected image after the transformation and corresponding features in the second corrected image may be determined.

[0284] Figure 6B An example of determining registration parameters according to an embodiment is shown, wherein the determined transformation is an affine transformation. Fig. 6A , a first plurality of first features 612 may be determined 1-3 , a second plurality of second features 614 1-3 and a plurality of alignment vectors 616 1-3 However, in this example, the average alignment vector 618, which has almost zero length, is not representative of the determined alignment vectors, which are typically longer and point in a different direction.

[0285] Therefore, to compensate for this motion, a more general transformation is needed, such as an affine transformation. Affine transformations include translation, rotation, mirroring, scaling, and shearing transformations and their combinations. Transformations can be selectively excluded by restricting the transformation parameter values. For example, mirroring can be excluded as a possible transformation because mirroring is generally physically impossible.

[0286] In general, an affine transformation can be calculated using a transformation matrix with six degrees of freedom as described in equation (1), which acts on points represented in homogeneous coordinates. By limiting the potential values ​​of the affine transformation matrix, the affine transformation can be limited to predefined operations. For example, a more specific transformation matrix limited to, for example, a rotation matrix can be obtained, which can be more suitable for certain applications.

[0287] Here, A is a transformation matrix that transforms a point p with coordinates x and y (usually in pixel coordinates) into a transformed point p' with coordinates x' and y'. Matrix A includes six free parameters, which are x and t y Defines the translation, and the submatrix Reflection, rotation, scaling and / or shearing may be defined. In this case, the transformation size may be based, for example, on the norm of the transformation matrix A or the norm of the matrix AI, where I is the identity matrix.

[0288] To solve equation (1), at least three alignment vectors may be used to provide a solvable system of six equations and six unknowns. Such a linear system can be solved in a deterministic manner known in the art. In a typical embodiment, many alignment vectors may be determined, each of which may include a small error. Therefore, a more robust approach can be to use multiple alignment vectors and use an appropriate fitting algorithm, such as a least squares fit as shown in equation (2). [A] = arg(min A (∑ i ‖p′ i -A p i ‖ 2 )) (2)

[0289] The reliability of the determined transformation can be determined again, as described above with reference to Fig. 6A As described.

[0290] Figure 6C An example of determining registration parameters according to an embodiment is shown, where the determined transformation is a projective transformation. Projective transformations include and are more general than affine transformations. Projective transformations include, for example, skew transformations. They may be needed to compensate for, for example, angular changes between the camera and the target area.

[0291] Similar to Fig. 6A and Figure 6B , the first plurality of first features 622 may be determined 1-3 , a second plurality of second features 624 1-3 and multiple alignment vectors 626 1-3 . In this example, the translation on the left side of the image is much smaller than the translation on the right side of the image. Therefore, applying an average translation will over-transform the pixels on the left side and under-transform the pixels on the right side. This displacement can be corrected by a projective transformation. Various methods of determining a projective transformation based on four or more alignment vectors are known in the art.

[0292] In general, the projective transformation can be calculated using the non-homogeneous coordinates (x1, y1, z1) and (x2, y2, z2) using the projective matrix depicted in equation (3).

[0293] The non-homogeneous coordinates (x1, y1, z1) can be related to the pixel coordinates (x, y) of the feature via x1=x, y1=y, and z1=1, and the transformed pixel coordinates (x', y') can be obtained from the non-homogeneous coordinates via x'=x2 / z2. In some embodiments, the alignment vector can be determined by (x'-x, y'-y). In other embodiments, the alignment vector is not explicitly constructed.

[0294] Therefore, equation (3) can be rewritten as two independent equations, as shown in equation (4). x′(h 31 x+h 32 y+h 33 )=h 11 x+h 12 y+h 13 (4) y′(h 31 x+h 32 y+h 33 )=h 21 x+h 22 y+h 23

[0295] Solving for H, equation (4) can be rewritten as equation (5): v x h=0 (5) v y h=0 in h=(h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 32 ,h 31 ,h 22 ,h 33 ) T v x =(-x,-y,-1,0,0,0,x′x,x′y,x′) (6) v y =(0,0,0,-x,-y,-1,x′,y,x′)

[0296] Using a set of N feature points, we can set up the system of equations as shown in equation (7): Vh=0 (7) in V=(v x1 ,v y1 ,v x2 ,v y2 ,…,v xN ,v yN ) T (8)

[0297] Since the matrix H is homogeneous by definition and can be scaled by any constant, the projective transformation matrix contains eight degrees of freedom. Therefore, equation (7) can be solved using only four sets of coordinates provided by the four features. For example, the projective transformation matrix H can be normalized using equation (9): h 33 =1 (9) Or equation (10): j 11 2 +h 12 2 +h 13 2 +h 21 2 +h 22 2 +h 23 2 +h 31 2 +h 32 2 +h 33 2 =0 (10)

[0298] Equation (7) can be solved deterministically using at least four points, but using more points can lead to more robust results, similar to what was explained above for affine transformations. In the case of a projective transformation or homography, this can be done using singular value decomposition (SVD). In some embodiments, the step of determining the alignment vector or point pair is combined with the determination of the homography step to find the most accurate projective transformation, which can be done with an algorithm such as RANSAC.

[0299] In an embodiment, the algorithm may first calculate a relatively simple transformation, such as a translation. The algorithm may then determine whether the calculated transformation reproduces the determined alignment vector with sufficient accuracy. If not, the algorithm may try a more general transformation, such as an affine transformation, and repeat the same process. If the translation is sufficient in a sufficiently large number of cases, this may reduce the required computation time. In a different embodiment, the algorithm may always calculate a general transformation, for example, always calculate a general homography. This may result in a more accurate registration of the original speckle image or the speckle contrast image.

[0300] Reference above Figure 6A-6C The examples discussed are based on feature-based transformation parameters. However, other methods of providing registration parameters for multiple locations in the first speckle image and / or the second speckle image may be used to obtain the same or similar results. For example, the first speckle image and / or the second speckle image or a transformation thereof may be divided into multiple regions, and registration parameters may be determined for each individual region. The registration parameters for multiple regions may be as described above with reference to Figure 6A-6C The above are combined.

[0301] Fig.6DAn example of determining multiple transformations according to an embodiment is shown. Each corrected image 650 in a series of corrected images can be divided into a plurality of regions 652 1-n , preferably a plurality of non-intersecting regions, such as a rectangular grid, covering the entire image. Subsequently, each region 6541, 6542, ..., 654 n Determine transformation 6541, 6542, ..., 654 n , for example, in the above reference Figure 5A-5D or Figure 6A-6C The method is described in the manner described (where the method is applied to the entire image). Each region may then be transformed using the transformation determined for that region. Alternatively, the determined transformation may be assigned to, for example, a central pixel in a region, and the remaining pixels may be transformed according to an interpolation scheme based on the transformation of the region including that pixel and the transformations of neighboring regions.

[0302] In an embodiment, each region may be a single pixel. In such an embodiment, the transformation may be determined based on the pixel value and based on the pixel values ​​of pixels in a region surrounding the pixel.

[0303] Fig. 7A and Figure 7B A flow chart of laser speckle contrast imaging combining more than two raw speckle images according to an embodiment is depicted. At a first time t=t1, a first raw speckle image 7021 based on light of a first wavelength may be obtained and may be used to calculate a first laser speckle contrast image 7041. A first corrected image 7061 based on light of at least a second wavelength and associated with the first raw speckle image may be obtained, and a first plurality of first features may be detected 7081 in the first corrected image. Items 7021-7081 may be obtained by performing step 7101, which may be similar to steps 302-308 as described with reference to FIG3 . In some embodiments, the first corrected image and the first raw speckle image may be the same image.

[0304] At a second time instance t=t2, step 7101 may be repeated as step 7102, thereby generating a second raw speckle image 7022 based on light of the first wavelength, a second laser speckle contrast image 7042, a second corrected image 7062 based on light of at least the second wavelength, and a plurality of second features 7082. Based on the plurality of first features 7081 and the plurality of second features 7082, a plurality of first alignment vectors 7121 may be calculated. Based on the plurality of first alignment vectors, a first transformation 7141 may be determined, which may be used to transform the first laser speckle contrast image 7041 to register the first laser speckle contrast image with the second laser speckle contrast image 7042, thereby obtaining a first registered laser speckle contrast image 7181. As described above, in some embodiments, the transformation may be determined without explicitly detecting features and / or alignment vectors.

[0305] Optionally, a first weight 7161 may be determined based on the plurality of first alignment vectors 7121. The weight may be related, preferably inversely related, to the length of the representative alignment vector (e.g., the maximum, average, or median alignment vector), or to the average or median length of the plurality of first alignment vectors, for example. In embodiments where the image is divided into a plurality of regions and a transformation is determined for each region, as described above with reference to Fig.6D As explained, the weight of each region may be determined based on the registration parameters associated with that region, or a single weight may be determined for the entire image, for example based on representative parameters such as maximum, average or median displacement. Fig. 9A and Fig. 9B The determination of the weighted average is discussed in more detail.

[0306] Then, the first registered laser speckle contrast image 7181 may be combined with the second laser speckle contrast image 7042 to obtain a first combined laser speckle contrast image 7201. For example, the combined laser speckle contrast image may be a pixel-by-pixel average or maximum value of the first registered laser speckle contrast image 7181 and the second laser speckle contrast image 7042. Optionally, the first weight 7161 and / or the second weight 7162 may be used to determine a weighted average.

[0307] At a third time t=t3, step 7101 may be repeated as step 7103, thereby obtaining a third raw speckle image 7023 based on light of the first wavelength, a third laser speckle contrast image 7043, a third corrected image 7063 based on light of at least the second wavelength, and a plurality of second features 7083. Based on the plurality of second features 7082 and the plurality of third features 7083, a plurality of second alignment vectors 7122 may be calculated. Based on the plurality of second alignment vectors, a second transformation 7142 may be determined. The second transformation may be used to transform the first combined laser speckle contrast image 7201 to register the first combined laser speckle contrast image with the third laser speckle contrast image 7043, thereby obtaining a first registered combined laser speckle contrast image 7221. Then, the first registered laser speckle contrast image 7181 may be combined with the second laser speckle contrast image 7042, thereby obtaining a first combined laser speckle contrast image 7201.

[0308] The first registered combined laser speckle contrast image 7201 may then be combined with the third laser speckle contrast image 7043 to obtain a second combined laser speckle contrast image 7202. Thus, the second combined laser speckle contrast image may include information from the first laser speckle contrast image 7041, the second laser speckle contrast image 7042, and the third laser speckle contrast image 7043. By repeating these steps, the nth image may include information from all previous n-1 images. Preferably, the weighting may then be skewed to give more recent images a higher weight than older images. This embodiment is particularly useful for streaming video, where each captured frame is processed and output with minimal delay. Another advantage of this method is that between two subsequent frames, it can be assumed that the motion is relatively small, which can speed up processing and can allow a wider range of algorithms to be used, as some algorithms may not perform well for large motions. In general, feature-based algorithms may be more reliable for relatively large displacements.

[0309] Figure 7B A flow chart depicts an alternative method for laser speckle contrast imaging for combining more than two raw speckle images according to an embodiment. In this embodiment, a predetermined number of images are combined into a single combined image. Steps 752 related to image acquisition, calculation of speckle contrast images, and determination of features 1-n -760 1-n Can refer to the above Fig. 7A Step 702 of the description 1-n -710 1-n same.

[0310] However, with Fig. 7A Unlike the method described in , all alignment vectors 762 are determined relative to a single reference image (eg, the first or last image in a sequence of n images). 1,2In the depicted example, the image is registered with the nth image. Thus, by applying the transformation 764 to the first speckle contrast image and the second speckle contrast image, respectively 1,2 , the first speckle contrast image and the second speckle contrast image are registered with the nth speckle contrast image. Therefore, the weight 766 is also determined based on the transformation parameters (e.g., average displacement) relative to the nth image. 1,2 In a final step, all n registered speckle contrast images may be combined into a single combined speckle contrast image 770 .

[0311] Figure 7B The method described in the above may result in a combined image having a Fig. 7A The method described in gives higher image quality, but at the expense of a greater delay between image capture and display. Therefore, this method is particularly advantageous for recording snapshot images.

[0312] In an embodiment, Figure 7B The method depicted in can be applied to a sliding group of images, such as the last n images of a video feed. In order to keep the time delay low, in this case n should preferably not be too large, for example, when the frame rate is, for example, 50-60fps, n can be about 5-20. Of course, the number of frames that can be processed also depends on the hardware and the algorithm, so a larger number of frames is still feasible. Therefore, some advantages of both methods can be combined. The advantage of using a relatively small number of frames is that dynamic phenomena (for example, the effects of a heartbeat) can be imaged. The advantage of a larger number of frames is that such transient effects can be filtered out, in particular when the number of frames is chosen to cover an integer multiple of the heartbeat and / or respiratory cycle.

[0313] Figure 8 A flow chart for calculating a corrected laser speckle contrast image according to an embodiment is depicted. Typically, one may be interested in the relative motion of one or more objects in a target area relative to one or more other objects in the target area; for example, the motion of body fluids or red blood cells relative to tissue. In this case, the noise in the signal derived from the moving object (the expected signal) can be compensated by a signal derived from a reference object (the reference signal). The basic principle is that the expected signal may include a first component based on the motion of the quantity of interest relative to the reference object, and a second component based on the motion of the entire target area relative to the camera. The reference signal may include only or mainly a component based on the motion of the entire target area relative to the camera. Therefore, the reference signal may be correlated with the second component of the expected signal. The correlation may be used to correct or compensate for the expected signal. For example, a correction term based on the signal strength of the reference signal may be added to the expected signal.

[0314] exist Figure 8In the depicted embodiment, target area 802 is illuminated with coherent light of a first wavelength (e.g., red or infrared light), and target area 812 is illuminated with coherent light of a second wavelength (e.g., green or blue light). Preferably, the light of the first wavelength is primarily scattered by an object or fluid of interest (e.g., blood). Preferably, the light of the second wavelength is mostly scattered by a surface of the target area and / or mostly absorbed by the object or fluid of interest. Preferably, the second wavelength is selected such that the reflectance of the second wavelength by the blood is at least 25%, at least 50%, or at least 75% less than the reflectance of the tissue. Preferably, the target area is illuminated simultaneously with the first and second wavelengths of light.

[0315] The scattered light of the first wavelength may result in a first raw speckle image, which may be captured by a first image sensor 804. The scattered light of the second raw speckle image may be captured by a second image sensor 814, which is preferably different from the first image sensor. The first raw speckle image may be referred to as a desired signal raw speckle image, and the second raw speckle image may be referred to as a reference signal image or a correction signal image.

[0316] Based on the first original speckle image, a first speckle contrast image may be calculated 806. Based on the second original speckle image, a second speckle contrast image may be calculated 806. Preferably, the speckle contrast is calculated in the same manner for the first original speckle image and the second original speckle image. Figure 3A Step 304 illustrates the way to calculate speckle contrast.

[0317] In a next step 808, a corrected speckle contrast image may be calculated based on the first speckle contrast image and the second speckle contrast image. Calculating the corrected speckle contrast image may include, for example, adding a correction term or multiplying a correction factor. The correction term or correction terms may be based on a determined amount of speckle contrast in the speckle contrast image compared to a reference amount of speckle contrast. The reference amount of speckle contrast may, for example, be predetermined, or may be dynamically determined based on, for example, amounts of speckle contrast in a plurality of previous second contrast images, or based on a speckle contrast image with very little motion, as determined, for example, by a motion correction algorithm as described above.

[0318] The corrected speckle contrast image may then be stored 810 for further processing, e.g., as referenced Figure 3A and Figure 3B The second raw speckle image and / or the second speckle contrast image may also be stored 818 for further processing, for example, to determine an alignment vector in a plurality of second raw speckle images to realign or realign a plurality of simultaneously captured corrected speckle contrast images. In an embodiment, steps 802-818 may be replaced by Figure 3B Steps 332-336 in .

[0319] Therefore, an advantage of the method of the present disclosure is that the second wavelength image can be used for multi-spectral coherence correction (or "dual laser correction"), as shown in reference Figure 8 As described, and used to register the speckle contrast images, as in reference Figure 3A and Figure 3B As described.

[0320] Fig. 9A Schematically depicts the determination of a motion compensated speckle contrast image based on a weighted average according to an embodiment. 1-N The size of the transformation may be determined based on a registration parameter defining the determined transformation and / or based on a plurality of alignment vectors. For example, the size of the transformation may be based on the length or a statistical representation of the plurality of alignment vectors, such as their average value 904 1-N , or based on the matrix norm of the matrix representing the transformation. Non-feature-based image registration methods typically provide only a set of global registration parameters, or a limited set of registration parameters; in this case, the weights can be derived directly from the registration parameters rather than from a set of alignment vectors. Optionally, different types of transformations (e.g., translation, rotation, and scaling) can be given different weights.

[0321] The combined speckle contrast image 906 may be a weighted average of the speckle contrast images in the sequence of registered speckle contrast images, each image being weighted by a weight parameter w'.

[0322] The weight parameter w′ can be based on the alignment vector ‖p i ′-p i ‖ is determined, where high displacement corresponds to small weight and vice versa, for example, as defined in equation (11):

[0323] Here, the alignment vector can be given by the coordinates (x i ′,y i ′) defines a total of P points p i ' and the coordinates (x i ,y i ) defines the point p i to define.

[0324] The weight parameter w′ may also be determined based on dense or sparse optical flow parameters defining the optical flow between the first image (usually the reference image) and the second image. For example, the weight may be inversely related to the average optical flow over all pixels in the image or in a region of interest in the image, for example, as defined in equation (12):

[0325] Here, v ij ′ is the optical flow of pixel (i, j) including the x and y components of the optical flow, and W and H are the width and height of the pixel of the reference image or reference region of interest, respectively.

[0326] Alternatively, the weight parameters may be determined for each pixel, for example, based on the optical flow of each pixel as defined in equation (13):

[0327] Instead of determining weight parameters for each pixel or for an image temporarily, weight parameters may also be determined for predefined regions of the image, such as rectangular or triangular grids. For example, the weight parameters may be based on the average optical flow in the corresponding predefined region.

[0328] It can also be based on the speckle contrast value s ij ′ is used to determine the weight parameter, preferably, the weight parameter is proportional to the speckle contrast amplitude, for example, as defined in equation (14):

[0329] Here, s ij ' is the speckle contrast of the reference image, and W and H are the width and height in pixels of the reference image or reference region of interest, respectively. The advantage of using speckle contrast is that any noise occurring in a small temporal window over the speckle contrast will blur the speckle contrast. Such noise may be due to, for example, motion of the image object or camera, or to other sources such as loose fiber connections. The advantage of using weights based on alignment vectors or optical flow is that they can have a higher temporal resolution for LSCI, whereas weights based on speckle contrast may lag, especially when perfusion increases.

[0330] Alternatively, the weight parameters can be determined based on the speckle contrast per pixel.

[0331] Alternatively, the weight parameters can be determined based on average speckle contrast in a predefined area, such as a square grid or a triangle grid.The weight parameters may also be determined for dynamically determined areas, where the areas may be determined, for example, based on detected motion.

[0332] In embodiments, various of these weights may be combined. For example, the weights may be normalized and added, multiplied or compared, such that, for example, the lowest weight is selected. Alternatively, a first weight based, for example, on speckle contrast values ​​may be used to filter out images that do not meet a predefined quality criterion, e.g., images with a contrast magnitude drop exceeding a predetermined threshold receive a weight of 0, and all other images receive a weight of 1. Subsequently, a weight based on optical flow or displacement may be used to determine a weighted average of images that have not been filtered out.

[0333] By using N images Img k The buffer and the corresponding N weight factors w k The buffer uses a single weight factor for each image to determine a weighted average, where k = 1, 2, ..., N. For each new image acquired, the following steps can be performed: 1) Add the new image to the buffer as Img N+1 , which can be selected as a reference image; 2) removing the first weight factor w1 and the first image Img1 from the buffer; 3) Apply the geometric transformation to other images Img2, Img3, ..., Img in the buffer N , to register them to the reference image Img N+1 If the images have been previously registered, for example to a previous reference image, the same transformation can be applied to all images Img 2-N If the unregistered images are stored in a buffer, the transformation may be determined separately for each image in the buffer and applied to each image in the buffer; 4) Calculate the weight factor w as described above N+1 and add it to the buffer; 5) Normalize the weight factor w2 to w N+1 , for example according to equation (16): Or according to equation (17): Here, β1 is a constant that can be positive or negative, and w min is defined as the minimum weight in the buffer and β2 is a constant that should be greater than zero. To avoid negative weights and division by zero, any weight that is negative or zero can be set to a small positive number. min The advantage of the second normalization is that it increases the impact of the weighting factors. Increasing β1 or β2 will reduce the impact of the weighting factors and make the algorithm behave more like an averaging algorithm, while decreasing β1 or β2 will increase the impact of the weighting factors. These constants can be predetermined based on the application. 6) Calculate the high-quality combined image Img by calculating the weighted average using the previously calculated weights N+1 ', for example, as defined in equation (18): Here, i and j are used to index the pixels of the image.

[0334] In an alternative embodiment, the image buffer may include only the combined image. In such an embodiment, the image buffer may be composed by using N weight factors w corresponding to the N most recent images. k A buffer and a combined image Img based on the N most recent images N 'To determine the weighted average, where k = 1, 2..., N. For each new image Img N+1 , you can take the following steps: 1) Add the new image to the buffer as Img N+1 , which can be selected as a reference image; 2) removing the first weight factor w1 from the buffer; 3) Apply the geometric transformation to the stored image Img N ', to align it to the reference image Img N+1 . 4) Calculate the weight factor w as described above N+1 and add it to the buffer; 5) For example, according to equation (16) or (17), the weight factor w2 is normalized to w N+1 ; 6) By calculating the stored image Img′ N and the reference image Img N+1 The weighted average of the high-quality combined image Img is calculated N+1 ', for example, as defined in equation (19): Img′ N+1 (i,j)=(1-w N+1 )·Img N (i,j)+w N+1 ·Img N+1 (i,j) (19) where i and j are used to index the pixels of the image; and 7) Remove Img from the buffer N ' and Img N+1 'Add to the buffer.

[0335] The advantage of this algorithm is that the calculation is much faster since only one geometric transformation needs to be applied and the amount of processing steps is low. The size of the buffer storing the weight factors determines how large the influence of the historical images is compared to the influence of the new images. When the buffer is small, the new images will be more present in the final image, while if the buffer is large, the new images will be less present and images with high weight factors will be more present.

[0336] As explained above, weights may be determined for the image as a whole, for each pixel in the image, or for predetermined or dynamically determined regions in the image. Similarly, a single transformation may be applied to each image as a whole, each pixel may be transformed individually, or a transformation may be determined to apply to predetermined or dynamically determined regions. Thus, weights may be defined as scalars, matrices, or in different formats.

[0337] An advantage of using an algorithm based on a geometric transformation, eg a rectangular or triangular mesh, and using weights determined per mesh segment is that such an algorithm may be more robust to registration errors while still being able to locally correct for noise such as local motion.

[0338] A weight based on the amount of displacement or transformation can be quickly determined for each image independently of the other images. Images with large amounts of displacement are typically noisier and can therefore be assigned a lower weight, thereby improving the quality of the combined image.

[0339] Alternatively or additionally, a normalized amount of speckle contrast or an amount of change in speckle contrast relative to one or more previous and / or subsequent images in the sequence of first speckle contrast images may be determined for each first raw speckle image. In this case, a weighted average may be determined using a weight based on the determined normalized amount of speckle contrast or the determined change in speckle contrast associated with the corresponding first speckle contrast image.

[0340] Weights based on differences or changes (especially sudden changes) in speckle contrast can be indicative of image quality. In general, speckle contrast, and therefore these weights, can be affected by various factors in the overall system, such as motion of the camera relative to the target area, movement of the optical fiber or other factors affecting the optical path length, loose connections, or fluctuating lighting conditions. Therefore, using weights based on speckle contrast, a higher quality combined image can be obtained. In general, speckle contrast is determined in relative units, so the weights can be determined by analyzing a sequence of raw speckle images. Since speckle contrast is negatively correlated with perfusion, perfusion units based on speckle contrast can be similarly used.

[0341] Preferably, the images may be normalized in such a way that the relationship between speckle contrast and weight is linear with a constant. In this case, correction based on speckle contrast may be more real-time, since each image may be normalized directly without reference to a time window of the image. Alternatively, a delta average may be used.

[0342] Fig. 9B Determining a motion compensated speckle contrast image based on a weighted average according to an embodiment is schematically depicted. In this example, a circular image buffer 910 is used. When the image buffer is updated, the oldest image is removed 912 from the buffer, the image in the buffer is transformed 914 using the determined image registration parameters, and the new image is added 916 to the buffer. Masks are processed in a similar manner. A circular mask buffer 920 of equal size to the image buffer contains masks associated with images in the image buffer. When the mask buffer is updated, the oldest mask is removed 922 from the buffer, the mask in the buffer is transformed 924 using the determined image registration parameters, and the new mask associated with the new image is added 916 to the mask buffer. The masks in the mask buffer may be blurred (e.g., using a combination of Gaussian blur or box filter) to reduce sensitivity to image registration inaccuracies.

[0343] The masks in the updated buffer may then be averaged and normalized 928 to obtain an averaged normalized mask. The average is typically a normal average, but a weighted average can similarly be used, for example, based on the above reference Fig. 9A Thus, pixels in the mask have similar or equal weights to pixels in the weighted average 918 of the speckle contrast image.

[0344] In step 918, a weighted average of the images in the image buffer is determined. The weighted average may be referred to as a temporal filter. The weighted average may use several weights. For example, motion-based weights may be used as described above. The weights may be local (pixel-based), regional, or global (image-based) weights. Additionally, mask-based weights may be used. Mask-based weights are typically applied locally, i.e., on a pixel-by-pixel basis.

[0345] If an average normalized mask has been determined 928, a perfusion value may be calculated when the average normalized value is below a predetermined value (assuming that a high mask value indicates a low reliability score), e.g., below 0.2. This indicates that in most input images, the pixel value is considered sufficiently reliable. If the average normalized value is above a predetermined value, the corresponding pixel in the combined image may be given, e.g., an error value, an interpolated value, or no value, as described above.

[0346] The pixels for which perfusion values ​​are calculated can then be determined by calculating a weighted average, where each pixel in the input image in the image buffer is given a weight based on the associated mask in the mask buffer. In some cases this is a binary weight, in other cases a multi-valued weight may be used.

[0347] This results in a combined image 930, which can then be displayed or used for further processing. In some embodiments, the combined image is shown, for example, as an overlay on a different image (which can be referred to as an underlying image) (e.g., the most recent image in the image buffer or an associated white light image). In some cases, the underlying image can be modified based on a mask associated with the image; for example, the intensity of specular reflections in the underlying image can be mitigated by reducing the corresponding pixel values. This can result in a less distracting image.

[0348] Similar implementations can be used for other temporal filters, such as the decay buffer.In general, the processing of masks in the mask buffer should be similar to the processing of images in the image buffer.

[0349] Fig.10 1 is a block diagram showing an exemplary data processing system described in the present disclosure. Data processing system 1000 may include at least one processor 1002 coupled to memory element 1004 via system bus 1006. In this way, the data processing system can store program codes in memory element 1004. In addition, processor 1002 can execute program codes accessed from memory element 1004 via system bus 1006. In one aspect, the data processing system can be implemented as a computer suitable for storing and / or executing program codes. However, it should be understood that data processing system 1000 can be implemented in the form of any system including a processor and a memory capable of performing the functions described in this specification.

[0350] Memory element 1004 may include one or more physical memory devices, such as, for example, local memory 1008 and one or more mass storage devices 1010. Local memory may refer to a random access memory or other non-persistent memory device that is typically used during the actual execution of program code. Mass storage devices may be implemented as hard drives or other persistent data storage devices. Processing system 1000 may also include one or more cache memories (not shown) that provide temporary storage of at least some program codes to reduce the number of times program codes must be retrieved from mass storage devices 1010 during execution.

[0351] Input / output (I / O) devices, depicted as key device 1012 and output device 1014, are optionally capable of being coupled to the data processing system. Examples of key devices may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, and the like. Examples of output devices may include, but are not limited to, for example, a monitor or display, a speaker, and the like. The key device and / or the output device may be coupled to the data processing system directly or through an intermediate I / O controller. A network adapter 1016 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intermediate private or public network. The network adapter may include: a data receiver for receiving data sent to the data by the system, device, and / or network; and a data transmitter for sending data to the system, device, and / or network. Operation modems, cable operation modems, and Ethernet cards are examples of different types of network adapters that may be used with data processing system 1000.

[0352] like Fig.10 As shown, memory element 1004 can store application 1018. It should be appreciated that data processing system 1000 can further execute an operating system (not shown) that can facilitate the execution of applications. Applications implemented in the form of executable program code can be executed by data processing system 1000 (e.g., by processor 1002). In response to executing the application, the data processing system can be configured to perform one or more operations described in further detail herein.

[0353] In one aspect, for example, data processing system 1000 may represent a client data processing system. In that case, application 1018 may represent a client application that, when executed, configures data processing system 1000 to perform the various functions described herein with reference to a "client." Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, and the like.

[0354] On the other hand, the data processing system may represent a server. For example, the data processing system may represent a (HTTP) server, in which case the application 1018, when executed, may configure the data processing system to perform (HTTP) server operations. On the other hand, the data processing system may represent a module, unit, or function mentioned in this specification.

[0355] The terms used herein are only used for the purpose of describing specific embodiments and are not restrictive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0356] The corresponding structures, materials, actions and equivalents of all means or step-plus-function elements in the following claims are intended to include any structure, material or action for performing a function in conjunction with other claimed elements specifically claimed. The description of the present invention has been presented for the purpose of illustration and description, but is not intended to be exhaustive or to limit the present invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments are selected and described in order to best illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the various embodiments of the present invention with various modifications suitable for the intended specific use.

Claims

1. A method for motion-compensated laser speckle contrast imaging, comprising: exposing a target area to coherent first light of a first wavelength, the target area comprising living tissue; capturing at least one sequence of images, the at least one sequence of images comprising a first speckle image, the first speckle image being captured during exposure with the first light; determining one or more registration parameters of an image registration algorithm for registering the first speckle images with each other; as well as determining a registered first speckle image by registering the first speckle image based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or A first speckle contrast image is determined based on the first speckle image, a registered speckle contrast image is determined by registering the first speckle contrast image based on the one or more registration parameters and the image registration algorithm, and a combined speckle contrast image is determined based on the registered first speckle contrast image.

2. The method according to claim 1, wherein: The determination of the one or more registration parameters is based on a plurality of images, preferably, images of the plurality of images are selected from the first speckle image and / or from images associated with the first speckle image and / or from images derived from the first speckle image or images associated with the first speckle image.

3. The method according to claim 2, wherein: The registration parameters are based on a similarity measure of pixel values ​​in one or more pixel groups in each of the plurality of images.

4. The method according to claim 2 or 3, further comprising: determining a transformed image by transforming an image of the plurality of images, the transform preferably comprising one or more of: a Fourier transform, a Mellin transform, and a log-polar transform; Therein, the registration parameters are based on a comparison of the transformed images.

5. The method according to claim 4, wherein: The transform is a transform to the frequency domain, preferably a Fourier transform or a Fourier-Mellin transform, and wherein the comparison of the transformed images comprises: determining a cross-correlation of the transformed images; determining a transform of the cross-correlation into the spatial domain, preferably using an inverse Fourier transform; and A peak in the cross-correlation in the spatial domain is determined.

6. The method according to claim 4, wherein: The transformation is a log polar coordinate transformation; wherein the comparison of the transformed images comprises determining a displacement of the transformed images relative to each other; and Therein, determining the registration parameters comprises determining a rotation and / or a scaling based on the determined shift.

7. The method according to any one of the preceding claims, further comprising: determining a plurality of masks for the first speckle image, each mask in the plurality of masks being associated with a corresponding first speckle image, and each mask in the plurality of masks associating a reliability score with one or more pixels in the associated first speckle image; determining a plurality of registration masks by registering the plurality of masks based on the one or more registration parameters and the image registration algorithm; as well as The combined speckle contrast image is determined based on the plurality of registration masks.

8. The method according to claim 7, in, At least one of the plurality of masks is based on an artifact, preferably a specular artifact, identified in a corresponding speckle image; and / or Wherein at least one of the plurality of masks is based on pixels identified as not representing living tissue.

9. The method according to claim 7 or 8, in, Determining the plurality of masks comprises identifying deviating input pixel values ​​and / or deviating speckle contrast values, for example values ​​above a predetermined absolute or relative upper threshold or values ​​below a predetermined absolute or relative lower threshold; and / or Wherein determining the plurality of masks comprises identifying pixels that do not represent living tissue based on an image recognition algorithm.

10. The method according to any one of claims 7 to 9, wherein: The determination of the first speckle contrast image is based on a mask associated with the corresponding first speckle image.

11. The method according to any one of the preceding claims, further comprising: exposing the target area to a second light of one or more second wavelengths, preferably to coherent light of a second wavelength or to light of a plurality of second wavelengths comprising the visible spectrum, wherein the exposure to the second light is performed alternately with the exposure to the first light or simultaneously with the exposure to the first light; wherein the at least one sequence of images comprises a sequence of second images, the second images being captured during exposure to the second light; and The plurality of images are selected from the second image sequence or from images derived from the second image, and each of the second image or the images derived therefrom is associated with the first speckle image.

12. The method according to claim 11, wherein: The at least second wavelength of light is coherent light of a predetermined second wavelength, the predetermined second wavelength preferably being in the green or blue part of the electromagnetic spectrum, preferably in the range of 380 nm - 590 nm, more preferably in the range of 470 nm - 570 nm, even more preferably in the range of 520 nm - 560 nm, and wherein the second sequence of images is a second sequence of speckle images; The method further comprises: determining a second speckle contrast image based on the second speckle image sequence; and The first speckle contrast image is adjusted based on a change in speckle contrast amplitude in the second speckle contrast image sequence.

13. A method according to any one of the preceding claims, wherein: The first wavelength is a wavelength in the red part of the electromagnetic spectrum, preferably in the range of 600-700 nm, more preferably in the range of 620-660 nm, or a wavelength in the infrared part of the electromagnetic spectrum, preferably in the range of 700-1200 nm.

14. A method according to any one of the preceding claims, wherein: Determining the combined speckle contrast image also includes: An average value of the registered first speckle image and the registered first speckle contrast image is calculated respectively, the average value is preferably a weighted average value, and the weight of the image is preferably based on the registration parameter or on the relative amplitude of the speckle contrast.

15. A method according to claim 14 when dependent on any one of claims 7 to 9, wherein: The average is a weighted average, and wherein masked pixels have a weight based on the reliability score associated with the pixel.

16. The method of claim 15, further comprising determining a combined speckle contrast image mask, pixels in the combined speckle contrast image mask being associated with pixels in the combined speckle contrast image, The combined speckle contrast image mask indicates whether at most a predetermined percentage of the input pixels are masked, or The combined speckle contrast image mask indicates a reliability score.

17. The method according to claim 16, wherein: The pixels marked as having invalid pixel values ​​are removed or rendered transparent, or wherein the pixels marked as having invalid pixel values ​​are assigned error values, or wherein the pixels marked as having invalid pixel values ​​are assigned interpolated values ​​based on surrounding pixel values.

18. A method according to any one of the preceding claims, wherein: The one or more pixel groups represent a predetermined feature in the plurality of images, the predetermined feature preferably being associated with an object in the target region, preferably being associated with an anatomical structure in the target region.

19. The method according to claim 18, further comprising: The plurality of images are filtered using a filter adapted to increase the probability that a group of pixels represents a feature corresponding to an anatomical feature.

20. A method according to any one of the preceding claims, wherein: The one or more pixel groups are selected based on pixel coordinates, preferably, the pixel groups in a first image from the plurality of images are smaller than the pixel groups in a second image from the plurality of images.

21. A method according to any one of the preceding claims, wherein: Determining one or more registration parameters includes: determining a plurality of associated pixel groups based on the similarity measure, each pixel group belonging to a different image from the plurality of images, determining a plurality of alignment vectors based on the positions of the pixel groups relative to respective images from the plurality of images, the alignment vectors representing motion of the target region relative to the image sensor; and The registration parameters are determined based on the plurality of alignment vectors.

22. The method according to any one of the preceding claims, further comprising: dividing the first speckle image, and accordingly each of the first speckle contrast images and each of the plurality of images, into a plurality of regions, preferably into disjoint regions; and wherein Determining the registration parameters includes determining the registration parameters for each region; as well as Determining a sequence of registered first speckle images, respectively first speckle contrast images, comprises registering each region of the first speckle images, respectively first speckle contrast images, based on a transformation according to a corresponding region in the second image.

23. A method according to any one of the preceding claims, wherein: The target area comprises a perfused organ, preferably perfused by body fluid, more preferably perfused by blood and / or lymph, and / or comprises one or more blood vessels and / or lymphatic vessels, the method further comprising: A perfusion intensity, preferably a blood perfusion intensity or a lymphatic perfusion intensity, is calculated based on the combined speckle image.

24. A method according to any preceding claim, further comprising displaying the combined speckle contrast image or an image derived therefrom.

25. A hardware module for an imaging device, preferably a medical imaging device, comprising: a first light source for exposing a target area to a coherent first light of a first wavelength, the target area comprising living tissue; at least one image sensor system for capturing at least one image sequence, the at least one image sequence comprising a first speckle image, the first speckle image being captured during exposure with the first light; A computer-readable storage medium having a computer-readable program code embedded therein, and a processor, preferably a microprocessor, more preferably a graphics processing unit, coupled to the computer-readable storage medium, wherein, in response to executing the computer-readable program code, the processor is configured to: determining one or more registration parameters of an image registration algorithm for registering the first speckle images with each other; as well as determining a registered first speckle image by registering the first speckle image based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or A first speckle contrast image is determined based on the first speckle image, a registered speckle contrast image is determined by registering the first speckle contrast image based on the one or more registration parameters and the image registration algorithm, and a combined speckle contrast image is determined based on the registered first speckle contrast image.

26. The hardware module according to claim 25, further comprising: a second light source for illuminating the target area with light of at least a second wavelength different from the first wavelength simultaneously or alternately with the first light source; wherein the at least one image sensor is further configured to capture a second sequence of images, the second images being captured during exposure to the second light; and The plurality of images are selected from the second image sequence, and each of the second images is associated with a first speckle image.

27. The hardware module according to claim 25 or 26, further comprising: A display for displaying the combined speckle image and / or an image derived therefrom, preferably a perfusion intensity image.

28. The hardware module according to any one of claims 25 to 27, wherein: The processor is further configured to perform any of the method steps according to any of claims 2-24.

29. A medical imaging device, comprising a hardware module according to any one of claims 25-27, wherein the device is preferably one of the following: an endoscope, a laparoscope, a surgical robot, a handheld laser speckle contrast imaging device or an open surgical laser speckle contrast imaging system.

30. A computing module for a laser speckle imaging system, comprising a computer-readable storage medium and a processor, the computer-readable storage medium having at least a portion of a program embedded therein, the processor coupled to the computer-readable storage medium, the processor preferably being a microprocessor, more preferably being a graphics processing unit, wherein: In response to executing the computer readable storage code, the processor is configured to perform executable operations, the executable operations comprising: receiving at least one image sequence, the at least one image sequence comprising a first speckle image captured during exposure of a target area to coherent first light of a first wavelength, the target area comprising living tissue; determining one or more registration parameters of an image registration algorithm for registering the first speckle images with each other; and determining a registered first speckle image by registering the first speckle image based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or A first speckle contrast image is determined based on the first speckle image, a registered speckle contrast image is determined by registering the first speckle contrast image based on the one or more registration parameters and the image registration algorithm, and a combined speckle contrast image is determined based on the registered first speckle contrast image.

31. The computing module according to claim 30, in, The at least one sequence of images comprises a sequence of second images, the second images being captured during exposure to second light, the second light having one or more second wavelengths, preferably the second light being coherent light of a second wavelength or the second light comprising a plurality of second wavelengths of the visible spectrum, wherein exposure to the second light is performed alternately with exposure to the first light or simultaneously with exposure to the first light; and The plurality of images are selected from the second image sequence, and each of the second images is associated with a first speckle image.

32. The computing module according to claim 30 or 31, wherein: The executable operations also include any one of the method steps according to any one of claims 2-24.

33. A computer program or a computer program suite comprising at least one software code portion or a computer program product storing at least one software code portion, which software code portion is configured to perform the method steps according to any one of claims 1 to 24 when run on a computer system.

Images