Motion-compensated laser speckle contrast imaging

Abstract

A method and system for motion corrected and motion compensated laser speckle contrast imaging is disclosed. The method includes exposing a target area to a coherent first light of a first wavelength, where the target area includes biological tissue; and capturing at least one sequence of a plurality of images. The at least one sequence of a plurality of images includes a plurality of first speckle images, the plurality of first speckle images being captured during exposure with the first light. The method further includes determining one or more registration parameters of an image registration algorithm to register the plurality of first speckle images to one another. The method may further include determining a registered plurality of first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle images; or determining a plurality of first speckle contrast images based on the plurality of first speckle images, determining a registered plurality of speckle contrast images by registering the plurality of first speckle contrast images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle contrast images.

Description

[Technical field]

[0001] The present disclosure relates to motion-compensated laser speckle contrast imaging, and in particular, but not exclusively, to methods and systems for motion-compensated laser speckle contrast imaging, modules for motion-compensation in laser speckle contrast imaging systems and methods, and computer program products that enable a computer system to perform such methods. [Background technology]

[0002] Laser speckle contrast imaging (LSCI) provides a fast, full-field, and in vivo imaging method for determining two-dimensional (2D) perfusion maps of living biological tissues. Perfusion can be an indicator of tissue viability and thus can provide useful information for diagnosis and surgery. For example, during intestinal surgery, selection of highly perfused intervention sites can reduce anastomotic leakage.

[0003] LSCI is based on the principle that backscattered light from tissue illuminated with coherent laser light forms a random interference pattern at the detector due to differences in optical path length. The resulting interference pattern is called a speckle pattern and can be imaged in real time using a digital camera. Particle movement within the tissue causes fluctuations in this speckle pattern, resulting in speckle blurring in those parts of the images where perfusion occurs.

[0004] For example, this blurring can be related to blood flow if the fluctuations are caused by the movement of red blood cells. Thus, blood perfusion can be imaged relatively easily in living tissue. An example of a state-of-the-art clinical perfusion imaging technique using LSCI is described in the review article "Clinical applications of laser speckle contrast imaging: a review', J. Biomed. Opt. 24:8 (2019)" by W. Heeman et al. Perfusion by other body fluids, such as lymphatic perfusion, can be imaged in a similar manner.

[0005] However, LSCI is extremely sensitive to any kind of motion: blurring can be caused not only by blood flow motion, but also by other kinds of motion, such as tissue motion due to breathing, heartbeat, muscle contractions, or even camera, especially handheld, motion.

[0006] In many medical applications, such as diagnosis and surgery, it is desirable for an LSCI system to be able to generate accurate, high-resolution blood flow images, especially microcirculation images, in real time with significantly reduced motion artifacts. Therefore, measures are needed to minimize motion artifacts during processing of raw speckle images so that accurate, high-resolution perfusion images can be obtained. This can improve the discrimination between well-perfused and poorly perfused areas, and thus improve diagnostic and treatment outcomes.

[0007] Various techniques for reducing motion artifacts in speckle images are known in the prior art. For example, International Publication WO2020 / 045015 A1 discloses a laser speckle contrast imaging system capable of capturing near-infrared speckle images and white light images of an imaging target. A simple motion detection technique may include using a reference marker on the image target to track feature points in a visible light image or changes in speckle shape in a speckle image to determine a global motion vector indicating the amount of motion of the image target between two subsequent images. A plurality of speckle contrast images may be generated based on the plurality of speckle images, and the amount of motion may be corrected based on the motion vector.

[0008] Motion is even more pronounced in handheld LSCI systems compared to systems supported by, for example, a tripod. For example, Lertsakdadet et al. in their paper 'Correcting for motion artefact in handheld laser speckle images', Journal of biomedical optics 23(2), March 2018, describe a motion compensation method for laser speckle imaging that uses fiducial markers attached to the tissue that needs to be imaged. The use of markers is not possible in many applications.

[0009] Similarly, P. Miao et al., in their paper 'High resolution cerebral blood flow imaging by registered laser speckle contrast analysis', IEEE transactions on bio-medical engineering 57(5):1152-1157, describe a method to generate high-resolution LSCI images by registering raw speckle images based on convolution filtering and correction / interpolation techniques. The registered images are then retrospectively analyzed using temporal laser speckle contrast analysis. However, such registration of raw speckle images requires significant computational resources and is therefore not suitable for accurate real-time imaging applications. Summary of the Invention [Problem to be solved by the invention]

[0010] It therefore follows from the above that there is a need in the art for improved motion compensated laser speckle contrast imaging techniques, in particular for improved methods and systems for laser speckle contrast imaging that enable real-time, robust, marker-free, high-resolution perfusion imaging, in particular microcirculation imaging, in which motion artifacts are substantially eliminated or at least reduced. [Means for solving the problem]

[0011] As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Functions described in the present disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in, e.g., stored on, one or more computer-readable medium(s) having computer-readable program code embedded in the computer program product.

[0012] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can hold or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0013] A computer-readable signal medium may include a propagated data signal with computer-readable program code embedded therein, for example, in baseband or as part of a carrier wave, in a data signal. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium and may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0014] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for performing operations related to aspects of the present invention may be written in any combination of one or more programming languages, including functional or object-oriented programming languages, such as Java™, Scala, C++, Python, etc., and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer, server, or virtualized server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to or from an external computer (e.g., through the Internet using an Internet Service Provider).

[0015] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, in particular a microprocessor or central processing unit (CPU), or a graphics processing unit (GPU), to produce a machine such that the instructions, executed via a processor of a computer or other programmable data processing device, or other device, create means for performing the functions / operations specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0016] These computer program instructions may also be stored on a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium constitute an article of manufacture including instructions that implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0017] The computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device, such that the instructions, executed on a computer, or other programmable apparatus, provide a process for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and cause the computer, other programmable apparatus, or other device to perform a series of operating steps to generate a computer-implemented process.

[0018] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may be performed out of the order noted in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or a combination of special-purpose hardware and computer instructions.

[0019] It is an object of the embodiments of the present disclosure to mitigate or eliminate at least one of the disadvantages known in the prior art.

[0020] In a first aspect, the present embodiments may relate to a method of motion compensated laser speckle contrast imaging, the method including exposing a target area to a coherent first light of a first wavelength, where the target area includes living tissue, and capturing at least one sequence of a plurality of images, the at least one sequence of a plurality of images including a plurality of first speckle images, the plurality of first speckle images being captured during exposure with the first light, the method further including determining one or more registration parameters of an image registration algorithm to register the plurality of first speckle images to one another. The method may further include either determining a registered plurality of first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle images, or determining a plurality of first speckle contrast images based on the plurality of first speckle images, determining a registered plurality of speckle contrast images by registering the plurality of first speckle contrast images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle contrast images.

[0021] Thus, a sequence of either multiple speckle images or multiple speckle contrast images can be registered or aligned without the use of markers. The registered images can then be combined into a single combined speckle contrast image with high resolution and accuracy. The single combined image can include information from the multiple speckle images, such as a pixel-wise average, preferably a weighted average, or the single combined image can be the single most reliable image, such as the image that has the least transformation (i.e., closest to an identity transformation using a suitable scale) compared to subsequent images in a sequence of multiple images, for example within a moving window of images.

[0022] In this way, the image is less sensitive to noise due to motion, which can be caused, for example, by camera motion, especially in handheld systems, or by patient motion, for example due to muscle contractions. Thus, embodiments in the present disclosure may enable or improve speckle contrast imaging in a wide range of applications, including, for example, perfusion imaging of the colon tract, which requires camera motion along the entire surface to be imaged, or perfusion imaging of skin burns, where it may be impossible for the patient to remain motionless due to pain.

[0023] The advantage of the method is that the method can be performed in (essentially) real-time using commonly available hardware. In some embodiments, there can be a small delay, for example, based on a certain number of frames (for example, 20 frames, or 20 frames of sufficient quality), a fixed amount of time (for example, 1 second), or a time based on physiological properties (for example, the time corresponding to one or two heartbeats or one or two breaths). Such delays are generally not detrimental to clinical use. Physiological properties can be based on the image analysis of the speckle image, the speckle contrast image, and / or the image from the multiple images, a predetermined constant based on knowledge of physiological phenomena, or can be based on external input, for example, from a heart rate monitor.

[0024] Another advantage of the method is that it does not require that a fiducial marker be placed within the field of view, which is particularly true when imaging areas where it is undesirable to place a fiducial marker, such as internal organs, burned tissue, or brain tissue, or relatively large image targets that would otherwise require multiple fiducial markers or repeated replacement of a single marker.

[0025] The method corrects and / or compensates for motion of the target area relative to the camera so that more reliable perfusion images can be produced. In some cases, for example, reliable images may be acquired faster or easier when a target with irregular motion is imaged and an operator would otherwise have to wait for a period of little motion to acquire an image. For example, motion correction may include transforming one or more images based on the apparent motion detected. Motion compensation may include combining multiple images, thereby improving the signal-to-noise ratio.

[0026] By registering the plurality of first speckle images or the plurality of first speckle contrast images, the plurality of first speckle images from the one sequence of the plurality of first speckle images can be used to determine a combined one speckle contrast image with enhanced contrast and / or spatial resolution compared to the plurality of one speckle contrast images separately. The number of registered plurality of speckle images to be combined can depend on clinical requirements and / or quality parameters of the plurality of first speckle images.

[0027] A plurality of speckle contrast images may be calculated based on a plurality of unregistered, and therefore untransformed, speckle images. The plurality of speckle contrast images may then be transformed and then combined. The transformation may distort the speckle pattern or a portion thereof, e.g., speckles may be enlarged, reduced, or distorted, especially for transformations more general than simple translation or rotation. Thus, calculating a plurality of speckle contrast images based on a plurality of untransformed speckle images may prevent noise due to the transformation from being introduced into the plurality of speckle contrast images.

[0028] Alternatively, the multiple speckle images can be first registered and then transformed, and then one speckle contrast image can be calculated. In this way, the temporal or spatiotemporal speckle contrast can be calculated based on two or more registered speckle images. Using the temporal or spatiotemporal speckle contrast can lead to higher spatial resolution. In this embodiment, the images are preferably registered with sub-pixel accuracy.

[0029] A further advantage of this embodiment is that only a single sequence of images is required, i.e., the same sequence of images can be used to register the images (to determine the registration parameters therefor), to determine the averaging weights, and to determine a combined laser speckle contrast image or a combined perfusion image. However, in other embodiments, a second sequence of images can be used to determine the registration parameters for image registration and to determine the averaging weights.

[0030] In one embodiment, determining the one or more registration parameters is based on a plurality of images, preferably an image in the plurality of images is selected from the plurality of first speckle images, and / or from an image associated with the plurality of first speckle images, and / or from an image derived from the plurality of first speckle images or an image associated with the plurality of first speckle images.

[0031] The images in the plurality of images can be, for example, images obtained from the respective first speckle images using image processing or images derived from the respective first speckle images. For example, the images derived from the respective first speckle images can represent the texture of the first speckle image, a normalized version of the first speckle image, a filtered (e.g., blurred or sharpened) first speckle image, or any other suitable image derived from the first speckle image. In other embodiments described below, a separate image is captured and associated with the first speckle image. The images in the plurality of images can also be images derived from an image associated with the first speckle image. As another example, an image in the plurality of images can be an image obtained by transforming the first speckle image, an image derived from the first speckle image, an image associated with the first speckle image, and / or an image derived from an image associated with the first speckle image.

[0032] In one embodiment, the registration parameters are based on a similarity measure of pixel values ​​within one or more pixel groups in each of a plurality of images of the at least one sequence of a plurality of images, where an image within the plurality of images is selected from the plurality of first speckle images and / or from an image associated with the plurality of first speckle images.

[0033] The one or more pixel groups may cover the entire image or only a portion of it. The pixel groups of different images may have the same size or different sizes. For example, if the pixel groups correspond to features, the groups corresponding to the same or similar features typically have the same or similar size. In another example, if the pixel groups correspond to positions in the image, the pixel groups corresponding to the same or similar positions in different images may have different sizes. Matching pixel groups based on one or more positions, typically having different sizes, may also be known as template matching.

[0034] An advantage of using pixel groups is that matching of pixel groups can typically be more reliable than whole images, especially when the pixel groups are relatively small compared to the image size. Moreover, the use of one or more pixel groups can improve the reliability of image registration when two images are related by a non-homogeneous transformation, where the transformation of the image group may correspond to a local transformation of the images.

[0035] In one embodiment, the method further includes determining a transformed image by transforming the plurality of first speckle images and / or an image associated with the plurality of first speckle images. The transformation may include one or more of a transformation to a frequency domain, such as a Fourier transformation, a Mellin transformation, a Laplace transformation, and a Radon transformation, and a coordinate transformation, such as a log-polar coordinate transformation. In such an embodiment, the registration parameters may be based on a comparison of the transformed images.

[0036] In general, the registration parameters may be determined using feature-based, intensity-based and / or frequency-based methods. The advantage of using a feature-based method is that it is related to real-world features and therefore less susceptible to image artifacts. Moreover, features may be selected such that only high quality data is used to determine the registration parameters, potentially improving accuracy and / or robustness. Moreover, feature-based methods are generally able to handle both translations and rotations and scaling of a single image, especially when the changes are relatively small compared to the image size. In principle, for feature-based methods, either intensity-based or frequency-based methods may be used to determine the features. The advantage of the intensity-based approach is that it is generally simple to implement and computationally less intensive to perform. The advantage of the frequency-based method is that it is less susceptible to local disturbances and image artifacts, since information from the entire image is used.

[0037] In one embodiment, the transformation is a transformation to frequency domain, preferably a Fourier transform.In such an embodiment, the comparison of the transformed images may include determining the cross-correlation of the transformed images, preferably using an inverse Fourier transform to determine the transformation of the cross-correlation to spatial domain, and determining a peak in the cross-correlation in the spatial domain.The determined peak position corresponds to the translation between two images.

[0038] In one embodiment, the transformation is a log-polar transformation, where comparing the transformed images includes determining a shift of the transformed images relative to one another; and where determining the registration parameters includes determining a rotation and / or a scaling based on the determined shift. The advantage of using the log-polar transformation is that the horizontal shift of the transformed image corresponds to the rotation of the untransformed image, and the vertical shift of the transformed image corresponds to the scaling of the untransformed image. Thus, rotation and scaling can be determined in a relatively simple manner, especially global scaling and rotation. The log-polar transformation is typically applied in combination with different methods to determine image translation.

[0039] The combination of the Fourier transform and the log-polar transform is sometimes referred to as the Fourier-Mellin transform.

[0040] The determined registration parameters may include, for example, a translation vector, a rotation angle, and a scale factor. Based on these registration parameters, the multiple first speckle images may be aligned (transformed) with respect to one another. These registration parameters may also be used to determine weights for a weighted average in which the multiple first speckle images or multiple first speckle contrast images may be combined. In general, neither the image registration itself nor the weight determination requires explicitly determining alignment vectors for multiple image regions.

[0041] An advantage of using large image regions to determine registration parameters, as is typical of template matching or frequency-based methods, is that noise may be suppressed, as compared to methods that use many small regions, for example. This is particularly relevant for registration parameters, such as rotation and scaling, when determined using a log-polar or Fourier-Melling transform, since rotation and scaling can be difficult to determine accurately in the untransformed spatial domain.

[0042] In one embodiment, the method further comprises determining a plurality of masks for the plurality of first speckle images. Each of the plurality of masks is associated with a respective first speckle image. Each of the plurality of masks can associate a reliability score with one or more pixels in the associated first speckle image. In such an embodiment, determining the combined speckle contrast can be based directly or indirectly on the plurality of masks.

[0043] The reliability score may be binary. Alternatively, the reliability score may have more than one potential value, e.g., 255 (byte value), or a floating point value (typically selected between 0 and 1). Different image types (e.g., speckle image, speckle contrast image, and combined speckle contrast image) may use different mask types, e.g., a binary mask for the speckle image, a byte-valued mask for the speckle contrast image. The mask may be determined for the entire image or only for one or more regions of interest within the image. For example, a speckle contrast value may have a reliability score based on the number of unreliable input pixel values ​​used to determine the speckle contrast. The reliability score may also be based on registration parameters (e.g., based on the amount of motion determined using optical flow, etc.). Depending on whether the registration parameters are determined globally (for the entire image), regionally (for image patches or groups of pixels), or locally (for each pixel), the confidence scores can likewise be determined globally, regionally, or locally. Confidence scores from different sources may be combined into a single confidence score.

[0044] In principle, the mask can be applied before or after the image registration (and application of the corresponding transformation), and before or after the calculation of the speckle contrast. The information encoded in the mask may be used several times during the calculation, or may be changed during the calculation.

[0045] In one embodiment, the method further comprises determining a plurality of registered masks by registering the plurality of masks based on the one or more registration parameters and the image registration algorithm. In such an embodiment, the determination of the combined speckle contrast may be based on the plurality of registered masks.

[0046] In one embodiment, at least one of the masks is based on an artifact, preferably a specular reflection artifact, identified in the respective speckle image. The artifact may be due to several causes. For example, specular reflection artifacts may occur (especially in endoscopy / laparoscopy setups) where a pixel or pixel group may be fully saturated, and therefore no contrast may be calculated. The artifact may also be due to, for example, defects in the pixel elements of the camera, dirt on the lens that collects light, etc.

[0047] The artifacts may be identified by identifying deviant input pixel values ​​and / or deviant speckle contrast values. For example, pixel values ​​above a predefined absolute or relative upper threshold or below a predefined absolute or relative lower threshold may be marked as deviant pixel values. Additionally or alternatively, pixel contrast values ​​below or above respective absolute or relative lower or upper thresholds may be identified. The relative threshold may be based, for example, on an analysis of the pixel's environment, such as a mean or median or other statistical representation.

[0048] In one embodiment, at least one of the masks is based on pixels that are identified as not representing biological tissue. For example, the image may include pixels that represent surgical instruments (especially in an open surgery setup), clamps, stitches, gauze, etc. It may be beneficial not to calculate speckle contrast or perfusion values ​​for these pixels, or at least not to display speckle contrast or perfusion values ​​for these pixels.

[0049] In such an embodiment, identifying pixels that do not represent biological tissue may be based on an image recognition algorithm. Such image recognition algorithms are known in the art. The image recognition algorithm may use a speckle image and / or a speckle contrast image as input. In an embodiment in which the at least one image sequence also includes other images other than a speckle image (e.g., a white light image), pixels that do not represent biological tissue may additionally or alternatively be identified in the other images. The identified pixels that do not represent biological tissue may be used to improve perfusion calculations in the biological tissue and / or to determine overall apparent motion, for example, by determining the inverse of a perfusion value determined for pixels that do not represent biological tissue and using the inverse to determine a motion correction value.

[0050] In one embodiment, the method further comprises exposing the target area to one or more second wavelengths of second light, preferably coherent light of a second wavelength or light comprising a plurality of second wavelengths in the visible spectrum, wherein the exposure to the second light alternates with the exposure to the first light or is simultaneous with the exposure to the first light.In this embodiment, the at least one sequence of a plurality of images comprises a sequence of a plurality of second images, wherein the plurality of second images are captured during the exposure to the second light; and the plurality of images are selected from the sequence of the plurality of second images or from a plurality of images derived from the plurality of second images, wherein each of the plurality of second images or each of the plurality of images derived from the plurality of second images is associated with a first speckle image.

[0051] Instead of a plurality of unprocessed second images, an image derived from the plurality of second images may be associated with the first speckle image, such as a texture image, a normalized image, or other preprocessed image.

[0052] Thus, in one embodiment, the method of laser speckle contrast imaging includes exposing a target area alternately or simultaneously to a coherent first light of a first wavelength and a second light of one or more second wavelengths, preferably coherent light of a second wavelength or light including a plurality of second wavelengths in the visible spectrum, where the target area preferably includes biological tissue. The method may further include capturing a sequence of a plurality of first speckle images during the exposure with the first light, and a sequence of a plurality of second images during the exposure with the second light, where the speckle images of the sequence of a plurality of first speckle images are associated with images of the sequence of a plurality of second images. One or more registration parameters of a registration algorithm for registering at least a portion of the sequence of a plurality of first speckle images may be determined based on a similarity measure of pixel values ​​of pixel groups in at least a portion of the sequence of a plurality of second images associated with the plurality of first speckle images. The method may further include determining a registered plurality of first speckle images by registering the at least a portion of the sequence of a plurality of first speckle images based on the one or more registration parameters and the registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle images. Alternatively, the method may include determining a plurality of first speckle contrast images based on the at least a portion of the sequence of a plurality of first speckle images, determining a registered plurality of speckle contrast images by registering the plurality of first speckle contrast images based on the one or more registration parameters and the registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle contrast images.

[0053] In the first embodiment defined above, the first light and the second light are the same light having the same wavelength, and the one sequence of the first speckle images and the one sequence of the second images are the same sequence of images. However, in an advantageous embodiment, at least one of the one or more second wavelengths is different from the first wavelength, so that the second images are different from the first speckle images. The first speckle image and the associated second image can also be different parts of a single image. The images in the one sequence of images can be frames in a video stream or a multi-frame snapshot. In the present disclosure, the second images can also be referred to as correction images. Typically, at least one pixel group in each of the at least a portion of the one sequence of the second images is used to determine the one or more registration parameters.

[0054] Thus, the first wavelength and the capture of the sequence of a plurality of first speckle images may be optimized for speckle contrast imaging (e.g., optimized exposure time) depending on the quantity to be measured, while the capture of the one or more second wavelengths and the sequence of a plurality of second images may be optimized to obtain images that can be easily and accurately registered, for example, by ensuring high contrast between anatomical features, such as between blood vessels and normal tissue. Typical combinations may also be a speckle image acquired using infrared light and a second image acquired using white light, or a speckle image acquired using red light and a second image acquired using green or blue light, although other combinations are of course possible.

[0055] In one embodiment, pixel groups may represent a predetermined feature within the plurality of images, where the predetermined feature is preferably associated with an object, preferably an anatomical structure, within the target area. Pixel groups may be selected by a feature detection algorithm.

[0056] The features may be, for example, features associated with physical objects in the target area, such as features related to blood vessels, rather than image features not directly related to objects, such as overexposed image parts or speckles. Predetermined features may be determined, for example, by belonging to a class of features, such as corners or areas with large intensity differences. They may further be determined, for example, by quality measures, constraints on mutual distances between features, etc. In some embodiments, a neighborhood of one or more determined features may be used to determine the displacement vector and / or the transformation.

[0057] Determining the displacement based on a relatively small number of features compared to the total number of pixels can significantly reduce computation time while still providing accurate results, especially in cases of relatively simple motion, for example where the entire target area is displaced due to camera motion.

[0058] In one embodiment, the method may further include filtering the plurality of images with a filter adapted to increase the probability that pixel groups represent features corresponding to anatomical features. For example, the filter may determine over-exposed and / or under-exposed areas and / or other image artifacts and create a mask based on those areas or artifacts. Thus, features related to those areas or artifacts may be prevented from being determined.

[0059] In one embodiment, determining the registration parameters based on similarities in pixel values ​​of groups of pixels may include determining, for each group of pixels in an image from the plurality of images, a convolution or cross-correlation with at least a portion of a different image from the plurality of images. The method may also include determining a polynomial expansion to model pixel values ​​in a neighborhood of a pixel in the plurality of images, comparing expansion coefficients, and determining an alignment vector based on the comparison.

[0060] In one embodiment, determining one or more registration parameters may include: determining a plurality of associated pixel groups based on the similarity measure, where each pixel group belongs to a different image from the plurality of images; determining a plurality of alignment vectors based on positions of the pixel groups relative to a respective image from the plurality of images, where the alignment vectors represent movement of the target area relative to an image sensor; and determining the registration parameters based on the plurality of alignment vectors.

[0061] By determining an alignment vector for multiple features or pairs of corresponding features, any motion of the camera relative to the imaging target can be determined and corrected. The alignment vector can be used, for example, to determine an affine transformation, a projective transformation, or a homography, which can correct, for example, image translation, rotation, scaling, and shear, based on image data alone. Thus, the method does not require information about, for example, the distance between the camera and the target, the angle of incidence, etc.

[0062] The determination of the alignment vector and / or the determination of the one or more transformations may be based on optical flow parameters determined using a suitable optical flow algorithm, for example a dense optical flow algorithm or a sparse optical flow algorithm.

[0063] In one embodiment, determining a combined speckle contrast image may further include calculating an average of the registered first speckle images or an average of the registered first speckle contrast images, where the average is preferably a weighted average, and the weight of an image is preferably based on the registration parameters or based on the relative magnitude of the speckle contrast. Alternatively, combining the speckle images or the speckle contrast images may include filtering at least a portion of the first speckle images or the first speckle contrast images, such as with a median filter or a minimum or maximum filter. The weight of the weighted average may be based on a quantity derived from the speckle contrast or derived from the alignment vectors, for example.

[0064] In one embodiment, calculating a combined laser speckle contrast image can include calculating an average, preferably a weighted average, of the registered sequence of speckle images.

[0065] In one embodiment, the average is a weighted average, and masked pixels have a weight equal to zero. If the mask defines a multivalued reliability score, the weight may depend on the reliability score, where pixels with higher reliability scores have a higher weight than pixels with lower reliability scores.

[0066] In one embodiment, the method further includes determining a combined speckle contrast image mask, which may indicate whether up to a predetermined percentage of the input pixels are masked. For example, the weighted average is calculated only if less than a predetermined percentage of the input pixels are masked, and if more than a predetermined percentage of the input pixels are masked, the pixels are marked as having an invalid pixel value.

[0067] Alternatively, the combined speckle contrast image mask may indicate a confidence score for the pixels in the combined speckle contrast image mask, which may depend on the proportion of masked pixels in the weighted average and / or on the confidence scores of the individual masks.

[0068] The speckle contrast or perfusion values ​​are typically displayed as an overlay on top of the captured image. There are various options for handling pixels in the overlay for which a valid perfusion value could not be calculated. For example, pixels marked as having an invalid pixel value may be removed or rendered as transparent, assigned an error value, or assigned a value based on an interpolation of the surrounding pixel values. Two or more of these options may be combined, for example, based on the cause of the invalid pixel value and / or based on the size of the region of connected invalid pixel values. For example, invalid pixel regions due to the presence of a non-tissue object (e.g., a surgical instrument) may be shown as transparent so that the instrument or other non-tissue object can be clearly seen. As a further example, small invalid pixel regions may be filled using interpolation to provide a clean image with the most likely correct values, while large invalid pixel regions may be assigned an error value to indicate that valid perfusion data was not acquired.

[0069] Where a confidence score is assigned to each pixel, or at least to each pixel in a relevant region, the confidence scores may be rendered with varying transparency (such as using an alpha channel), with more reliable pixels having lower transparency (higher opacity) and less reliable pixels having higher transparency (lower opacity).

[0070] In one embodiment, the method may further include determining, for each first speckle image or each first speckle contrast image associated with an image from the plurality of images, a transformation size associated with the respective first speckle image or first speckle contrast image based on the plurality of alignment vectors, preferably based on lengths of the plurality of alignment vectors and / or parameters defining the determined transformation. The weighted average may be determined using weights based on the determined transformation sizes associated with the respective first speckle contrast images, preferably the weights being inversely related to the determined transformation sizes.

[0071] Weights based on the size or amount of displacement or on the size or amount of transformation can be rapidly determined for each image, independent of other images. The transformation size can be based, for example, on the norm of a matrix representing the transformation, or on the norm of a matrix representing the difference between the transformation and the identity transformation. The transformation size can also be based, for example, on a statistically representative measure of the alignment vectors, such as the mean, median, n-th percentile, or maximum alignment vector length. Images with a large amount of displacement are generally noisier and therefore may be assigned a lower weight, thereby improving the quality of the combined image.

[0072] In one embodiment, the method may further include determining, for each first speckle image, a normalized speckle contrast amount or a change in speckle contrast relative to one or more preceding and / or subsequent images in the sequence of a plurality of first speckle contrast images. The weighted average may be determined using a weight based on the determined normalized speckle contrast amount or the determined change in speckle contrast associated with each of the first speckle contrast images. Alternatively or additionally, a weight may be determined based on a normalized speckle contrast amount or a change in speckle contrast for the plurality of second speckle contrast images.

[0073] Weights based on differences or changes, especially abrupt changes, in speckle contrast can indicate image quality. Typically, speckle contrast, and therefore these weights, can be affected by various factors in the overall system, such as the movement of the camera relative to the target area, the movement of the fiber, or other factors that affect the optical path length or change the lighting conditions. Thus, using weights based on speckle contrast can result in a higher quality combined image. Typically, speckle contrast is determined in arbitrary units, so that weights can be determined by analyzing the sequence of multiple speckle images. Because speckle contrast is inversely related to perfusion, perfusion units based on speckle contrast can be used as well.

[0074] In one embodiment, the algorithm may be applied to a predefined region of interest within the field of view of the camera. Such a region of interest may be determined by a user or may be predefined. For example, the outer boundaries of the image may be ignored and / or hidden from view, e.g. to prevent the transformed image boundaries from becoming visible. It may be faster to apply the algorithm or a part of the algorithm to only a part of the image. The region of interest may be transformed based on the determined transformation. In a different embodiment, the algorithm may be applied to the entire image.

[0075] In one embodiment, the multiple images can be the one sequence of multiple first speckle images.In such an embodiment, the first wavelength is preferably a wavelength in the green or blue part of the electromagnetic spectrum.In this way, a balance can be achieved between good speckle signal and good visual distinguishability (i.e., high contrast of anatomical features to be distinguished from high speckle contrast), which is advantageous for determining features.Therefore, no pre-processing step is required to increase the contrast of the one speckle image.

[0076] Alternatively, a first wavelength within the red portion of the electromagnetic spectrum, preferably 600-700 nm, more preferably 620-660 nm, or within the infrared portion of the electromagnetic spectrum, preferably 700-1200 nm, may be used. Depending on the type of tissue and imaging parameters, such as exposure time, visual distinctiveness may be sufficient for proper determination of features. Since red and infrared light are mostly reflected by red blood cells, these wavelengths result in relatively high intensity speckles and are therefore very suitable for speckle contrast imaging of blood flow.

[0077] As with these embodiments, the multiple first speckle images and the multiple images from the multiple images are the same image, and the multiple images can be acquired with a relatively simple system, requiring only a single light source and a single camera.

[0078] In one embodiment, the light of at least a second wavelength may be light of at least a second wavelength different from the first wavelength, preferably coherent light of a predetermined second wavelength, preferably in the green or blue portion of the electromagnetic spectrum, preferably between 380 and 590 nm, more preferably between 470 and 570 nm, and even more preferably between 520 and 560 nm. Blue or especially green light is absorbed much more strongly in blood vessels than in normal tissue, resulting in high contrast or visual differentiation. Thus, features of blood vessels, such as edges or corners, may be used to determine the registration parameters.

[0079] Since the first speckle images themselves are inherently noisy (as far as imaging of anatomical features is concerned), it may be preferable to use a second image based on a different wavelength to determine the alignment vector. In this way, the first speckle images may be acquired based on light selected to optimize speckle contrast signal, while the second images may be acquired based on light selected to optimize visual distinctiveness. Such a system is particularly advantageous for imaging tissues with relatively deep blood perfusion, such as skin. In such tissues, most of the green or blue light does not penetrate deep enough to interact with blood cells, resulting in a relatively noise-free image.

[0080] In one embodiment, the first wavelength is in the red portion of the electromagnetic spectrum, preferably 600-700 nm, more preferably 620-660 nm, or in the infrared portion of the electromagnetic spectrum, preferably 700-1200 nm. Red and infrared light are mostly reflected by red blood cells and are very suitable for speckle contrast imaging of blood flow. Infrared light has a greater penetration depth than red light. Red light may be easier to integrate into existing systems, for example using the red channel of an RGB camera to acquire a speckle image.

[0081] Preferably, the first wavelength is selected to be scattered or reflected by the fluid of interest, for example, red or near infrared light may be used for imaging blood in blood vessels. Preferably, the first wavelength may be selected based on the required penetration depth. Light with a relatively high penetration depth may allow light scattered by the fluid of interest to be detected with a sufficient signal-to-noise ratio even at some depth within the imaged tissue.

[0082] Preferably, the second wavelength is selected to provide an image with high visual differentiation resulting in consistent tissue surface features within the image. For example, green light may be used to image blood vessels within internal organs, as green light is typically absorbed much more strongly by blood than by tissue. The light at the second wavelength may be either coherent or non-coherent light. The multiple second images may also be based on multiple wavelengths, e.g., white light may be used. The light at the second wavelength may be generated, for example, by a second coherent light source. Alternatively, the light at the first wavelength and the light at the second wavelength may be generated by a single coherent light source configured to generate multiple wavelengths of coherent light.

[0083] In one embodiment, the sequence of a plurality of second images may be the sequence of a plurality of second speckle images, and the method may further include determining a plurality of second speckle contrast images based on the sequence of the plurality of second speckle images, and adjusting or correcting the plurality of first speckle contrast images based on a change in magnitude of a speckle contrast in the sequence of the plurality of second speckle contrast images.

[0084] Multi-spectral coherent correction (also called dual laser correction) may remove or reduce noise in the first speckle contrast image by adjusting the determined speckle contrast in the plurality of first speckle images based on changes in the determined speckle contrast in the one sequence of a plurality of second images. The adjustment may be based on a predetermined correlation between the speckle contrast of the plurality of first speckle contrast images and the speckle contrast of the plurality of second speckle contrast images.

[0085] Multispectral coherent correction can be advantageously combined with image registration using a second image based on the second wavelength, by using the second image for both multispectral coherent correction and image registration. In such an embodiment, the second wavelength preferably has a relatively small penetration depth. In this way, the multiple second images can include information related mainly to the surface of the target area. This is particularly true for tissues that are poorly perfused near the surface, such as skin, scar tissue, and some tumor types.

[0086] In one embodiment, the method may further include dividing each image in at least a portion of a sequence of the plurality of first speckle images or a sequence of a plurality of first speckle contrast images and each image in the plurality of images into a plurality of regions, preferably disjoint regions. Preferably, the regions in the images from the plurality of images correspond to regions in the associated first speckle image or first speckle contrast image. Determining registration parameters may include determining registration parameters for each region, and determining a sequence of registered plurality of first speckle images or a sequence of a plurality of first speckle contrast images may include registering each region of the first speckle image or first speckle contrast image based on a transformation based on a corresponding region in the image from the plurality of images.

[0087] The regions may be determined, for example, based on the geometry of the image, such as a grid of rectangular or triangular regions, or based on image properties, such as light intensity or groups of pixels that appear to belong to an anatomical structure.

[0088] In that way, local motion in parts of the images can be corrected, leading to a higher quality combined image. Local motion is typically caused by motion within the target, for example due to a person's movement, breathing, heartbeat, or muscle contractions, such as peristalsis in the lower abdomen. The regions can be as small as a single pixel. If a weighted average is used to combine two or more images, weights can be assigned to each region separately and / or to the entire image. Combining images can also be based on regions or done on an image-by-image basis.

[0089] In one embodiment, the target area may include a perfused organ, preferably perfused by a body fluid, more preferably perfused by blood and / or lymphatic fluid, and / or may include one or more blood vessels and / or lymphatic vessels. The method may further include calculating a perfusion intensity, preferably blood perfusion intensity or lymphatic perfusion intensity, based on the combined speckle image.

[0090] The method may further include post-processing the image, such as thresholding, pseudocoloring, superimposing it with another image, such as a white light image, and / or displaying the combined image or a derivative thereof.

[0091] In a second aspect, an embodiment may relate to a hardware module for an imaging device, preferably for a medical imaging device. The hardware module may comprise a first light source for exposing a target area to a coherent first light of a first wavelength, the target area preferably comprising biological tissue. The hardware module may further comprise an image sensor system having one or more image sensors for capturing at least one sequence of a plurality of images, the at least one sequence of a plurality of images comprising a plurality of first speckle images, the plurality of first speckle images being captured during exposure with the first light.The hardware module may further comprise a computer readable storage medium having computer readable program code embedded therein, and a processor, preferably a microprocessor, more preferably a graphics processing unit, coupled to the computer readable storage medium, wherein in response to executing the computer readable program code, the processor determines one or more registration parameters of an image registration algorithm to register the plurality of first speckle images with one another, wherein the registration parameters are based on a similarity measure of pixel values ​​of a plurality of pixel groups in a plurality of images, the images in the plurality of images being selected from the first laser speckle images or associated with the plurality of first speckle images, and the registration parameters preferably define one of a homography, a projective transformation, or an affine transformation. and determining a registered plurality of first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle images, or determining a plurality of first speckle contrast images based on the plurality of first speckle images, determining a registered plurality of speckle contrast images by registering the plurality of first speckle contrast images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle contrast images.

[0092] In one embodiment, the hardware module may include a second light source for illuminating the target area with light of at least a second wavelength different from the first wavelength, simultaneously or alternately with the first light source.The at least one image sensor may be configured to capture the one sequence of a plurality of second images, where the plurality of second images are captured during exposure with the second light.The plurality of images may be selected from the one sequence of a plurality of second images, where each of the plurality of second images is associated with a first speckle image.

[0093] The image sensor system may include a first image sensor for capturing the one sequence of a plurality of first images and a second image sensor for capturing the one sequence of a plurality of second images, or a single image sensor for capturing both the one sequence of a plurality of first images and the one sequence of a plurality of second images.

[0094] In one embodiment, the hardware module may further comprise a display for displaying the combined speckle image and / or a derivative thereof, preferably a perfusion intensity image. Alternatively or additionally, the hardware module may comprise a video output for outputting the combined speckle image and / or a derivative thereof.

[0095] The image sensor system may comprise a first image sensor for capturing an image at the first wavelength and a second image sensor for capturing an image at the at least second wavelength. The first image sensor and the second image sensor may be the same image sensor, different portions of a single image sensor, e.g., red and green channels from an RGB camera, or different image sensors. The module may further comprise optics for directing light from the first light source and the optional second light source to a target area and / or for directing light from the target area to the first and second image sensors.

[0096] The present disclosure further relates to a medical imaging device, preferably an endoscope, a laparoscope, a surgical robot, a handheld laser speckle contrast imaging device, or an open surgery laser speckle contrast imaging system, comprising such a hardware module.

[0097] In a further aspect, the present disclosure relates to a computation module for a laser speckle imaging system, the computation module comprising a computer readable storage medium having at least a portion of a program embedded therein, and a processor, preferably a microprocessor, more preferably a graphics processing unit, coupled to the computer readable storage medium, the processor configured to perform executable operations in response to executing the computer readable storage code, the executable operations being: receiving at least one sequence of a plurality of images, wherein the at least one sequence of a plurality of images includes a plurality of first speckle images captured during exposure of a target area to a coherent first light at a first wavelength, the target area including biological tissue; determining one or more registration parameters of an image registration algorithm for registering the plurality of first speckle images with one another, wherein the registration parameters are based on a similarity measure of pixel values ​​of a plurality of pixel groups in a plurality of images of the at least one sequence of a plurality of images, wherein the image in the plurality of images is selected from or related to the plurality of first speckle images, and the registration parameters preferably define one of a homography, a projective transformation, or an affine transformation; and and determining a registered plurality of first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle images; or determining a plurality of first speckle contrast images based on the plurality of first speckle images, determining a registered plurality of speckle contrast images by registering the plurality of first speckle contrast images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle contrast images.

[0098] Such a computation module can be added to existing or new medical imaging devices, such as laparoscopes or endoscopes, for example, to improve laser speckle contrast imaging, especially perfusion imaging. In one embodiment, the method steps described in the present disclosure can be performed by a processor in a device for coupling coherent light into an endoscopy system. Such a device can be connected between a light source and a video processor of an endoscopy system and an endoscope, such as a laparoscope, of the endoscopy system. The connection device can thereby add laser speckle imaging capabilities to the endoscopy system. Such a connection device is described in more detail in Dutch Patent Application No. NL2026240, which is incorporated by reference.

[0099] In one embodiment, the at least one sequence of a plurality of images comprises the one sequence of a plurality of second images, wherein the plurality of second images are captured during exposure with a second light, the second light has one or more second wavelengths, preferably the second light is a coherent light of a second wavelength, or the second light comprises a plurality of second wavelengths in the visible spectrum, and the exposure to the second light is alternated with the exposure to the first light or is simultaneous with the exposure to the first light.The plurality of images can be selected from the one sequence of a plurality of second images, wherein each of the plurality of second images is associated with a first speckle image.

[0100] The present disclosure also relates to a computer program or suite of computer programs comprising at least one software code portion, or a computer program product having stored thereon at least one software code portion, which, when executed on a computer system, is configured to perform any of the method steps described above.

[0101] The present disclosure may further relate to a non-transitory computer-readable storage medium having stored thereon at least one software code portion, which, when executed or processed by a computer, is configured to perform any of the method steps described above.

[0102] The present embodiments will be further explained with reference to the attached drawings, which show, in a schematic manner, embodiments according to the invention, it being understood that the invention is in no way restricted to these specific embodiments.

[0103] The following description of specific embodiment diagrams is merely exemplary in nature and is not intended to limit the present teachings, their application, or uses. [Brief description of the drawings]

[0104] [Figure 1A] FIG. 1A illustrates generally a system for motion-compensated laser speckle contrast imaging, according to one embodiment. [Figure 1B] FIG. 1B illustrates a flow diagram for laser speckle contrast imaging, according to one embodiment. [Figure 1C] FIG. 1C illustrates a flow diagram for laser speckle contrast imaging, according to one embodiment. [Figure 1D] FIG. 1D illustrates a flow diagram for laser speckle contrast imaging, according to one embodiment. [Figure 1E] FIG. 1E illustrates a flow diagram for laser speckle contrast imaging, according to one embodiment. [Figure 2A] FIG. 2A illustrates a raw speckle image. [Figure 2B] FIG. 2B illustrates a laser speckle contrast image based on the raw speckle image. [Figure 2C] FIG. 2C illustrates a perfusion image based on the laser speckle contrast image. [Figure 3A]FIG. 3A illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 3B] FIG. 3B illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 3C] FIG. 3C illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 3D] FIG. 3D illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 4A] FIG. 4A illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 4B] FIG. 4B illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 4C] FIG. 4C illustrates a flow diagram for motion compensated laser speckle contrast imaging, according to an embodiment. [Figure 5A] FIG. 5A illustrates a method for determining registration parameters, according to an embodiment. [Figure 5B] FIG. 5B illustrates a method for determining registration parameters, according to an embodiment. [Figure 5C] FIG. 5C illustrates a method for determining registration parameters, according to an embodiment. [Figure 5D] FIG. 5D illustrates a method for determining registration parameters, according to an embodiment. [Figure 5E] FIG. 5E illustrates a method for determining registration parameters, according to an embodiment. [Figure 5F] FIG. 5F illustrates a method for determining registration parameters, according to an embodiment. [Figure 6A]FIG. 6A illustrates a method for determining registration parameters, according to an embodiment. [Figure 6B] FIG. 6B illustrates a method for determining registration parameters, according to an embodiment. [Figure 6C] FIG. 6C illustrates a method for determining registration parameters, according to an embodiment. [Figure 6D] FIG. 6D illustrates a method for determining registration parameters, according to an embodiment. [Figure 7A] FIG. 7A illustrates a flow diagram for motion compensated laser speckle contrast imaging combining three or more raw speckle images, according to one embodiment. [Figure 7B] FIG. 7B illustrates a flow diagram for motion compensated laser speckle contrast imaging combining three or more raw speckle images, according to one embodiment. [Figure 8] FIG. 8 illustrates a flow diagram for calculating a corrected laser speckle contrast image, according to one embodiment. [Figure 9A] FIG. 9A illustrates a schematic diagram of a motion-compensated speckle contrast image determination based on a weighted average, according to one embodiment. [Figure 9B] FIG. 9B illustrates a schematic diagram of a motion-compensated speckle contrast image determination based on a weighted average, according to one embodiment. [Figure 10] FIG. 10 is a block diagram illustrating an example data processing system that may be used to implement the methods and software products described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0105] Laser speckle contrast images can be based on spatial contrast, temporal contrast, or a combination thereof. In general, using spatial contrast leads to high temporal resolution, but relatively low spatial resolution. In addition, individual images are subject to degradation due to, for example, motion or lighting artifacts, resulting in image quality that can vary from image to image. On the other hand, using temporal contrast involves relatively high spatial resolution and relatively low temporal resolution. However, the quality of the temporal contrast can be strongly affected by the motion of the target relative to the camera, which can lead to pixels being inaccurately combined. The mixed method can share some advantages and disadvantages of both methods.

[0106] In this disclosure, a speckle image may also be referred to as an unprocessed speckle image to better distinguish between (unprocessed) speckle images and speckle contrast images. Thus, the term "unprocessed speckle image" may refer to an image representing a speckle pattern with pixels having pixel values ​​representing light intensity. An unprocessed speckle image may be an unprocessed image or a (pre)processed image. The term "speckle contrast image" may be used to refer to a processed speckle image with pixels having pixel values ​​representing the magnitude of the speckle contrast, typically the relative standard deviation over a predetermined neighborhood of pixels.

[0107] FIG. 1A illustrates a schematic diagram of a system 100 for motion-compensated laser speckle contrast imaging according to one embodiment of the present invention. The system may comprise a first light source 104 for generating coherent light, e.g. laser light, of a first wavelength for illuminating a target area 102. The target is preferably a biological tissue, e.g. skin, intestine, or brain tissue. The first wavelength may be selected to interact with a body fluid, e.g. blood or lymph, that may move through the target. The first wavelength may be in the red or (near) infrared part of the electromagnetic spectrum, e.g. 600-700 nm, preferably 620-660 nm, or 700-1200 nm. The first wavelength may be selected based on the body fluid of interest and / or the tissue to be imaged. The first wavelength may also be selected based on the nature of the imaging sensor. Depending on the exposure time, size of the area imaged, speckle size, and image resolution, among others, various quantities of interest can be imaged, such as flow in individual large or small blood vessels, microvascular perfusion of the target area, etc.

[0108] In one embodiment, the system may further comprise a second light source 106 for generating at least a second wavelength of light for illuminating the target area 102, preferably including light in the green portion of the electromagnetic spectrum. The at least a second wavelength may be selected to include a wavelength that produces an image with high visual distinction, i.e., high contrast, of anatomical features. The second wavelength may be selected based on the tissue in the target area. In general, the light of the at least a second wavelength may be coherent or non-coherent light, and may be monochromatic, e.g., blue or green narrow band imaging light, or polychromatic, e.g., white light. In other embodiments, only the first light source is used. In the embodiment depicted in FIG. 1C, the light of the at least a second wavelength is monochromatic coherent light. The second wavelength may be generated, for example, by a second coherent light source. Alternatively, the light at the first wavelength and the light at the second wavelength may be generated by a single coherent light source configured to generate coherent light at multiple wavelengths.

[0109] The system may further include one or more image sensors 108 for capturing an image associated with the light of the first wavelength and, if applicable, an image associated with the light of the at least second wavelength, which interacts with the target in the target area. In different embodiments, the system may include multiple cameras, such as a first camera for acquiring a first raw speckle image associated with the first wavelength and a second camera for acquiring a corrected image associated with the second wavelength. The system may further include additional optics, such as optical fibers, lenses, or beam splitters, for directing light from the one or more light sources to and from the target area to the one or more image sensors.

[0110] When the target is illuminated with coherent light, a laser speckle pattern may be formed through self-interference. The image may be received and processed by a processing unit 110. Examples are described in more detail with reference to Figures 1B-1D. The processing unit may output the processed image, for example to a display or computer 112, essentially in real time. The processing unit may be a separate unit or may be part of the computer. The processed image may be displayed by the display or computer.

[0111] In one embodiment, an endoscope or laparoscope can be used to direct light to the target area and acquire images. The one or more light sources, the one or more image sensors, the processing unit, and the display can be part of an endoscope system, for example.

[0112] 1B-1E illustrate a flow diagram for motion-compensated laser speckle contrast imaging according to one embodiment. Alternative aspects mentioned with respect to one of these embodiments can also be applied to the other embodiments, unless such combination leads to a contradictory description.

[0113] FIG. 1B illustrates a flow diagram for motion-compensated laser speckle contrast imaging according to one embodiment. In this embodiment, only the first wavelength of coherent light generated by the first light source 104 is used, which can be, for example, green light, or preferably red light. The one or more image sensors 108 are single image sensors, typically monochromatic image sensors optimized or at least suitable for the wavelength used. As pointed out above and shown in the examples of FIG. 1C and FIG. 1D below, other embodiments may use different configurations.

[0114] In a first step 120, a sequence of a plurality of raw speckle images is obtained, e.g., captured or received from an external source. In this example, these speckle images are also used as correction images. A first speckle contrast image may be calculated (126) based on each first raw speckle image. The speckle contrast may be determined in any suitable manner, e.g., by convolution with a predefined convolution kernel or by determining the relative standard deviation of pixel value intensities within a sliding window. Since the speckle contrast is correlated with perfusion, a perfusion unit weight may be determined based on the speckle contrast values ​​of the first speckle contrast images.

[0115] In a single wavelength embodiment, either the raw speckle image or the speckle contrast image can be used as a correction image. In an optional step 129, the correction image can be transformed. Based on the optionally transformed correction images, registration parameters can be calculated. For example, in each correction image, the position of a predetermined object feature can be determined. Based on the position of the predetermined object feature in two or more correction images, an alignment vector that identifies the movement of the target area can be determined, for example, using an optical flow algorithm. A (coarse) optical flow algorithm can be used to determine the alignment vector based on selected features. Other embodiments can use, for example, a dense optical flow algorithm, in which case no features need to be determined.

[0116] Based on the alignment vector, a transformation for registering the images in the second sequence of images may be determined. Preferably, the transformation is selected from a class of homographies, from a class of projective transformations, or from a class of affine transformations. Optionally, optical flow weights may be determined based on the alignment vector or parameters defining the transformation.

[0117] Alternatively, the registration parameters can be calculated in any other suitable manner, for example using template matching, phase matching, matching in a log-polar coordinate system, etc. Various examples of suitable methods for determining the registration parameters will be described in further detail below with reference to Figures 5A-5D and 6.

[0118] The determined registration parameters may then be used to register (132), or geometrically align, the speckle contrast images with one another, resulting in a plurality of registered first speckle contrast images. In an alternative embodiment, the raw speckle images may be registered prior to calculating the speckle contrast. However, such an embodiment is less preferred, since image registration may affect pixel values ​​and therefore contrast.

[0119] The registered first speckle contrast images can then be combined (134), for example using a temporal filter. The temporal filter can include averaging the first speckle contrast images. The averaging can be a weighted average, where the weights are based on optical flow weights and / or based on perfusion unit weights. In an alternative embodiment, the registered raw speckle images can be combined using the temporal filter, and a speckle contrast image can be determined based on the combined raw speckle images. This results in a motion-compensated speckle contrast image (136).

[0120] The combined first raw speckle image can then be post-processed, e.g., converted to a perfusion value, thresholded to show areas of high or low perfusion, superimposed on a white light image of the target area, etc. The post-processing can be performed, for example, by the processing unit 110 or the computer 112. FIG. 1C illustrates a flow diagram for motion-compensated laser speckle contrast imaging according to one embodiment of the present invention. In the illustrated embodiment, the light of the first wavelength generated by the first light source 104 is red light, and the light of the at least second wavelength generated by the second light source 106 is coherent green light. The one or more image sensors 108 are single sensors with red, green, and blue channels. The first and second wavelengths are selected to minimize crosstalk. The signal in the red channel caused by the green light is significantly smaller than the signal caused by the red light, and the signal in the green channel caused by the red light is significantly smaller than the signal caused by the green light. As noted above, other embodiments may use different configurations.

[0121] In a first step 140, a sequence of RGB images is received. A first sequence of a plurality of first raw speckle images may be extracted from a red channel of the sequence of RGB images (142), and a second sequence of a plurality of corrected images may be extracted from a green channel of the RGB images (144). The sequence of a plurality of corrected images may be a second sequence of a plurality of second raw speckle images. Each first raw speckle image may be associated with a corrected image extracted from the same RGB image. In other embodiments, the plurality of first raw speckle images and the plurality of corrected images may be acquired by different cameras, by different sensors of a multi-sensor camera (e.g., a 3CCD camera), or by other (color) channels of a single camera (e.g., a YUV camera), or the images may be acquired alternately by a single monochromatic camera when the target is alternately illuminated with the first wavelength of light and the at least second wavelength of light.

[0122] A first speckle contrast image may be calculated (146) based on each first raw speckle image. Optionally, a second speckle contrast image may be calculated (148) based on each second raw speckle image. Speckle contrast may be determined in any suitable manner, such as by convolution with a predefined convolution kernel or by determining the relative standard deviation of pixel value intensities within a sliding window. Since the speckle contrast is correlated with perfusion, a perfusion unit weight may be determined based on the speckle contrast values ​​of the first speckle contrast images. The second speckle contrast images may optionally be used to correct (152) the first speckle contrast images, as described in more detail with reference to FIG. 8. The corrected speckle contrast values ​​may then be used to determine the perfusion unit weights.

[0123] Based on the corrected image (i.e., in this embodiment, the second unprocessed speckle image), registration parameters may be calculated (150). For example, the position of a predetermined object feature may be determined. Based on the position of the predetermined object feature in the two or more corrected images, an alignment vector that identifies the movement of the target area may be determined, for example, using an optical flow algorithm. A (coarse) optical flow algorithm may be used to determine the alignment vector based on selected features. Other embodiments may use, for example, a fine optical flow algorithm, in which case no features need to be determined. Alternatively, the registration parameters may be calculated in any other suitable manner, for example, using template matching, phase matching, matching in a log-polar coordinate system, etc. Some of these examples may include determining a transformed image based on the corrected images. Various examples of suitable methods for determining registration parameters will be described in more detail below with reference to Figures 5A-5D and 6.

[0124] A transformation for registering the images in the second sequence of images may be determined based on the registration parameters and the alignment vector. Preferably, the transformation is selected from a class of homographies, from a class of projective transformations, or from a class of affine transformations. Optionally, optical flow weights may be determined based on the alignment vector or parameters defining the transformation.

[0125] The determined registration parameters may then be used to register (154) or geometrically position the first plurality of speckle contrast images associated with the correction plurality of images.

[0126] The registered first speckle contrast images may then be combined (156), for example using a temporal filter, which may include averaging the first speckle contrast images. The averaging may be a weighted average, where the weights are based on optical flow weights and / or based on perfusion unit weights.

[0127] In a final step 158, the combined first raw speckle image may be post-processed, e.g., converted to a perfusion value, thresholded to show areas of high or low perfusion, superimposed on a white light image of the target area, etc. The post-processing may be performed, for example, by the processing unit 110 or the computer 112.

[0128] FIG. 1D illustrates a flow diagram for motion-compensated laser speckle contrast imaging according to one embodiment. In the illustrated embodiment, the light of the first wavelength generated by the first light source 104 is infrared light, but other colors are possible, such as red, green, or blue. The light of the at least second wavelength generated by the second light source 106 is (non-coherent) white light. The advantage of using infrared light and white light is that both lights can be used simultaneously without significantly affecting each other.

[0129] As shown, the one or more image sensors 108 are two image sensors in two cameras: an infrared camera and a color camera. This can be practical when laser speckle imaging is added to a system that already has a color camera, for example in an open surgery situation. In addition, having a dedicated image sensor can allow hardware and / or equipment parameters, such as exposure time, to be optimized separately. As pointed out above, other embodiments can use different configurations. For example, in some embodiments, a single camera can be used to capture both infrared and (white light) color images. The advantage of using a single camera is that the infrared and color images are automatically aligned and correlated with each other.

[0130] In a first step 160, a first sequence of a plurality of first images is captured by the infrared camera, which may be stored as a sequence of speckle images (162). The first sequence may be stored by the image processor as a sequence of a plurality of raw speckle images. In a second step 161, a second sequence of a plurality of second images is captured by the color camera. The second sequence may be stored as a sequence of a plurality of corrected images (164). Each raw speckle image may be associated with one or more corrected images. Preferably, each raw speckle image is associated with at least a corrected image that was captured proximately and preferably simultaneously with the raw speckle image. Preferably, the frame rates of the first and second cameras are selected to allow a straightforward association, for example by selecting one frame rate as an integer multiple of the other frame rate.

[0131] A speckle contrast image may be calculated 166 based on each raw speckle image in the sequence of speckle images. The speckle contrast may be determined in any suitable manner, such as by convolution with a predefined convolution kernel or by determining the relative standard deviation of pixel value intensities within a sliding window. Because the speckle contrast is correlated with perfusion, a perfusion unit weight may be determined based on the speckle contrast values ​​of the plurality of first speckle contrast images.

[0132] Based on the corrected image (i.e., in this embodiment, the white light image), registration parameters may be calculated (170). For example, the position of a predetermined object feature may be determined. Based on the position of the predetermined object feature in the two or more corrected images, an alignment vector that identifies the movement of the target area may be determined, for example, using an optical flow algorithm. A (coarse) optical flow algorithm may be used to determine the alignment vector based on selected features. Other embodiments may use, for example, a fine optical flow algorithm, in which case no features need to be determined. Alternatively, the registration parameters may be calculated in any other suitable manner, for example, using template matching, phase matching, matching in a log-polar coordinate system, etc. Some of these examples may include determining a transformed image based on the corrected images. Various examples of suitable methods for determining registration parameters will be described in more detail below with reference to Figures 5A-5D and 6.

[0133] Based on the alignment vector, a transformation for registering images in the sequence of corrected images may be determined. Preferably, the transformation is selected from a class of homographies, from a class of projective transformations, or from a class of affine transformations. Optionally, optical flow weights may be determined based on the alignment vector or parameters defining the transformation.

[0134] The determined transformation can then be used to register (174), or geometrically align, the plurality of first speckle contrast images associated with the plurality of correction images. In this embodiment, since two cameras are used, the two cameras do not share a field of view and a reference frame. As a result, registering the plurality of speckle contrast images based on the registration parameters determined using the plurality of correction images can include applying a transformation to the registration parameters to account for this change in the reference frame. If the two cameras are located at fixed positions relative to each other, this transformation can be predetermined. Otherwise, the transformation can be determined, for example, based on image processing of a calibration image or a marker. The marker can be either natural or artificial.

[0135] The registered first speckle contrast images may then be combined (176), for example using a temporal filter, which may include averaging the first speckle contrast images. The averaging may be a weighted average, where the weights are based on optical flow weights and / or based on perfusion unit weights.

[0136] In a final step 178, the combined first raw speckle image may be post-processed, e.g., converted to a perfusion value, thresholded to show areas of high or low perfusion, superimposed on a white light image of the target area, etc. The post-processing may be performed, for example, by the processing unit 110 or the computer 112.

[0137] FIG. 1E illustrates a flow diagram of motion compensated laser speckle contrast imaging according to one embodiment. Step 180 includes acquiring an image sequence, e.g., as described above with reference to FIG. 1B. Moreover, in step 182, artifacts are detected in the acquired images. In this context, "artifact" should be understood broadly and may refer to any pixel value that may adversely affect tissue perfusion calculations. Artifacts may be due to several causes. For example, specular artifacts may occur when a pixel or group of pixels is fully saturated (especially in an endoscopy / laparoscopy setup) and therefore no contrast is calculated. Artifacts may also be due to, e.g., defects in pixel elements in the camera, dirt on the lens that collects the light, etc.

[0138] The artifacts may be identified by identifying deviant input pixel values ​​and / or deviant speckle contrast values. For example, pixel values ​​that exceed a predefined upper threshold, absolute or relative, or pixel values ​​that fall below a predefined lower threshold, absolute or relative, may be marked as deviant pixel values. For example, fully saturated pixels (i.e., pixels having a maximum pixel value) may be excluded, but it may be beneficial to also exclude nearly saturated pixels, e.g., pixels having a pixel value that is 99% or more of the maximum pixel value.

[0139] The relative threshold may be based, for example, on an analysis of the pixel's environment, e.g., the mean or median or other statistical representation. For example, pixels that deviate from the regional median by more than a predetermined amount, or from the regional mean by more than a predetermined number of standard deviations, may be identified as artifacts. Alternatively, a (Gaussian) blur may be applied to the image to determine a relative background value to which the relative threshold may be applied. The region for determining the outliers is typically selected to be substantially larger than the window for calculating the spatial speckle contrast.

[0140] Moreover, a border region around these pixels may be included in the mask, for example, by growing a region with the identified pixels. The size of the border region may be based on the size of the region for which the spatial contrast is calculated. For example, if the border region is at least equal in size to the radius of the window for spatial contrast, the spatial contrast may be calculated without considering the mask, and all speckle contrast values ​​based on the identified pixels will be masked.

[0141] Additionally or alternatively, pixel contrast values ​​below or above absolute or relative individual lower or upper thresholds may be identified. Certain artifacts may result in very low or very high speckle contrast values. Furthermore, the size and / or shape of regions with very high or low calculated speckle contrast values ​​may be used to identify artifacts.

[0142] In this context, image artifacts may also refer to pixels that do not represent living tissue. For example, an image may include pixels that represent surgical instruments (especially in open surgery), clamps, sutures, gauze, etc. Because these objects are not living tissue, they are (generally) not perfused. Therefore, in addition to or instead of the speckle image, a speckle contrast image may be used to identify these types of artifacts. In embodiments in which the at least one sequence of images also includes other images (e.g., white light images) than the speckle image, pixels that do not represent living tissue may additionally or alternatively be identified in the other images.

[0143] Image recognition algorithms can also be used to identify artifacts, which is particularly useful for artifacts that have a known shape or other known or learnable visual characteristics.

[0144] In some embodiments, artifact detection is limited to one or more regions of interest within the image, in which case subsequent steps may also be limited to such regions of interest.

[0145] In step 184, for each image in the sequence of input images, a mask is created based on the detected artifacts for that image. The mask associates confidence scores with one or more pixels of the corresponding input image. This is typically a binary mask, indicating which pixels are considered to be unreliable. However, multi-values ​​(i.e., non-binary) may also be used. For example, pixels identified as unreliable may be surrounded by a region of pixels of increasing confidence.

[0146] In some cases, multiple masks may be determined for each input image, for example one mask representing deviant pixel values ​​and one mask representing non-tissue objects. The use of multiple masks allows for different downstream processing of the masked pixels. Alternatively, a multivalued mask may be used, with different values ​​indicating different sources of uncertainty or error.

[0147] In step 185, the registration parameters are calculated as described above. Calculating image registration parameters based on the masked image can be beneficial to prevent fitting to image artifacts. For example, some feature-based image registration algorithms may attempt to register overexposed spots or specular artifacts. Using a mask that removes these artifacts may force the registration algorithm to register to anatomy features instead.

[0148] In step 186, the speckle contrast is calculated. In some cases, the speckle contrast calculation takes into account the calculated mask. Depending on the implementation, if the input for the speckle contrast calculation contains one or more masked pixels, these masked pixels are ignored, or the calculation is skipped, or an error value is assigned. The treatment of masked pixels may depend on the absolute or relative number of masked pixels in the input.

[0149] In some cases, further unreliable pixels may be determined based on the calculated speckle contrast, for example based on deviant speckle contrast values.

[0150] Therefore, in optional step 188, the mask is updated (or a new mask is created) based on the calculated speckle contrast.

[0151] In principle, it is possible to apply the mask directly to the speckle or speckle contrast image, i.e., to replace pixel values ​​in the speckle (contrast) image with mask values. However, it is often useful to store the mask in a separate image (or separate image layer). However, care must be taken in that case to ensure that the mask matches the image. Therefore, in step 192, the mask is transformed using the same transformation algorithm and parameters used for image registration of the speckle contrast image in step 190.

[0152] In step 194, a time filter is applied to the speckle contrast images, as described above. For example, a pixel-by-pixel weighted average may be calculated, where the weight of each pixel may depend, for example, on the image registration parameters and / or the local or global speckle contrast in the images. Additionally (or alternatively), the weight may depend on the mask. In particular, a masked pixel may have a weight equal to zero. If the mask defines a multi-valued confidence score, the weight may depend on the confidence score, with pixels having a higher confidence score having a higher weight than pixels having a lower confidence score. Thus, a motion-compensated speckle contrast image sequence may be determined based on the speckle contrast image sequence and the associated mask.

[0153] In optional step 198, a combined speckle contrast image mask is determined. The combined speckle contrast image mask may indicate whether up to a predetermined percentage of the input pixels are masked. For example, the weighted average is calculated only if less than a predetermined percentage of the input pixels are masked, and if more than a predetermined percentage of the input pixels are masked, the pixels are marked as having an invalid pixel value.

[0154] Alternatively, the combined speckle contrast image mask may indicate a confidence score of the pixels in the combined speckle contrast image mask. The confidence score may depend on the proportion of masked pixels in the weighted average and / or on the confidence scores of the individual masks. The mask may also be based on other weighting factors that go into the temporal filter, such as local or global registration parameters and / or local or global speckle contrast values.

[0155] The speckle contrast or perfusion values ​​are typically shown as an overlay on the captured image. There are various options for handling pixels in the overlay for which a valid perfusion value could not be calculated. For example, pixels marked as having an invalid pixel value may be removed or rendered as transparent, assigned an error value, or assigned a value based on an interpolation of the surrounding pixel values. Two or more of these options may be combined, for example, based on the cause of the invalid pixel value and / or based on the size of the region of connected invalid pixel values. For example, invalid pixel regions due to the presence of a non-tissue object (e.g., a surgical instrument) may be shown as transparent so that the instrument or other non-tissue object can be clearly seen. As a further example, small invalid pixel regions may be filled using interpolation to provide a clean image with the most likely correct values, while large invalid pixel regions may be assigned an error value to indicate that valid perfusion data was not acquired.

[0156] If a confidence score is assigned to each pixel, or at least to each pixel in a relevant region, the confidence scores may be rendered with varying transparency (e.g., using an alpha channel), with more reliable pixels having lower transparency (higher opacity) and less reliable pixels having higher transparency (lower opacity).

[0157] In one embodiment, the method steps described in this disclosure may be performed by a processor in a device for coupling coherent light into an endoscopic system. Such a device may be connected between a light source and a video processor of an endoscopic system and an endoscope, such as a laparoscope, of the endoscopic system. The coupling device may thereby add laser speckle imaging capabilities to the endoscopic system. Such a coupling device is described in more detail in Dutch Patent Application No. NL2026240, which is incorporated by reference.

[0158] In alternative embodiments, the method steps described in this disclosure may be applied in an open surgical setting, optionally in combination with an existing imaging system.

[0159] FIG. 2A illustrates raw speckle images and laser speckle contrast images based on the raw speckle images of targets with low and high perfusion. Images 202 and 204 are raw speckle images of the tip of a human finger, including the nail bed. When image 202 was obtained, blood flow through the finger was restricted, resulting in low blood perfusion (artificial ischemia) of the finger. When image 204 was obtained, blood flow was not restricted, resulting in much higher blood perfusion compared to the previous situation. It is difficult for a human observer to see the difference in speckle patterns associated with the difference in perfusion. A magnified portion 210 of image 204 is also shown, showing the speckle structure in more detail.

[0160] Images 206 and 208 are speckle contrast images based on images 202 and 204, respectively. Light colors represent low contrast and therefore high perfusion, while dark colors represent high contrast and therefore low perfusion. In these images, the differences in perfusion are immediately apparent, especially in the nail bed where blood flow occurs relatively close to the surface.

[0161] FIG. 2B illustrates a series of laser speckle contrast images of a low perfusion target exhibiting motion, before and after motion correction, according to one embodiment. 1~5 is a speckle contrast image of the tip of a human finger, including the nail bed. Image 220 2~4 During the acquisition of the image 220, the finger moved, resulting in a loss of contrast due to the finger movement. If the user is interested in blood flow, the image 220 is represented by a bright color. 2~4 The low speckle contrast in image 222 can be considered a motion artifact. 1~5 Each has 220 statues. 1~5 1 is based on the same raw speckle image as in , but corrected by a motion correction algorithm according to one embodiment.

[0162] FIG. 2C illustrates speckle contrast images from a series of speckle contrast images and a graph representing perfusion levels based on the series of speckle contrast images. Graph 230 depicts perfusion measurements of the nail bed, showing a first curve 232 representing perfusion determined based on uncorrected measurements, and a second curve 234 representing perfusion determined based on measurements corrected using the motion correction algorithm described in this disclosure. Around the 10 second time mark, blood flow to the finger is artificially restricted, and approximately 30 seconds later, the restriction is lifted. In particular, in the time range of 23-38 seconds, the uncorrected perfusion measurements show some motion artifacts, where the perfusion appears to rise sharply and then fall again.

[0163] The figure further shows the results before processing with a single wavelength based motion correction and compensation algorithm (image 236 1~3 ) and after (Image 238 1~3) show three exemplary speckle contrast images. In these images, light colors represent low speckle contrast and therefore high perfusion, while dark colors represent high speckle contrast and therefore low perfusion. The three uncorrected images look roughly the same, making it difficult for the user to recognize times (or regions in other applications) with low or high perfusion. In contrast, the motion-corrected images show a clear difference between the image acquired during the blood flow restriction (middle) and the other images where blood flow is not restricted, allowing the user to select times or locations with high perfusion.

[0164] Additionally, in the motion compensated image, anatomical structures such as the edges of the fingers and the nail bed can be more easily recognized, whereas in the uncorrected image, details can be difficult to recognize due to its grainy nature, making the image easier to interpret by a user.

[0165] FIG. 3A illustrates a flow diagram for motion-compensated laser speckle contrast imaging according to one embodiment. In a first step 302, a first raw speckle image can be obtained at a first time t=t1. A light source can illuminate a target area with coherent light of a predetermined wavelength, where an image sensor can capture a first raw speckle image based on the predetermined wavelength. Based on the first raw speckle image, a first laser speckle contrast image can be calculated (304). For example, the laser speckle contrast image can be determined by determining the relative standard deviation of pixel values ​​in a sliding window, for example, a 3×3 window, a 5×5 window, or a 7×7 window. In general, a (2n+1)×(2n+1) window can be selected for a natural number n depending on the speckle size. Alternatively, convolution with a kernel can be used, and the size of the kernel can be selected based on the speckle size. The relative standard deviation may be determined by calculating the standard deviation of the pixel intensity values ​​in an area divided by the average pixel intensity value in that area. Alternatively, the laser speckle contrast value may be determined in any other suitable manner.

[0166] Steps 306 and 308 are similar to steps 302 and 304, respectively, and are performed at a second time instance t=t2. Thus, the second raw speckle image may be acquired (306) at a second time instance t=t2. The light source may illuminate the target area with coherent light of a predetermined wavelength, and the image sensor may capture a second raw speckle image based on the predetermined wavelength. Based on the second raw speckle image, a second laser speckle contrast image may be calculated (308).

[0167] In a next step 310, the processor may determine registration parameters based on the first speckle image and the second speckle image, based on images associated with the first speckle image and the second speckle image, and / or based on a transformation of the first speckle image and the second speckle image. The registration parameters describe a geometric relationship between the first speckle image and the second speckle image that allows the first speckle image and the second speckle image to be aligned with each other. For example, the registration parameters may include an alignment vector that describes a displacement of one or more pixels in real space or transformed space.

[0168] In a next step 312, the processor may determine an alignment transformation, such as an affine transformation or a more general homography, to register or align the first and second speckle images with each other based on the registration parameters. Based on the alignment transformation, the processor may register or align the first and second laser speckle contrast images with each other by transforming the first and / or the second laser speckle contrast images (314). Typically, an older image may be transformed to be registered or aligned with a newer image.

[0169] The processor may then calculate 316 a combined, e.g., averaged, laser speckle contrast image based on the registered first and second laser speckle contrast images. Calculating the combined image may include calculating a weighted average, where the weights are preferably based on the normalized amount of speckle contrast, the relative change in speckle contrast, and / or the determined registration parameters. Calculating the combined image may further include applying one or more filters, e.g., a median filter or an outlier filter.

[0170] In other embodiments, these steps may be performed in a different order. For example, a laser speckle contrast image may be calculated after the raw speckle images are registered. In this way, a temporal or spatiotemporal speckle contrast image may be calculated. However, if the alignment transformation is more general than a translation (e.g., includes rotation, scaling, and / or shear), the alignment transformation may distort the speckle pattern and thus introduce a source of noise. In some embodiments, a single laser contrast raw speckle image may be calculated based on the combined, e.g., averaged, raw speckle images. In this case, the images are preferably registered with sub-pixel accuracy.

[0171] 3B illustrates a flow diagram of motion-compensated laser speckle contrast imaging according to one embodiment. In a first step 322, a first raw speckle image may be obtained at a first time instance t=t1, and a first laser speckle contrast image may be calculated 324 based on the first raw speckle image.

[0172] In the next step 325, a processor may determine a first plurality of first features in the first raw speckle image. Alternatively, the first plurality of first features may be determined in the first speckle contrast image. Preferably, the image with the most clearly defined anatomical features is used, and whether the anatomical structures have a higher visual distinction in the raw speckle image or the speckle contrast image may depend on the imaging parameters, such as wavelength and exposure time. In the remainder of the description of FIG. 3A, the term "speckle image" may refer to either a raw speckle image or a speckle contrast image. The steps associated with feature detection are described in more detail below with reference to FIGS. 5A and 6A-6D.

[0173] Steps 326-329 are similar to steps 322-325, respectively, and are performed at a second time t=t2. Thus, a second raw speckle image may be acquired (326) at a second time t=t2. The light source may illuminate a target area with coherent light at the predetermined wavelength, where the image sensor may capture a second raw speckle image based on the predetermined wavelength. A second laser speckle contrast image may be calculated (328) based on the second raw speckle image.

[0174] In the next step 329, the processor may determine a second plurality of second features in the second speckle image. At least some of the second plurality of second features should correspond to at least some of the first plurality of first features. Typically, the second speckle image will be similar to the first speckle image in typical applications because the target area does not change significantly between t=t1 and t=t2. Therefore, when a deterministic algorithm is used to detect features, most of the features detected in the second speckle image will generally correspond to the features detected in the first speckle image in the sense that the features detected in both images represent the same or substantially the same anatomical features in the imaged target. Thus, a plurality of second features may be associated with a plurality of first features.

[0175] In a next step 330, the processor may determine a plurality of alignment vectors based on the plurality of first features and the plurality of corresponding second features, where the alignment vectors describe the displacement of features relative to an image. For example, the processor may determine feature pairs consisting of one first feature and one second feature, determine a first position of the first feature relative to the first speckle image, determine a second position of the second feature relative to the second speckle image, and determine the difference between the first and second positions. Typically, corresponding feature pairs may be pairs of a first feature and an associated second feature that represent the same anatomical feature.

[0176] In a next step 332, the processor may determine a transformation, such as an affine transformation or a more general homography, based on the multiple displacement vectors to register corresponding features with each other. The transformation may be found, for example, by selecting a transformation from a class of transformations that minimizes the distance between pairs of corresponding features. Based on the transformation, the processor may register or align the first and second laser speckle contrast images with each other by transforming the first and / or second laser speckle contrast images (334). Typically, an older image may be transformed so that it is registered with a newer image.

[0177] In other embodiments, steps 325 and 329 may be omitted, and an alignment vector may be determined based on the plurality of first speckle images and the plurality of second speckle images. For example, a dense optical flow algorithm, such as a Pyramid Lucas-Kanade algorithm or a Farneback algorithm, may be used to determine an alignment vector. Such algorithms typically perform a convolution of a pixel neighborhood from the plurality of first speckle images with a portion or all of the plurality of second speckle images, thereby matching a neighborhood corresponding to each pixel in the plurality of first speckle images with a neighborhood in the plurality of second speckle images. Such a method may also include determining a polynomial expansion to model pixel values ​​in pixel neighborhoods in the plurality of first speckle images and the plurality of second speckle images, comparing the expansion coefficients, and determining an alignment vector based on the comparison.

[0178] In that way, an alignment vector can be determined, for example, for individual pixels or groups of pixels, based on pixel values ​​within groups of pixels within the multiple speckle images.

[0179] In some embodiments, step 330 may be omitted, where the transformation may be determined based on pixel values ​​of pixel groups in the plurality of first speckle images and pixel values ​​of associated pixel groups in the plurality of second speckle images, for example, using a trained neural network that receives a first image and a second image as input and provides as output a transformation for registering the first image with the second image, or alternatively, for registering the second image with the first image.

[0180] The processor may then calculate a combined, e.g., averaged, laser speckle contrast image based on the registered first and second laser speckle contrast images (336). Calculating the combined image may include calculating a weighted average, where the weights are preferably based on the normalized speckle contrast amount, the relative change in speckle contrast, the determined alignment vector, and / or parameters associated with the determined transformation. Calculating the combined image may further include applying one or more filters, e.g., a median filter or an outlier filter.

[0181] FIG. 3C illustrates a flow diagram for laser speckle contrast imaging according to one embodiment of the present invention. In a first step 342, a first raw speckle image may be obtained at a first time t=t1. A first light source may illuminate a target area with coherent light of a first wavelength, where a first image sensor may capture a first raw speckle image based on the first wavelength. Based on the first raw speckle image, a first laser speckle contrast image may be calculated (344). The laser speckle contrast image may be determined as described above with reference to step 304.

[0182] In a next step 343, a first corrected image associated with the first speckle image may be acquired at a first time t=t1. The first corrected image may comprise pixels having pixel values ​​and pixel coordinates, the pixel coordinates identifying the location of the pixel in the image. The first corrected image is preferably acquired simultaneously with the first unprocessed speckle image, although in alternative embodiments the first corrected image may be acquired, for example, before or after the first unprocessed speckle image.

[0183] A second light source may illuminate the target area with light of at least a second wavelength different from the first wavelength, and a second image sensor may capture a first corrected image based on the at least second wavelength. The second light source may use coherent light or non-coherent light. The second light source may generate monochromatic light or polychromatic light, e.g., white light. The second image sensor may be the same sensor as the first image sensor, or a different sensor.

[0184] In the embodiment described above with reference to FIG. 3A, the second wavelength is the same as the first wavelength, and the first corrected image is the same image as the first unprocessed speckle image.

[0185] In a next step 345, a processor may determine a first plurality of first features in the first corrected image. The steps involved in feature detection are described in more detail with reference to Figures 5-6.

[0186] Steps 346-349 are similar to steps 342-345, respectively, and are performed at a second time t=t2. Thus, a second raw speckle image may be obtained (346) at a second time t=t2. The first light source may illuminate a target area with coherent light at the first wavelength, and the first image sensor may capture a second raw speckle image based on the first wavelength. A second laser speckle contrast image may be calculated (348) based on the second raw speckle image.

[0187] In a next step 345, a second corrected image associated with the second raw speckle image may be obtained at the second time t=t2. The second light source may illuminate the target area with light of the at least a second wavelength, and the second image sensor may capture a second corrected image based on the at least a second wavelength.

[0188] In a next step 349, the processor may determine a second plurality of second features in the second corrected image. At least some of the second plurality of second features should correspond to at least some of the first plurality of first features. Typically, the second corrected image will be similar to the first corrected image in typical applications, since the target area does not change significantly between t=t1 and t=t2. Therefore, when a deterministic algorithm is used to detect features, most of the features detected in the second corrected image will generally correspond to features detected in the first corrected image, in the sense that the features detected in both images represent the same or substantially the same anatomical features in the imaged target. Thus, a plurality of second features may be associated with a plurality of first features.

[0189] In a next step 350, the processor may determine a number of alignment vectors based on the first feature and the corresponding second feature, where the alignment vector describes the displacement of the feature relative to an image. For example, the processor may determine feature pairs consisting of one first feature and one second feature, determine a first position of the first feature relative to the first corrected image, determine a second position of the second feature relative to the second corrected image, and determine the difference between the first and second positions. Typically, corresponding feature pairs may be pairs of a first feature and an associated second feature that represent the same anatomical feature.

[0190] In a next step 352, the processor may determine a transformation, such as an affine transformation or a more general homography, based on the alignment vectors to register corresponding features to each other. The transformation may be found, for example, by selecting a transformation from a class of transformations that minimizes the distance between pairs of corresponding features. Based on the transformation, the processor may register or align the first and second laser speckle contrast images to each other by transforming the first and / or second laser speckle contrast images (354). Typically, an older image may be transformed so that it is registered with a newer image. In embodiments using two or more image sensors, the registration parameters determined based on the correction images may be adjusted to account for differences between the fields of view of the two or more image sensors.

[0191] In other embodiments, steps 345 and 349 may be omitted, and an alignment vector may be determined based on the first corrected image and the second corrected image. For example, a dense optical flow algorithm, such as a Pyramid Lucas-Kanade algorithm or a Farneback algorithm, may be used to determine a displacement vector. Such algorithms typically convolve a pixel neighborhood from the first corrected image with a portion or all of the second corrected image, thereby matching a neighborhood corresponding to each pixel in the first corrected image with a neighborhood in the second corrected image. Such a method may also include determining a polynomial expansion to model pixel values ​​within pixel neighborhoods in the first corrected image and the second corrected image, comparing the expansion coefficients, and determining an alignment vector based on the comparison.

[0192] In that way, an alignment vector may be determined, for example for an individual pixel or group of pixels, based on pixel values ​​within a group of pixels in the first corrected image and an associated group of pixels in the second corrected image.

[0193] In some embodiments, step 350 may be omitted and the transformation may be determined based on pixel values ​​of a group of pixels in the first corrected image and pixel values ​​of an associated group of pixels in the second corrected image, for example, using a trained neural network that receives as input a first image and a second image and provides as output a transformation for registering the first image with the second image, or alternatively, for registering the second image with the first image.

[0194] The processor may then calculate (356) a combined, e.g., averaged, laser speckle contrast image based on the registered first laser speckle contrast image and the second laser speckle contrast image, as described above with reference to step 316.

[0195] In other embodiments, these steps can be performed in a different order. For example, the laser speckle contrast image can be calculated after the raw speckle images are registered. In this way, a temporal or spatiotemporal speckle contrast image can be calculated. However, if the transformation is more general than translation and rotation (e.g., includes scaling or shearing), the transformation can distort the speckle pattern, thereby introducing a source of noise. This is especially true for embodiments that use two or more cameras. In some embodiments, a single laser contrast raw speckle image can be calculated based on the combined, e.g., averaged, raw speckle images. In this case, the images are preferably registered with sub-pixel accuracy.

[0196] 3D illustrates a flow diagram of motion compensated laser speckle contrast imaging according to one embodiment. In a first step 362, a first raw speckle image may be calculated 364 based on the first raw speckle image at a first time instance t=t1.

[0197] In a next step 365, a processor may determine a first transformed image, for example a Fourier transform, a Mellin transform, or a log-polar transform, based on the first raw speckle image or based on the first speckle contrast image. Steps relating to image transformation are described in more detail below with reference to Figures 5C-5F.

[0198] Steps 366-369 are similar to steps 362-365, respectively, and are performed at a second time instance t=t2. Thus, a second raw speckle image may be acquired (366) at the second time instance t=t2. The light source may illuminate the target area with coherent light of a predetermined wavelength, and the image sensor may capture the second raw speckle image based on the predetermined wavelength. Based on the second raw speckle image, a second laser speckle contrast image may be calculated (368). Based on the second speckle image, a second transformed image may be determined (369) using a transformation similar to step 365.

[0199] In a next step 370, the processor may determine registration parameters based on the first speckle image and the second speckle image, based on an image associated with the first speckle image and the second speckle image, and / or based on a transformation of the first speckle image and the second speckle image. The registration parameters describe a geometric relationship between the first speckle image and the second speckle image that allows the first speckle image and the second speckle image to be aligned with each other. For example, the registration parameters may include an alignment vector that describes a displacement of one or more pixels in real space or transformed space.

[0200] In a next step 372, the processor may determine an alignment transformation, such as an affine transformation or a more general homography, for registering or aligning the first and second speckle images with each other based on the registration parameters. Based on the alignment transformation, the processor may register or align (374) the first and second laser speckle contrast images with each other by transforming the first and / or the second laser speckle contrast images. Typically, an older image may be transformed to be registered or aligned with a newer image.

[0201] The processor may then calculate a combined, e.g., averaged, laser speckle contrast image based on the registered first and second laser speckle contrast images (376). Calculating the combined image may include calculating a weighted average, where the weights are preferably based on the normalized amount of speckle contrast, the relative change in speckle contrast, the determined alignment vector, and / or parameters associated with the determined transformation. Calculating the combined image may further include applying one or more filters, e.g., a median filter or an outlier filter.

[0202] 3A-3D may be combined. For example, registration parameters may include translation, rotation, and scaling parameters, where the translation parameters are determined based on the speckle image, e.g., using template matching, while the rotation and scaling parameters are determined based on the transformed image, e.g., using cross-correlation of log-polar images.

[0203] FIG. 4A illustrates a flow diagram of a motion-compensated speckle contrast imaging method according to one embodiment. In a first step 402, the method may include exposing a target area to coherent light of a predetermined wavelength. Preferably, the target area includes a living tissue, such as skin, a burn, or an internal organ, such as intestine, or brain tissue. Preferably, the living tissue is perfused and / or includes blood vessels and / or lymphatic vessels. The predetermined wavelength may be a wavelength in the visible spectrum, such as in the red, green, or blue portion of the visible spectrum, or the predetermined wavelength may be a wavelength in the infrared portion of the spectrum, preferably in the near infrared portion.

[0204] In a next step 404, the method may include capturing, for example by an image sensor, at least one sequence of multiple images, where the at least one sequence of multiple images includes a plurality of (unprocessed) speckle images, the plurality of (unprocessed) speckle images being captured during exposure to the first light.

[0205] Each raw speckle image may include pixels, which may be defined by pixel coordinates and have a pixel value, which may define the pixel's location relative to the image and is typically associated with one sensor element of the image sensor, and which may represent light intensity.

[0206] The image sensor may include a 2D image sensor, such as a CCD, such as a monochrome camera or a color camera. The images in the sequence of images may be frames in a video stream or multi-frame snapshots.

[0207] In a next step 406, the method may further include determining one or more registration parameters of an image registration algorithm for registering the speckle images with one another. In principle, the registration parameters may be determined in any suitable manner, for example as described with reference to Figures 5A-5F and 6A-6D.

[0208] For example, the registration parameters may be based on a similarity measure of pixel values ​​of a plurality of pixel groups in a plurality of images of the at least one sequence of a plurality of images, the images in the plurality of images being selected from the plurality of speckle images. Alternatively, the images in the plurality of images may include images other than the unprocessed speckle images associated with the plurality of unprocessed speckle images. The registration parameters preferably define one or more transformations from the group of homographies, projective transformations, or affine transformations.

[0209] Determination of the registration parameters based on pixel groups is described in more detail below with reference to step 416, where it is understood that in this embodiment, the (unprocessed) speckle image is used as the correction image.

[0210] The method may further include determining, using either step 408 or step 410, a combined laser speckle contrast image based on at least a portion of the sequence of the plurality of first speckle images and the one or more determined registration parameters.

[0211] In step 408, the method may further include determining a registered plurality of speckle images by registering the plurality of speckle images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of speckle images. In such an embodiment, the algorithm may first register a sequence of a plurality of raw speckle images using the determined transformation, then calculate a sequence of a plurality of speckle contrast images, and then combine the registered plurality of speckle contrast images.

[0212] In an alternative step 410, the method may further include determining a plurality of speckle contrast images based on the plurality of speckle images, determining a registered plurality of speckle contrast images by registering the plurality of speckle contrast images based on the one or more registration parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered plurality of speckle contrast images. In other words, the algorithm may first calculate a sequence of a plurality of speckle contrast images, then register the plurality of speckle contrast images using the determined transformation, and then combine the registered plurality of speckle contrast images.

[0213] Further alternatives are discussed below with reference to steps 418 and 420.

[0214] FIG. 4B illustrates a flow diagram of a motion-compensated speckle contrast imaging method according to one embodiment. In a first step 412, the method may include exposing a target area to a coherent first light of a first wavelength and a second light of one or more second wavelengths, preferably at least partially different from the first wavelength, alternately or simultaneously. The second light may be, for example, a coherent light of a second wavelength, a narrowband light, or a light including a plurality of second wavelengths in the visible spectrum, for example, white light. Preferably, the target area includes a living tissue, such as skin, a burn, or an internal organ, such as intestine, or brain tissue. Preferably, the living tissue is perfused and / or includes blood vessels and / or lymphatic vessels.

[0215] In a next step 414, the method may include capturing, e.g., by an image sensor system having one or more image sensors in a fixed relationship with respect to one another, a sequence of a first (unprocessed) speckle images during exposure with the first light and a sequence of a second (corrected) images during exposure with the second light. A speckle image of the sequence of a first speckle images may be associated with an image of the sequence of a second images.

[0216] When exposure and acquisition are simultaneous, each second image can be associated with one first speckle image acquired simultaneously.In the embodiment in which the plurality of second images are the same image as the plurality of first speckle images, each image can be considered to be associated with itself, and the image can be referred to as a first speckle image or a second image depending on its function in the algorithm (e.g., determining registration parameters or providing perfusion information).

[0217] In the case of alternating acquisition, a first speckle image may be associated with, for example, either a second image acquired immediately before or after the plurality of first speckle images, or both. In the case where the plurality of first speckle images are acquired at a higher rate than the plurality of second images, several first speckle images may be associated with a single second image.

[0218] Thus, a sequence of a plurality of first unprocessed speckle images of the target area and a sequence of a plurality of corrected images of the target area may be acquired, where each corrected image is associated with one or more of the first unprocessed speckle images. Each corrected image may include a plurality of pixels, the plurality of pixels being defined by pixel coordinates and having a pixel value. The pixel coordinates may define a position of the pixel relative to the image, typically associated with one sensor element of the image sensor. The pixel value may represent light intensity.

[0219] The image sensor system may include one or more 2D image sensors, such as CCDs. The first raw speckle image and the corrected image may be acquired using one or more image sensors, such as a grayscale camera or a color (RGB) camera. The images in the sequence of images may be, for example, frames in a video stream or multi-frame snapshots.

[0220] In a next step 416, the method may further include determining one or more registration parameters of a registration algorithm for registering at least a portion of the one sequence of a plurality of first speckle images based on a similarity measure of pixel values ​​of groups of pixels in at least a portion of the one sequence of a plurality of second images associated with the plurality of first speckle images, the registration parameters preferably defining one or more transformations from the group of homography, projective transformation, or affine transformation.

[0221] Determining the registration parameters may include selecting a first corrected image from the at least a portion of the sequence of corrected images and determining a first group of pixels in the first corrected image. The first corrected image may be a reference corrected image. In one embodiment, the first corrected image may be a first image, for example, when a single output image is generated based on input by a user. In a different embodiment, the first corrected image may be a most recent corrected image, for example, when a continuous stream of output images is being generated.

[0222] A first pixel group may be associated with a feature in the first corrected image, such as an edge or a corner. Preferably, the feature is associated with a physical or anatomical feature, such as a blood vessel, or more particularly, a sharp corner or a bifurcation in a blood vessel. Image features not related to physical features, such as overexposed image parts or speckle edges, may show large frame-to-frame variations and therefore it may not be very useful to register the images. The feature may be a predefined feature, such as a feature belonging to a class of features, such as a corner or a region with a large intensity difference. The feature may further be determined, for example, by a quality measure, a constraint on the mutual distance between features, etc.

[0223] Alternatively, a pixel group may be associated with an area in the first corrected image, for example a set of neighborhoods of a given pixel, for example all the pixels in the image. Thus, the pixel group may include, for example, all the pixels in the image, a contiguous area of ​​pixels at a given location, for example the center of the image, or a selection of pixels evenly distributed over the image.

[0224] Determining the registration parameters may further include selecting one or more second corrected images different from the first corrected image. For each of the selected one or more second corrected images, a plurality of second pixel groups may be determined. The second pixel groups may be associated with features in the second corrected images. If feature-based (image) registration is used, e.g. using a coarse optical flow algorithm, preferably the same algorithm is used to determine the first pixel groups and the second pixel groups.

[0225] If the first pixel group is determined based on pixel coordinates, the second pixel group may be determined by convolving or cross-correlating the first pixel group with the second corrected image and selecting the pixel group that is most similar to the first pixel group based on, for example, a suitable similarity measure. The convolution may be spatially constrained, for example, by searching only matching second pixel groups that are close to the location of the first pixel group. Alternatively, or additionally, the second pixel group may be constrained to preserve the mutual orientation of the first pixel groups, for example to prevent anatomically impossible combinations.

[0226] The second pixel group may then be associated with the first pixel group based at least on similarity of pixel values. In embodiments in which the second pixel group is determined by matching or convolution with the first pixel group, such association may be made as part of determining the second pixel group. If feature-based registration is used, the second pixel group may be associated with the first pixel group based on similarity between features associated with the second pixel group and features associated with the first pixel group.

[0227] A transformation for registering the second corrected image with the first corrected image, and thus the associated plurality of first speckle images or the derived plurality of first speckle contrast images, can be determined based on pixel coordinates of pixels in the associated first and second pixel groups. Determining a transformation can include determining 3D motion of the image sensor system relative to the target area, or can be informed about the effect that this 3D motion will have on the acquired image.

[0228] As an intermediate step, an alignment vector may be determined based on the positions of the first pixel group and the associated second pixel group, for example based on the positions of features in the first and second corrected images. The alignment vector may represent a movement of the target area relative to the image sensor or image capture device. In some embodiments, a neighborhood of one or more determined object features may be used to determine the alignment vector and / or the transformation.

[0229] Determining the alignment vector and / or determining the transformation may be based on optical flow parameters determined using any suitable coarse or fine optical flow algorithm. Determining the alignment vector may include determining corresponding or matching feature pairs, where one feature of the pair is determined in a corrected image associated with a first time point and the other feature of the pair is determined in a subsequent corrected image in a sequence of corrected images associated with a later time point.

[0230] Methods for determining the registration parameters, and optionally the features, are discussed in more detail below with reference to FIGS.

[0231] The method may further include determining, using either step 418 or step 420, a combined laser speckle contrast image based on at least a portion of the sequence of the plurality of first speckle images and the one or more determined registration parameters.

[0232] In step 418, the method may further include determining a registered plurality of first speckle images by registering the at least a portion of the sequence of a plurality of first speckle images based on the one or more registration parameters and the registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle images. In such an embodiment, the algorithm may first register a sequence of a plurality of raw speckle images using the determined transformation, then calculate the sequence of a plurality of speckle contrast images, and then combine the registered speckle contrast images.

[0233] In an alternative step 420, the method may further include determining a plurality of first speckle contrast images based on at least the portion of the sequence of a plurality of first speckle images, determining a registered plurality of speckle contrast images by registering the plurality of first speckle contrast images based on the one or more registration parameters and the registration algorithm, and determining a combined speckle contrast image based on the registered plurality of first speckle contrast images. In other words, the algorithm may first calculate a sequence of a plurality of speckle contrast images, then register the plurality of speckle contrast images using the determined transformation, and then combine the registered plurality of speckle contrast images.

[0234] In a further alternative embodiment, determining the combined laser speckle contrast image may include determining a sequence of a plurality of first unprocessed speckle images registered based on the plurality of first unprocessed speckle images and the determined transformation, determining a combined speckle contrast image based on two or more registered first unprocessed speckle images of the registered sequence of a plurality of first unprocessed speckle images, and determining a combined speckle contrast image based on the combined unprocessed speckle images. In such an embodiment, the algorithm may first register the sequence of a plurality of unprocessed speckle images using the determined transformation, then combine the registered plurality of speckle contrast images, and then calculate a speckle contrast image.

[0235] In yet another alternative embodiment, the combination and calculation of speckle contrast can be a single step, for example, by calculating temporal or spatio-temporal speckle contrast based on a sequence of the registered first raw speckle images.

[0236] Combining multiple raw speckle images or multiple speckle contrast images can include averaging, weighted averaging, filtering, e.g., with a median filter, etc. The weights in the weighted average can be based on quantities derived, e.g., from the speckle contrast, derived from the registration parameters, or derived from the alignment vectors. Methods for combining multiple raw speckle images or multiple speckle contrast images are discussed in more detail below with reference to FIG. 9.

[0237] 4C illustrates a flow diagram of a motion compensated speckle contrast imaging method according to one embodiment. Steps 422 and 424 may be similar to steps 402 and 404, or steps 412 and 414, respectively.

[0238] In a next step 425, the method may further include determining a transformed image by transforming the plurality of first speckle images and / or an image associated with the plurality of first speckle images (e.g., an image captured during exposure with the second light). The transformed image may be obtained, for example, using a Fourier transform, a Mellin transform, and / or a log-polar transform.

[0239] In a next step 426, the method may further include determining, based on the transformed images, one or more registration parameters of an image registration algorithm for registering the speckle images with one another. Examples are described below with reference to Figures 5D-5F and 6.

[0240] The method may further include determining a combined laser speckle contrast image based on at least a portion of the sequence of the first speckle images and the one or more determined registration parameters using either step 428 or step 430. These steps are similar to steps 408 and 410, and steps 418 and 420, respectively, described above.

[0241] FIG. 5A illustrates a method for determining registration parameters according to one embodiment. 1~3 may be determined in a first corrected image 502 acquired at a first time t=t1. Each of the plurality of first features may be associated with a pixel group in the first corrected image. Preferably, the first features are associated with an anatomical structure, e.g., a blood vessel 504. 1~2, or other stable features that may be assumed not to move between subsequent frames. Therefore, the first correction image is preferably acquired using light that allows such anatomical structures to be clearly visible. For example, green light may be used, which is strongly absorbed by blood vessels but not by most other tissues. Therefore, blood vessels may appear as dark structures in a bright environment, resulting in high visual distinctiveness. However, other embodiments may use one or more other wavelengths of light, such as blue light or white light.

[0242] Feature 506 1~3 can be determined using any suitable feature detection algorithm, for example a feature detector based on the Harris detector or the Shi-Tomasi detector, for example goodFeaturesToTrack from the OpenCV library. Other examples of suitable feature detectors and descriptors include Speeded-Up Robust Features (SURF), Features from Accelerated Segment Test (FAST), Binary Robust Independent Elementary Features (BRIEF), and combinations thereof, for example ORB. A variety of suitable algorithms are implemented in commonly available image processing libraries, for example OpenCV. Preferably, depending on the application, the algorithm should be fast enough to allow real-time image processing.

[0243] Typically, sharp corners (e.g., feature 506 1,3) and bifurcations (e.g., feature 5062) are preferred features. Preferably, a deterministic feature detection algorithm is used, i.e., a feature detection algorithm that detects the same feature in the same image. Preferably, the features should be distributed over a large portion of the image area. A good distribution of feature points across the image can be obtained by requiring a minimum distance between selected feature points. In some embodiments, based on, for example, a BRIEF or ORB implementation, the features can be assigned a descriptor that identifies the feature characteristics to facilitate feature discrimination and feature matching.

[0244] The minimum number of feature points depends on the type of transformation, e.g., an affine transformation has 6 degrees of freedom, whereas a homography has 8 degrees of freedom. Thus, the first plurality of first features may comprise at least 5 features, preferably at least 25, more preferably at least 250, even more preferably at least 1000 features. The number of features may depend on the amount of pixels in the image, with more features being used for images with more pixels. Typically, a larger number of features may result in a more accurate transformation, since random errors may be averaged out.

[0245] However, there are various reasons why the number of features may be limited. For example, there may be only a limited number of features that meet a given quality metric, such as the magnitude of local contrast or corner sharpness. In addition, computation time increases with the number of features, and therefore the number of features may be limited to allow real-time image registration; for example, for a 50 fps video feed, the entire algorithm should preferably be less than 20 ms per frame.

[0246] A second plurality of second features 516 associated with the second pixel group 1~3can be determined in a second corrected image 512 acquired at a second time t=t2. Preferably, the field of view of the second corrected image overlaps substantially, preferably more than half, with the field of view of the first corrected image. Preferably, the second feature is located on the same anatomical structure as the first feature, e.g., blood vessel 514. 1~2 , Preferably, the same feature detection algorithm is used to detect features in both the first and second corrected images.

[0247] The determined first and second features may include position information relative to the first and second corrected images, respectively. In this example, this is shown in comparison image 522. The first plurality of first features 506 1~3 and the second plurality of second features 516 1~3 Based on the above, multiple alignment vectors 524 1~2 may be determined, where an alignment vector describes the displacement of a feature relative to an image. In an intermediate step, corresponding feature pairs may be determined, e.g. feature 5061 may be associated with feature 5161, feature 5062 may be associated with feature 5162, and feature 5063 may be associated with feature 5163. Corresponding feature pairs may be determined, for example, based on feature parameters, such as local contrast or corner sharpness, or based on the distance between features in the first and second corrected images. In some embodiments, not all features in the first corrected image can be paired with a feature in the second corrected image. In some embodiments, alignment vector 524 may be determined based on feature parameters, such as local contrast or corner sharpness, or based on the distance between features in the first and second corrected images ... 1~3 The determination of and the corresponding feature pairs may be performed in a single step.

[0248] For example, if a minimum distance between features is imposed and the displacement is assumed to be less than the minimum distance, an algorithm that minimizes the distance between the points formed by the first and second features, respectively, may implicitly determine the corresponding feature pairs and the alignment vectors for each corresponding feature pair. In a typical embodiment, a typical frame-to-frame displacement is only a few pixels. In one embodiment, the alignment vectors may be filtered to remove possible outliers, such as alignment vectors that deviate from alignment vectors originating from nearby features by more than a predetermined amount.

[0249] In other embodiments, the alignment vector may be determined based on pixel values ​​in the first and second corrected images and based on associated pixel groups. As explained above with reference to FIG. 3, in such embodiments, feature detection may be omitted. Instead, the alignment vector may be determined, for example, using a dense optical flow algorithm, such as the Pyramid Lucas-Kanade algorithm or the Farneback algorithm. In principle, any method of determining an alignment vector based on pixel values ​​of corresponding pixel groups may be used.

[0250] The plurality of displacement vectors 524 1~3 A transformation can be determined based on the first unprocessed speckle image. In one embodiment, the transformation can be determined by an average or median displacement vector, or by another alignment vector that is statistically representative of the alignment vectors. In different embodiments, the transformation can be an affine transformation, a projective transformation, or a homography, for example combining translation, rotation, scaling, and shear transformations. Preferably, the transformation projects features from the first corrected image to corresponding features in the second corrected image. The transformation can then be applied to the first unprocessed speckle image to register the first unprocessed speckle image with the second unprocessed speckle image.

[0251] In one embodiment, the first corrected image and the second corrected image may be pre-processed before determining features. For example, as described above with reference to FIG. 1E, overexposed and / or underexposed regions may be identified based on pixel values. These regions may then be masked so that features cannot be detected within them. The mask may be slightly larger than the overexposed or underexposed regions, for example by increasing the identified regions by a predetermined number of pixels. Masking the overexposed and / or underexposed regions may improve the quality of the features, for example by preventing features associated with edges or corners of the overexposed regions. As described above with reference to FIG. 1E, the same mask may be applied before feature detection to prevent unreliable features from being detected if a mask has been determined to identify artifacts.

[0252] 5B shows a further method for determining registration parameters according to one embodiment. This method may be referred to as template matching. A first region 531 is selected in a first image 530 acquired at a first time instance t=t1. The first image may be, for example, an unprocessed speckle image, a speckle contrast image, or a correction image. The first region has a predefined size expressed in pixels and is selected at a predefined location in pixel coordinates. The first region is preferably selected at or near the center of the first image.

[0253] A second region 533 is selected in a second image 532 acquired at a second time instance t=t2. The second image is typically of the same type as the first image, for example both can be unprocessed speckle images. The second region has a predefined size expressed in pixels and is selected at a predefined location in pixel coordinates. The second region is typically smaller than the first region. The size difference can be selected based on an estimated or expected amount of motion. The center of the second region typically coincides with the center of the first region. Preferably, the second region is selected relative to the first region such that there is a high probability that the imaged (anatomical) region represented by the second region is contained in the imaged region represented by the first region. A single region in each image may be sufficient to determine the global translational registration parameters. 6D, for example, a plurality of first regions may be selected in the first image and a corresponding plurality of second regions may be selected in the second image to determine non-global (i.e., regionally or locally) registration parameters. The first regions and the second regions may be selected in an overlapping or non-overlapping manner.

[0254] Based on the first region and the second region, registration parameters may be determined. In this example, this is shown in a comparison image 534. For example, a sub-region in the first region that is most similar to the second region may be determined, this process is also known as matching. The registration parameters, e.g., alignment vector 535, may be determined based on the relative positions of the second region and a sub-region within the first region. The matching method can be, for example, feature-based, intensity-based, or frequency-based. An example of feature-based matching is described above with reference to FIG. 5A, and an example of frequency-based matching is described below with reference to FIG. 5C. An example of intensity-based matching is cross-correlation between the first region and the second region. The matching method may also include a coordinate transformation, for example, as described below with reference to FIG. 5D. Depending on the matching method, the registration parameters may include translation parameters, rotation parameters, and / or other parameters.

[0255] FIG. 5C illustrates a further method for determining registration parameters according to one embodiment. FIG. 5C illustrates a frequency-based registration algorithm, also known as phase correlation. A first image 540 acquired at a first time instance t=t1 is transformed using a transform resulting in a first transformed image 541. In this example, a discrete Fourier transform is used. Similarly, a second image 542 acquired at a second time instance t=t2 is transformed using a discrete Fourier transform resulting in a second transformed image 543. Based on a comparison (544) of the first transformed image and the second transformed image, registration parameters can be determined to register (546) the first speckle image and the second speckle image.

[0256] Image registration based on Fourier transform is an example of frequency-based image registration. These methods are generally well known in the art. The comparison of the transformed images may include calculating a cross-power spectrum of the transformed images. The calculation of the cross-power spectrum may include determining a complex conjugate of one of the transformed images and element-wise multiplying it with the other of the transformed images. The comparison may further include determining an (inverse) Fourier transform of the cross-power spectrum and determining a peak in the resulting image, for example by applying an argmax function. By using interpolation, the peak can be determined with sub-pixel accuracy. The location of the peak corresponds to the translation of the images to be aligned. In the illustrated example, image 544 represents the inverse Fourier transform of the cross-correlation of a first transformed image and a second transformed image, and the vector between the upper right corner and peak 545 represents the translation required to align the second image with the first image.

[0257] The comparison may include further operations to improve the results. For example, a two-dimensional Hanning window may be applied to the first and second images before applying the Fourier transform. As another example, a filter, such as a blurring filter or an interpolation filter, may be applied to the inverse Fourier transform of the cross-correlation of the transformed images to improve peak detection. Various other operations to improve frequency-based image registration are known in the art. For example, high frequencies may be filtered to reduce the effects of noise. Meanwhile, a high-pass filter may reduce image artifacts caused by image boundaries. In addition, frequencies corresponding to speckles may be suppressed to prevent matching of speckles rather than anatomical structures. Of course, care must be taken not to suppress all frequencies, especially frequencies where anatomical structures in the region of the image may give relatively strong signals. This may depend on the type of tissue being imaged.

[0258] FIG. 5D illustrates a further method for determining registration parameters, according to one embodiment. In this example, a so-called log-polar coordinate transformation is used. The log-polar coordinate transformation is typically applied in combination with a method for determining a translation, for which any of the methods described above with reference to FIGS. 5A-5C may be used. It may therefore be considered as a pre-processing step for any of these methods. A first image 550 acquired at a first time instance t=t1 is transformed using a coordinate transformation, resulting in a first transformed image 551. The horizontal axis of the transformed image represents the angle φ with respect to the horizontal axis of the first image, and the vertical axis of the transformed image represents the logarithm of the (relative) radial distance log(r) from the center of the first image. Since r should be dimensionless, r is typically expressed in units of pixels or r max It is expressed as a relative distance from maxrepresents the maximum distance from the origin (in this example, one of the diagonals), i.e., if pixel coordinates in the first image are represented by x and y (along the horizontal and vertical axes, respectively) and the origin is chosen at the center of the image, then

number

[0259] Similarly, a second image 552 acquired at a second time instance t=t2 is transformed using the same coordinate transformation, resulting in a second transformed image 553. In the illustrated example, the complete first speckle image and the complete second speckle image are transformed. In other embodiments, only one or more regions of the first speckle image and the second speckle image are transformed. Typically, at least one region includes the center of the image.

[0260] Based on a comparison 554 of the first transformed image and the second transformed image, a displacement or shift in the transformed (in this case log-polar) coordinate system may be determined. The displacement may be determined using any suitable method, such as one of the methods described above with reference to Figures 5A-5C. Based on the determined displacement, registration parameters may be determined (556) for registering the first and second speckle images with each other. In the illustrated example, the second image has undergone a small translation in addition to rotation and scaling, so that for small r (lower part of the image), the shift (and therefore rotation) appears much larger than it actually is. For larger r (upper part of the image), data does not exist for all values ​​of r. Therefore, it may be advantageous to use only a limited range of r values.

[0261] The advantage of using a log-polar transformation is that the horizontal shift 555 of the transformed image corresponds to the rotation 557 of the untransformed image, and the vertical shift of the transformed image corresponds to the scaling of the untransformed image. Thus, rotation and scaling can be determined in a relatively simple manner, especially global scaling and rotation. In this manner, a relatively large amount of image data may be used, since a relatively large image area is used to determine the (global) scaling and rotation registration parameters, and thus the effect of noise on the determined registration parameters can be reduced.

[0262] Using a non-feature-based method, such as those described with reference to Figures 5B-5D, it is generally not necessary to determine multiple alignment vectors, as may be done using a feature-based method, such as those described with reference to Figure 5A. Instead, for example, a single matrix can be determined based on the registration parameters, and when this matrix is ​​applied to the first image, the first image and the second image are aligned. This can reduce the amount of calculations.

[0263] 5E and 5F illustrate further methods for determining registration parameters, according to one embodiment. In these examples, two or more of the methods described with reference to FIGS. 5A-5D are combined. For example, the template matching method described with reference to FIG. 5B is used to determine the translation, while a log-polar transformation is used to determine the rotation and scaling. When multiple methods are combined, an intermediate registration step may be applied between the two registration parameter determinations. This tends to improve the image registration results.

[0264] FIG 5E illustrates an example where translation parameters are determined and applied to the first (speckle) image first in an intermediate registration step. Subsequently, rotation and scaling parameters are determined and applied based on the results of the intermediate scaling step. FIG 5F shows an example where rotation and scaling parameters are determined and applied first, followed by translation parameters. In general, it is advantageous to determine and apply the largest transformation first.

[0265] In particular, Figure 5E illustrates an example in which a first (corrected) image 560 and a second (corrected) image 562 are obtained. In step 563, a translation is determined based on the first and second images and applied to the first image, resulting in a translated first image (564). The translation may be determined using any suitable method, for example, as described above with reference to Figures 5A-5C. For example, template matching may be used where templates are matched using frequency-based phase correlation or intensity-based cross-correlation.

[0266] Subsequently, both the translated first image and the second image are transformed (565, 567) using a coordinate transformation, in this example a log-polar transformation (e.g., as described above with reference to FIG. 5D), resulting in a log-polar transformed first image 566 and a log-polar second image 568. In step 569, a shift or displacement is determined based on the log-polar transformed first image and the log-polar transformed second image. The shift may be determined using any suitable method, for example, one of the methods for determining a translation such as those described above with reference to FIGS. 5A-5C. The same method as in step 563 may be used, or a different method may be used. Based on the determined shift, rotation and / or scaling parameters are determined (571) and applied to the translated first image, resulting in a rotated and / or scaled translated first image (572).

[0267] The rotated and / or scaled transformed first image is then combined with the second image, resulting in a combined image (574), e.g., using a weighted average. As described in more detail below with reference to FIG. 9A, the weights may be based on the registration parameters. In this example, the registration parameters include both translation parameters and rotation and / or scaling parameters. Thus, the weights may be based on translation parameters only, rotation parameters only and / or scaling parameters only, or both. For example, a pixel-wise displacement vector may be determined based on the combined registration parameters, and the weights may be based on a statistical representation of the pixel-wise displacement vectors for all pixels or a subset thereof (e.g., a region of interest), e.g., the average or maximum of the norm of the displacement vectors.

[0268] Figure 5F illustrates a variation of Figure 5E. A first image 580 and a second (corrected) image 582 are obtained. The first image and the second image are transformed (583, 585) using a coordinate transformation, in this example a log-polar transformation, resulting in a log-polar transformed first image 584 and a log-polar transformed second image 586. In step 587, a shift is determined based on the log-polar transformed first and second images. Based on the determined shift, rotation and / or scaling parameters are determined (589) and applied to the first image, resulting in a rotated and / or scaled first image (590).

[0269] In step 591, a translation is determined based on the rotated and / or scaled first image and the second image, and the determined translation is applied to the rotated and / or scaled first image, resulting in the rotated and / or scaled translated first image (592). The rotated and / or scaled translated first image is then combined with the second image, resulting in a combined image (594).

[0270] In yet another embodiment, the rotation and / or scaling determination and the translation determination are both performed based on the (untransformed) first and second images, the advantage being that both (sets of) registration parameters can then be determined in parallel.

[0271] 6A shows an example of determining registration parameters according to one embodiment, where the transformation to be determined is a translation. As described with reference to FIG. 5A, the first plurality of first features 602 1~3 may be determined in a first corrected image acquired at t=t1, and a second plurality of second features 604 1~3 can be determined in a second corrected image acquired at t=t2. Based on the corresponding feature pairs, a number of alignment vectors 606 1~3 For ease of understanding, only the features and the alignment vectors are shown, not the (anatomical) structures.

[0272] In a typical situation, the determined alignment vector 606 1~3are not all exactly the same. In the illustrated example, alignment vector 6061 is slightly shorter than the average, while alignment vector 6063 is slightly larger than the average. Similarly, the direction of each alignment vector shows some variability. Based on these alignment vectors, an average alignment vector 608 can be determined. The translation can be defined by a single vector. For example, all pixels of the first raw speckle image acquired at t=t1 can be shifted by an amount equal to the average alignment vector. In principle, the translation can be determined based on a single alignment vector. However, by determining multiple alignment vectors, the accuracy of the transformation can be improved.

[0273] In one embodiment, the average alignment vector 608 and the determined alignment vector 606 1~3 A similarity between the features in the first corrected image and the corresponding features in the second corrected image after transformation can be calculated, for example based on the dispersion of the alignment vectors. In that way, an indication of how well the transformation compensates for the detected displacement of individual feature pairs can be obtained. Alternatively, the average distance between the features in the first corrected image after transformation and the corresponding features in the second corrected image can be determined.

[0274] 6B illustrates an example of determining a transformation according to one embodiment of the present invention, where the determined transformation is an affine transformation. Similar to FIG. 6A, a first plurality of first features 612 1~3 , a second plurality of second features 614 1~3 , and a plurality of alignment vectors 616 1~3 However, in this example, the average alignment vector 618, which has a length of approximately zero, is not representative of the determined alignment vectors, which are typically longer and point in different directions.

[0275] Therefore, more general transformations are needed to compensate for this type of motion, such as affine transformations. Affine transformations include translation, rotation, mirroring, scaling, and shear transformations, as well as combinations thereof. By restricting the transformation parameter values, it is possible to selectively exclude transformations. For example, mirroring can be excluded as a possible transformation, since it is typically not physically possible.

[0276] In general, an affine transformation can be calculated using a six degree of freedom transformation matrix, such as that described in equation (1) below, acting on points expressed in homogeneous coordinates. By restricting the possible values ​​of the affine transformation matrix, the affine transformation can be restricted to only predefined motions. For example, a more specific transformation matrix, e.g. restricted to a rotation matrix, can be obtained, which may be more suitable for certain applications.

number

[0277] where A is the transformation matrix that transforms a transformed point p' with coordinates x' and y', typically in pixel coordinates. The matrix A has six free parameters, of which t x and t y defines the translation, while the submatrix

number

[0278] To solve equation (1), at least three alignment vectors may be used to provide a solvable system of six equations and six unknowns. Such linear systems can be deterministically solved, as is known in the art. In a typical embodiment, many alignment vectors may be determined, each of which may contain a small error. Therefore, a more robust approach may be to use multiple alignment vectors and use a suitable fitting algorithm, such as least-squares fitting, as shown in equation (2) below.

number

[0279] The reliability of the determined transformation may again be determined as described above with reference to FIG. 6A.

[0280] 6C shows an example of determining a transformation according to one embodiment of the present invention, where the determined transformation is a projective transformation. Projective transformations include, and are more general than, affine transformations. Projective transformations include, for example, skewing transformations. They may be required, for example, to compensate for changes in angle between the camera and the target area.

[0281] Similar to FIGS. 6A and 6B, the first plurality of first features 622 1~3 , a second plurality of second features 624 1~3 , and a plurality of alignment vectors 626 1~3 can be determined. In this example, the translation on the left side of the image is much smaller than on the right side of the image. Thus, applying an average translation will result in the pixels on the left side being overly transformed and the pixels on the right side being under-transformed. This type of displacement can be corrected by a projective transformation. Various methods for determining a projective transformation based on four or more alignment vectors are known in the art.

[0282] In general, the projective transformation can be calculated using the inhomogeneous coordinates (x1, y1, z1) and (x2, y2, z2) using a projection matrix as shown in equation (3) below.

number

[0283] The non-homogeneous coordinates (x1, y1, z1) may be related to the pixel coordinates (x, y) of the feature via x1=x, y1=y, and z1=1, while the transformed pixel coordinates (x', y') may be obtained from the non-homogeneous coordinates via x'=x2 / z2, and y'=y2 / z2. In some embodiments, the alignment vector may be determined by (x'-x, y'-y). In other embodiments, the alignment vector is not explicitly constructed.

[0284] Therefore, equation (3) can be rewritten as two independent equations, as shown below in equation (4).

number

[0285] Solving for H, equation (4) can be rewritten as equation (5) below.

number

number

[0286] Using a set of N feature points, a system of equations can be set up as shown in equation (7) below.

number

number

[0287] By definition, the matrix H is homogeneous and can be scaled by any constant, so the homography matrix contains eight degrees of freedom. Thus, equation (7) can be solved using only four sets of coordinates given by the four features. The homography matrix H can be expressed, for example, as equation (9) below:

number

number

[0288] Equation (7) can be solved deterministically using at least four points, but using more points can give more robust results, similar to those described above for the affine transformation. In the case of a projective transformation or homography, this can be done using singular value decomposition (SVD). In some embodiments, the step of determining the alignment vector or point pair is combined with the step of determining the homography to find the most accurate projective transformation, which can be done using an algorithm, such as RANSAC.

[0289] In one embodiment, the algorithm may first calculate a relatively simple transformation, such as a translation. The algorithm may then determine whether the calculated transformation reproduces the determined alignment vector with sufficient accuracy. If not, the algorithm may try a more general transformation, such as an affine transformation, and repeat the same procedure. This may reduce the required computation time if translation is sufficient in a sufficiently large number of cases. In a different embodiment, the algorithm may always calculate a general transformation, such as a general homography. This may result in a more accurate registration of raw speckle images or speckle contrast images.

[0290] The examples described above with reference to Figures 6A-6C are based on feature-based transformation parameters. However, other methods of providing registration parameters for multiple locations in the first speckle image and / or the second speckle image may be used to obtain the same or similar results. For example, the first speckle image and / or the second speckle image, or a transform thereof, may be divided into multiple regions and registration parameters may be determined for each divided region. The registration parameters for multiple regions may be combined as described above with reference to Figures 6A-6C.

[0291] 6D illustrates an example of determining multiple transformations according to one embodiment of the present invention. Each corrected image 650 in the sequence of corrected images includes multiple regions 652. 1~n , preferably into a number of separate regions that together cover the entire image, e.g., a rectangular grid. Then, the transformations 6541, 6542, ..., 6543 are applied to the entire image, preferably in the same manner as described above with reference to Figures 6A-6C. n However, in each region 6541, 6542, ..., 654 nFor each one, a transformation may be determined, for example as described above, for example with reference to Figures 5A-5D or 6A-6C (when the method is applied to the entire image). Each region may then be transformed using the transformation determined for that region. Alternatively, the determined transformation may be assigned to, for example, a central pixel in the region, and the remaining pixels are transformed according to an interpolation technique based on the transformation of the region containing that pixel and the transformations of adjacent regions.

[0292] In one embodiment, each region may be a single pixel. In such an embodiment, the transformation may be determined based on the pixel value and based on the pixel values ​​of pixels in the region surrounding the pixel.

[0293] 7A and 7B illustrate a flow diagram for laser speckle contrast imaging combining three or more raw speckle images according to one embodiment of the present invention. At a first time t=t1, a first raw speckle image 7021 based on light of a first wavelength may be obtained and used to calculate a first laser speckle contrast image 7041. A first corrected image 7061 based on light of at least a second wavelength and associated with the first raw speckle image may be obtained, and a first plurality of first features may be detected in the first corrected image (7081). Items 7021 to 7081 may be obtained by performing step 7101, which may be similar to steps 302 to 308 described with reference to FIG. 3. In some embodiments, the first corrected image and the first raw speckle image may be the same image.

[0294] At a second time t=t2, step 7101 may be repeated as step 7102, resulting in a second raw speckle image 7022 based on the first wavelength of light, a second laser speckle contrast image 7042, a second corrected image 7062 based on the at least a second wavelength of light, and a plurality of second features 7082. A plurality of first alignment vectors 7121 may be calculated based on the plurality of first features 7081 and the plurality of second features 7082. Based on the plurality of first alignment vectors, a first transformation 7141 may be determined, which may be used to transform the first laser speckle contrast image 7041 to register the first laser speckle contrast image 7041 with the second laser speckle contrast image 7042, resulting in a registered first laser speckle contrast image 7181. As discussed above, in some embodiments, the transformation may be determined without explicitly detecting feature and / or alignment vectors.

[0295] Optionally, first weights 7161 may be determined based on the plurality of first alignment vectors 7121. The weights may be correlated, preferably inversely correlated, to the length of a representative alignment vector, e.g. the maximum, average or median alignment vector, or, e.g., to the average or median length of the plurality of first alignment vectors. In an embodiment in which an image is divided into regions and a transformation is determined for each region, as described above with reference to FIG. 6D, a weight may be determined for each region based on the registration parameters associated with that region, or a single weight may be determined for the entire image, e.g., based on a representative parameter, e.g. the maximum, average or median displacement. Determining the weighted average is discussed in more detail below with reference to FIGS. 9A and 9B.

[0296] The registered first laser speckle contrast image 7181 may then be combined with the second laser speckle contrast image 7042, resulting in a combined first laser speckle contrast image 7201. The combined laser speckle contrast image may be, for example, a pixel-by-pixel average or maximum of the registered first laser speckle contrast image 7181 and the second laser speckle contrast image 7042. Optionally, first weights 7161 and / or second weights 7162 may be used to determine the weighted average.

[0297] At a third time t=t3, step 7101 may be repeated as step 7103, resulting in a third raw speckle image 7023 based on the first wavelength of light, a third laser speckle contrast image 7043, a third corrected image 7063 based on the at least second wavelength of light, and a plurality of second features 7083. A plurality of second alignment vectors 7122 may be calculated based on the plurality of second features 7082 and the plurality of third features 7083. A second transformation 7142 may be determined based on the plurality of second alignment vectors. The second transformation may be used to transform the combined first laser speckle contrast image to register the combined first laser speckle contrast image 7201 with the third laser speckle contrast image 7043, resulting in a registered combined first laser speckle contrast image 7221. The registered first laser speckle contrast image 7181 can then be combined with the second laser speckle contrast image 7042 , resulting in a combined first laser speckle contrast image 7201 .

[0298] The registered combined first laser speckle contrast image 7201 may then be combined with the third laser speckle contrast image 7043, resulting in a combined second laser speckle contrast image 7202. Thus, the combined second laser speckle contrast image may include information from the first laser speckle contrast image 7041, the second laser speckle contrast image 7042, and the third laser speckle contrast image 7043. By repeating these steps, the nth image may include information from all previous n-1 images. Preferably, the weighting may then be skewed to give more recent images a higher weight than older images. This embodiment is particularly useful in the case of streaming video, where each captured frame is processed and output with minimal delay. Another advantage of this method is that the motion between two subsequent frames can be assumed to be relatively small, which may speed up processing and may allow a wider range of algorithms to be used since some algorithms may not work very well for large motions. In general, feature-based algorithms may be more reliable for relatively large displacements.

[0299] 7B illustrates a flow diagram for an alternative method for laser speckle contrast imaging combining three or more raw speckle images according to one embodiment of the present invention. In this embodiment, a predetermined number of images are combined into a single combined image. Steps 752, which relate to image acquisition, speckle contrast image calculation, and feature determination, are 1~n ~Process 760 1~n 7A. 1~n ~Process 710 1~n It can be the same as:

[0300] However, unlike the method illustrated in FIG. 7A, all alignment vectors 762 1,2is determined for a single reference image, e.g., the first or last image in a sequence of n images. In the illustrated example, the image is registered with the nth image. Thus, the transformation 764 1,2 The first and second speckle contrast images are registered with the nth speckle contrast image by applying weights 766 to the first and second speckle contrast images, respectively. 1,2 is also determined for the nth image based on a transformation parameter, e.g., average displacement. In a final step, all n registered speckle-contrast images may be combined into a single combined speckle-contrast image 770.

[0301] The method illustrated in Figure 7B may result in a combined image with higher image quality than the method illustrated in Figure 7A, but at the expense of a greater delay between image capture and display, and is therefore particularly advantageous for recording snapshots.

[0302] In one embodiment, the method illustrated in FIG. 7B can be applied to a sliding group of images, e.g. the last n images of a video feed. In order to keep the time delay low, in this case n should preferably not be too large, e.g. n can be about 5-20 when the frame rate is e.g. 50-60 fps. Of course, the number of frames that can be processed also depends on the hardware and algorithms, so a higher frame amount can still be feasible. Thus, some of the advantages of both methods can be combined. The advantage of using a relatively small number of frames is that dynamic phenomena, e.g. the effects of the heartbeat, can be imaged. The advantage of a higher frame amount is that such transient effects can be filtered out, especially if the number of frames is selected to cover an integer multiple of the heartbeat and / or respiratory cycle.

[0303] FIG. 8 illustrates a flow diagram for calculating a corrected laser speckle contrast image according to one embodiment. In general, one may be interested in the relative motion of one or more objects in the target area relative to one or more other objects in the target area, such as the motion of fluid or red blood cells relative to tissue. In such a case, noise in a signal derived from a moving object (desired signal) may be compensated by a signal derived from a reference object (reference signal). The underlying principle is that the desired signal may include a first component based on the motion of a quantity of interest relative to the reference object and a second component based on the motion of the entire target area relative to the camera. The reference signal may include only or mainly a component based on the motion of the entire target area relative to the camera. Therefore, the reference signal may be correlated to the second component of the desired signal. This correlation may be used to correct or compensate the desired signal. For example, a correction term based on the signal strength of the reference signal may be added to the desired signal.

[0304] In the embodiment illustrated in Fig. 8, a target area is illuminated with coherent light of a first wavelength 802, e.g., red or infrared light, and with coherent light of a second wavelength 812, e.g., green or blue light. Preferably, the light of the first wavelength is mostly scattered by an object or fluid of interest, e.g., blood. Preferably, the light of the second wavelength is mostly scattered by the surface of the target area and / or mostly absorbed by the object or fluid of interest. Preferably, the second wavelength is selected such that the reflection of the second wavelength by blood is at least 25%, at least 50%, or at least 75% less than the reflection by tissue. Preferably, the target area is illuminated with light of the first and second wavelengths simultaneously.

[0305] The scattered light at the first wavelength may result in a first raw speckle image, which may be captured by a first image sensor (804). The scattered light of the second raw speckle image may be captured by a second image sensor, preferably different from the first image sensor (814). The first raw speckle image may be referred to as a desired signal raw speckle image, and the second raw speckle image may be referred to as a reference signal image or a correction signal image.

[0306] A first speckle contrast image may be calculated based on the first raw speckle image (806). A second speckle contrast image may be calculated based on the second raw speckle image (816). Preferably, the speckle contrast is calculated in the same manner for the first and second raw speckle images. The speckle contrast may be calculated, for example, in the manner described above with reference to step 304 in Figures 1 and 3A.

[0307] In a next step 808, a corrected speckle contrast image can be calculated based on the first and second speckle contrast images. Calculating a corrected speckle contrast image can include, for example, adding a correction term or multiplying a correction factor. The correction term can be based on a determined speckle contrast amount in the speckle contrast image compared to a reference speckle contrast amount. The reference speckle contrast amount can be, for example, predetermined, or can be dynamically determined, for example, based on the speckle contrast amounts in a certain number of previous second contrast images, or based on a speckle contrast image with very little motion, for example determined by a motion correction algorithm as described above.

[0308] The corrected speckle contrast image may then be stored for further processing, such as reregistration or realignment and temporal averaging, as described with reference to Figures 3A and 3B (810). The second unprocessed speckle image and / or the second speckle contrast image may also be stored for further processing, such as to determine alignment vectors within a plurality of second unprocessed speckle images to reregister or realign a plurality of simultaneously captured corrected speckle contrast images (818). In one embodiment, steps 802-818 may replace steps 332-336 of Figure 3B.

[0309] Thus, an advantage of the method in the present disclosure is that the second wavelength image can be used for both multi-spectral coherent correction (or "dual laser correction") as described with reference to FIG. 8 and for speckle contrast image registration as described with reference to FIGS. 3A and 3B.

[0310] FIG. 9A illustrates a schematic diagram of a motion-compensated speckle contrast image determination based on a weighted average according to one embodiment. 1~N For example, the transformation size may be determined based on the length of the alignment vectors, or a statistical representation, such as the length of the alignment vectors, for example 904. 1~N, or a matrix norm of a matrix representing the transformation. Non-feature-based image registration methods typically provide only a set of global registration parameters, or only a limited set of registration parameters. In such cases, the weights may be derived directly from the registration parameters, rather than from a set of alignment vectors. Optionally, different types of transformations, e.g., translation, rotation, and scaling, may be given different weights. Non-feature-based image registration methods typically provide only a set of global registration parameters, or only a limited set of registration parameters. In such cases, the weights may be derived directly from the registration parameters, rather than from a set of alignment vectors. Optionally, different types of transformations, e.g., translation, rotation, and scaling, may be given different weights.

[0311] The combined speckle contrast image 906 may be a weighted average of the speckle contrast images in a sequence of registered speckle contrast images, each image being weighted with a weighting parameter w′.

[0312] The weight parameter w' is the weight of the alignment vector ||p' i -p i ||, where high displacement corresponds to low weight and vice versa, for example as defined in equation (11) below.

number

[0313] Here, the alignment vector has coordinates (x i ',y i Point P defined by i ' and coordinates (x i ,y i) is the point p i and may be determined by the sum of

[0314] The weighting parameter w' may also be determined based on fine or coarse optical flow parameters that define the optical flow between the first image, typically a reference image, and the second image. For example, the weighting may be inversely correlated with the average optical flow over all pixels in the image or in a region of interest in the image, e.g., as defined in equation (12) below.

number

[0315] where v ij where ' is the optical flow of pixel (i,j) including the x and y components of the optical flow, and W and H are the width and height in pixels of the reference image or reference region of interest, respectively.

[0316] Alternatively, a weighting parameter may be determined for each pixel based on the per-pixel optical flow, for example as defined in equation (13) below.

number

[0317] Instead of determining weighting parameters for each pixel or for the entire image, the weighting parameters can also be determined for a predetermined region of the image, for example a rectangular or triangular mesh, For example, the weighting parameters can be based on the average optical flow within the corresponding predetermined region.

[0318] The weighting parameter is also determined by the speckle contrast value s ij ', preferably the weighting parameter is proportional to the magnitude of the speckle contrast, for example as defined in equation (14) below.

number

[0319] Here, s ij ' is the speckle contrast of the reference image, and W and H are the width and height in pixels of the reference image or the reference region of interest, respectively. The advantage of using speckle contrast is that noise occurring within a small time window for the speckle contrast blurs the speckle contrast. Such noise may be due to, for example, the motion of the imaged object or the camera, or other sources, such as loose fiber connections. The advantage of using weights based on alignment vectors or optical flow is that it may have a higher temporal resolution in the case of LSCI, while weights based on speckle contrast may lag, especially when perfusion is increasing.

[0320] Alternatively, the weighting parameters can be determined based on the speckle contrast per pixel.

number

[0321] Alternatively, the weighting parameters can be determined based on the average speckle contrast within a predefined region, such as a square grid or a triangular mesh. The weighting parameters can also be determined for dynamically determined regions, such as regions determined based on detected motion.

[0322] In one embodiment, various ones of these weights can be combined. For example, the weights can be normalized and added, multiplied, or compared, for example, to select the lowest weight. Alternatively, for example, based on a speckle contrast value, a first weight can be used to filter out images that do not meet a predetermined quality criterion. For example, an image having a decrease in contrast magnitude exceeding a predetermined threshold receives a weight of 0, and all other images receive a weight of 1. Thereafter, weights based on optical flow or displacement can be used to determine a weighted average of the images not removed by filtering.

[0323] The weighted average can be determined by using a buffer of N images Img k and a corresponding buffer of N weight coefficients w k and using a single weight coefficient for each image, where k = 1, 2..., N. For each newly acquired image, the following steps can be performed: 1) Add the new image to the buffer as Img N+1 which can be selected as the reference image; 2) Remove the first weight coefficient w1 and the first image Img1 from the buffer; 3) Apply a geometric transformation to the other images Img2, Img3,..., Img N in the buffer to register them with the reference image Img N+1 . If these images are, for example, previously registered with a previous reference image, the same transformation can be applied to all images Img 2-N . If unregistered images are stored in the buffer, a transformation can be determined and applied separately for each image in the buffer. 4) Calculate the weight coefficient w N+1 as described above and add it to the buffer; 5) Shift the weight coefficients w2~w N+1 for example according to the following equation (16):

Equation

number

number

[0324] Here, i and j are used to index the pixels corresponding to the image.

[0325] In an alternative embodiment, the image buffer may contain only one combined image. In such an embodiment, the N weighting factors w k , where k=1,2...,N corresponds to the N most recent images, and a combined image Img based on the N most recent images. N ', the weighted average can be determined. N+1 For each, the following steps may be taken: 1) Copy the new image to Img N+1 to the buffer as, which may be selected as the reference image; 2) Retrieve a first weighting factor w1 from the buffer; 3) Stored image (Img) N ' and apply a geometric transformation to the reference image Img N+1 and register. 4) The weighting factor w is calculated as explained above. N+1 and add it to the buffer; 5) Weighting coefficients w2~w N+1 normalize, for example according to equation (16) or (17); 6) The stored image Img', for example as defined in equation (19) below: N and the reference image Img N+1 A high-quality combined image (Img) is created by calculating a weighted average of the N+1 ' to calculate:

number

[0326] The advantage of this algorithm is that it is much faster to compute, since only one geometric transformation needs to be applied and the amount of processing steps is smaller. The size of the buffer in which the weighting factor is stored determines how much influence the past image has compared to the influence of the new image. If the buffer is small, the new image will be closer to the present in the final image, whereas if the buffer is large, the new image will be less close to the present and the image with a high weighting factor will be closer to the present.

[0327] As explained above, the weights may be determined for the entire image, for each pixel in an image, or for a predefined or dynamically determined region in an image. Similarly, a single transformation may be applied to each entire image, each pixel may be transformed individually, or the transformation may be determined and applied to a predefined or dynamically determined region. Thus, the weights may be defined as scalars, matrices, or in a different format.

[0328] An advantage of algorithms that use geometric transformations based on, for example, rectangular or triangular meshes and use weights determined for each mesh segment is that such algorithms can be more robust to registration errors while being able to locally correct for noise, e.g. local motion.

[0329] A weight based on the amount of displacement or transformation can be quickly determined for each image, independent of the other images. Images with a large amount of displacement are generally noisier and therefore can be assigned a lower weight, thereby improving the quality of the combined image.

[0330] Alternatively or additionally, a normalized speckle contrast amount or a change in speckle contrast relative to one or more preceding and / or subsequent images in the sequence of a plurality of first speckle contrast images may be determined for each first unprocessed speckle image, in which case the weighted average may be determined using weights based on the determined normalized speckle contrast amount or the determined change in speckle contrast associated with each of the first speckle contrast images.

[0331] Weights based on differences or changes, especially sharp changes, in speckle contrast can indicate image quality. Typically, speckle contrast, and therefore these weights, can be affected by various factors in the overall system, such as the movement of the camera relative to the target area, fiber movement, or other factors that affect the optical path length, loose connections, or varying lighting conditions. Thus, using weights based on speckle contrast, a higher quality combined image can be obtained. Typically, speckle contrast is determined in relative units, so weights can be determined by analyzing a sequence of multiple raw speckle images. Since speckle contrast is inversely correlated with perfusion, perfusion units based on speckle contrast can be used as well.

[0332] Preferably, the images can be normalized so that the relationship between speckle contrast and weight is linear with a constant. In that case, the correction based on speckle contrast can be closer to real-time, since each image can be normalized directly and without reference to the image's time window. Alternatively, incremental averaging can be used.

[0333] FIG. 9B illustrates a schematic diagram of determining a motion-compensated speckle contrast image based on a weighted average, according to one embodiment. In this example, a cyclic image buffer 910 is used. When the image buffer is updated, the oldest image is deleted from the buffer (912), the image in the buffer is transformed using the determined image registration parameters (914), and the new image is added to the buffer (916). The masks are processed similarly. A cyclic mask buffer 920 of the same size as the image buffer contains masks associated with the images in the image buffer. When the mask buffer is updated, the oldest mask is deleted from the buffer (922), the mask in the buffer is transformed using the determined image registration parameters (924), and the new mask associated with the new image is added to the mask buffer (926). The mask in the mask buffer may be blurred (eg, using a combination of Gaussian blur or a box filter) to reduce sensitivity to inaccuracies in the image registration.

[0334] The masks in the updated buffer are then averaged and normalized (928), resulting in a normalized average mask. The average is typically a regular average, but a weighted average could be used as well, e.g., based on motion parameters as described above with reference to FIG. 9A. In this way, pixels in the mask have similar or equal weights as pixels in the weighted average 918 of the speckle contrast image.

[0335] At step 918, a weighted average is determined for the images in the image buffer. This weighted average may be referred to as a temporal filter. The weighted average may use multiple weights. For example, motion-based weights may be used, as described above. The weights may be local (pixel-based), regional, or global (image-based) weights. In addition, mask-based weights may be used. The mask-based weights are typically applied locally, i.e., on a pixel-by-pixel basis.

[0336] Once the normalized averaging mask has been determined (928), a perfusion value may be calculated if the normalized average value is below a predetermined value (assuming that a high mask value indicates a low confidence score), e.g., below 0.2. This indicates that in most input images, the pixel values ​​are deemed sufficiently reliable. If the normalized average value is above a predetermined value, the corresponding pixel in the combined image may be given, e.g., an error value, an interpolated value, or no value, as described above.

[0337] The pixel for which the perfusion value is calculated may then be determined by calculating a weighted average, where each pixel in the input image in the image buffer is given a weight based on the associated mask in the mask buffer, in some cases this is a binary weight, in other cases multi-valued weights may be used.

[0338] This results in a combined image 930, which may be displayed or used for further processing. In some embodiments, the combined image is displayed, for example, as an overlay on top of a different image (which may be referred to as an underlying image), such as the most recent image in the image buffer or an associated white light image. In some cases, the underlying image may be modified based on a mask associated with it. For example, the intensity of specular reflections in the underlying image may be reduced by decreasing the corresponding pixel values. This may result in a less noticeable image.

[0339] Similar implementations can be used for other temporal filters, e.g. decaying buffers. In general, the processing of masks in the mask buffer needs to be similar to the processing of images in the image buffer.

[0340] 10 is a block diagram illustrating an exemplary data processing system as described in this disclosure. The data processing system 1000 may include at least one processor 1002 coupled to memory elements 1004 through a system bus 1006. As such, the data processing system may store program code in the memory elements 1004. Further, the processor 1002 may execute the program code accessed from the memory elements 1004 via the system bus 1006. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be appreciated that the data processing system 1000 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0341] The memory elements 1004 may include one or more physical memory devices, such as a local memory 1008 and one or more mass storage devices 1010. The local memory may refer to a random access memory or other non-persistent memory device typically used during the actual execution of the program code. The mass storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1000 may also include one or more cache memories (not shown) that provide temporary storage of at least some of the program code to reduce the number of times the program code must be retrieved from the mass storage device 1010 during execution.

[0342] Input / output (I / O) devices depicted as key devices 1012 and output devices 1014 may optionally be connected to the data processing system. Examples of key devices may include, but are not limited to, for example, a keyboard, a pointing device, such as a mouse, and the like. Examples of output devices may include, but are not limited to, for example, a monitor or display, speakers, and the like. The key devices and / or output devices may be connected to the data processing system directly or through an intervening I / O controller. A network adapter 1016 may also be connected to the data processing system to enable it to become connected to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data transmitted to the system, device, and / or network by the system, device, and / or network, and a data transmitter for transmitting data to the system, device, and / or network. An operating modem, a cable operating modem, and an Ethernet card are examples of different types of network adapters that may be used with the data processing system 1000.

[0343] 10, memory element 1004 may store an application 1018. It should be appreciated that data processing system 1000 may further execute an operating system (not shown) that may facilitate execution of the application. Applications implemented in the form of executable program code may be executed by data processing system 1000, for example by processor 1002. In response to executing an application, the data processing system may be configured to perform one or more operations described in further detail herein.

[0344] In one aspect, for example, data processing system 1000 may represent a client data processing system, in which case application 1018 may represent a client application that, when executed, configures data processing system 1000 to perform various functions described herein with reference to a "client." Examples of clients include, but are not limited to, personal computers, portable computers, mobile phones, and the like.

[0345] In another aspect, the data processing system may represent a server. For example, the data processing system may represent an (HTTP) server, in which case the application 1018, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function as referenced herein.

[0346] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to encompass the plural forms unless the context clearly indicates otherwise. It will be further understood that the words "comprises" and / or "comprising", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0347] The equivalents of corresponding structures, materials, acts, and all means or steps plus function elements in the following claims are intended to encompass any structure, material, or act for performing the functions in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the invention in the disclosed form. Many changes and modifications will become apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been chosen and described in order to best explain the principles and practical application of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments with various modifications suitable for the particular use contemplated.

Claims

1. 1. A method of motion compensated laser speckle contrast imaging, comprising: exposing a target area to coherent first light at a first wavelength, wherein the target area comprises biological tissue; capturing at least one sequence of a plurality of images, wherein the at least one sequence of a plurality of images includes a plurality of first speckle images, the plurality of first speckle images being captured during exposure with the first light; determining one or more registration parameters of an image registration algorithm to register the first speckle images to one another based on the captured sequence of at least one of the plurality of images, wherein determining the one or more registration parameters includes determining transformed images by applying a transformation to the images in the at least one sequence of the plurality of images, and determining the registration parameters based on a comparison of the transformed images; determining a registered spatial speckle contrast image, wherein determining the registered spatial speckle contrast image comprises: determining a plurality of registered first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a plurality of registered spatial speckle contrast images based on the registered plurality of first speckle images; or determining a plurality of first spatial speckle contrast images based on the plurality of first speckle images, and determining a plurality of registered spatial speckle contrast images by registering the plurality of first spatial speckle contrast images based on the one or more registration parameters and the image registration algorithm; and determining a combined spatial speckle contrast image, wherein determining the combined spatial speckle contrast image comprises calculating an average of the registered spatial first speckle images, wherein a weight of an image, a weight of an image region, or a weight of a pixel in the combined spatial speckle contrast image is based on one or more of the registration parameters or differences or changes in speckle contrast; The method comprises:

2. 2. The method of claim 1, wherein determining the one or more registration parameters is based on an image selected from the plurality of first speckle images, and / or an image selected from an image associated with the plurality of first speckle images, and / or an image selected from the plurality of first speckle images or an image associated with the plurality of first speckle images.

3. The method of claim 2 , wherein the registration parameters are based on a similarity measure of pixel values ​​in one or more groups of pixels in each of the image sequences.

4. The method of claim 1 or 2, wherein the transforming includes one or more of a Fourier transform, a Mellin transform, a Laplace transform, a Radon transform, and a log-polar transform.

5. The transformation is a transformation into the frequency domain, and the comparison of the transformed images is determining a cross-correlation of the transformed images; determining a transform of the cross-correlation to the spatial domain; and determining a peak in the cross-correlation in the spatial domain; Including, 3. The method according to claim 1 or 2.

6. the transformation is a log-polar transformation, wherein comparing the transformed images includes determining a shift of the transformed images relative to one another; and wherein determining the registration parameters includes determining a rotation and / or a scaling based on the determined shift.

3. The method according to claim 1 or 2.

7. 1. A method of motion compensated laser speckle contrast imaging, comprising: exposing a target area to coherent first light at a first wavelength, wherein the target area comprises biological tissue; capturing at least one sequence of a plurality of images, wherein the at least one sequence of a plurality of images includes a plurality of first speckle images, the plurality of first speckle images being captured during exposure with the first light; determining one or more registration parameters of an image registration algorithm to register the plurality of first speckle images with one another based on the captured sequence of at least one of the plurality of images, wherein determining the one or more registration parameters includes: selecting one or more first groups of pixels in a first image of the at least one sequence of the plurality of images based on pixel coordinates; selecting one or more second pixel groups based on pixel coordinates in a second image of the at least one sequence of the plurality of images, wherein the one or more first pixel groups have a different size than the one or more second pixel groups; and determining the registration parameters based on a similarity measure of pixel values ​​in the one or more first pixel groups and the one or more second pixel groups; Includes; determining a registered spatial speckle contrast image, wherein determining the registered spatial speckle contrast image comprises: determining a plurality of registered first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a plurality of registered spatial speckle contrast images based on the registered plurality of first speckle images; or determining a plurality of first spatial speckle contrast images based on the plurality of first speckle images, and determining a plurality of registered spatial speckle contrast images by registering the plurality of first spatial speckle contrast images based on the one or more registration parameters and the image registration algorithm; and determining a combined spatial speckle contrast image, wherein determining the combined spatial speckle contrast image comprises calculating an average of the registered spatial first speckle images, wherein a weight of an image, a weight of an image region, or a weight of a pixel in the combined spatial speckle contrast image is based on one or more of the registration parameters or differences or changes in speckle contrast; The method comprises:

8. determining a plurality of masks for the plurality of first speckle images, each of the plurality of masks being associated with a respective first speckle image, and each of the plurality of masks associating a confidence score with one or more pixels in the associated first speckle image; determining a plurality of registered masks by registering the plurality of masks based on the one or more registration parameters and the image registration algorithm; and determining the combined speckle contrast image based on the plurality of registered masks; The method of claim 1 or 7, further comprising:

9. determining the plurality of masks includes identifying deviant input pixel values ​​and / or deviant speckle contrast values, values ​​above a predetermined upper absolute or relative threshold, or values ​​below a predetermined lower absolute or relative threshold; and / or determining the plurality of masks includes identifying pixels that do not represent biological tissue based on an image recognition algorithm; The method of claim 8.

10. The method of claim 8, wherein the masked pixels have weights based on the confidence scores associated with the pixels.

11. The method comprises: further comprising exposing the target area to a second light of one or more second wavelengths, wherein the exposure to the second light alternates with or is simultaneous with the exposure to the first light; the at least one sequence of images includes a sequence of a plurality of second images, wherein the plurality of second images are captured during exposure with the second light; and the one or more registration parameters are based on the sequence of the plurality of second images or on a plurality of images derived from the plurality of second images, wherein each of the plurality of second images or each of a plurality of images derived from the plurality of second images is associated with a first speckle image; The method according to claim 1 or 7.

12. The method of claim 1 or 7, wherein the one or more pixel groups represent a predetermined feature within the plurality of images.

13. 10. The method of claim 1 or 7, wherein a group of pixels in a first image from at least one sequence of said plurality of images is smaller than a group of pixels in a second image from at least one sequence of said plurality of images.

14. Determining one or more registration parameters includes: determining a plurality of associated pixel groups based on the similarity measure, wherein each pixel group belongs to a different image from the sequence of images; determining a plurality of alignment vectors based on positions of the pixel groups relative to individual images from the sequence of images, where the alignment vectors represent movement of the target area relative to the image sensor; and determining the registration parameters based on the plurality of alignment vectors; Including, The method according to claim 1 or 7.

15. The method comprises: Dividing each of the plurality of first speckle images or each of the plurality of first speckle contrast images and each image in the sequence of the plurality of images into a plurality of regions. and wherein: Determining the registration parameters includes determining registration parameters for each region; and determining a sequence of registered first speckle images or first speckle contrast images includes registering each region of the first speckle images or first speckle contrast images based on a transformation based on the corresponding region in the second image; The method according to claim 1 or 7.

16. 10. The method of claim 1 or 7, wherein the target area includes a perfused organ and / or includes one or more blood vessels and / or lymphatic vessels, and the method further comprises calculating a perfusion intensity based on the combined speckle image.

17. A computation module for a laser speckle imaging system, the computation module comprising: a computer-readable storage medium having at least a portion of a program embedded in the computer-readable storage medium; and a processor coupled to the computer-readable storage medium, wherein in response to executing the computer-readable storage code, the processor is configured to perform executable operations, including: receiving at least one sequence of a plurality of images, wherein the at least one sequence of a plurality of images includes a plurality of first speckle images captured during exposure of a target area to coherent first light at a first wavelength, the target area including biological tissue; determining, based on the captured at least one sequence of the plurality of images, one or more registration parameters of an image registration algorithm for registering the plurality of first speckle images with one another; wherein determining the one or more registration parameters includes determining transformed images by applying a transformation to the images in at least one sequence of images, and determining the registration parameters based on a comparison of the transformed images; or wherein determining the one or more registration parameters comprises: selecting one or more first groups of pixels in a first image of the at least one sequence of the plurality of images based on pixel coordinates; selecting one or more second pixel groups based on pixel coordinates in a second image of the at least one sequence of the plurality of images, wherein the one or more first pixel groups have a different size than the one or more second pixel groups; and determining the registration parameters based on a similarity measure of pixel values ​​in the one or more first pixel groups and the one or more second pixel groups; and determining a registered spatial speckle contrast image, wherein determining the registered spatial speckle contrast image comprises: determining a plurality of registered first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a plurality of registered spatial speckle contrast images based on the registered plurality of first speckle images; or determining a plurality of first spatial speckle contrast images based on the plurality of first speckle images, and determining a plurality of registered spatial speckle contrast images by registering the plurality of first spatial speckle contrast images based on the one or more registration parameters and the image registration algorithm; and determining a combined spatial speckle contrast image, wherein determining the combined spatial speckle contrast image comprises calculating an average of the registered spatial first speckle images, wherein a weight of an image, a weight of an image region, or a weight of a pixel in the combined spatial speckle contrast image is based on one or more of the registration parameters or differences or changes in speckle contrast; Including, The calculation module.

18. the at least one sequence of multiple images includes a sequence of multiple second images, wherein the multiple second images are captured during exposure to second light, the second light having one or more second wavelengths, and the exposure to the second light alternates with the exposure to the first light or is simultaneous with the exposure to the first light; and the registration parameters are determined based on a sequence of the plurality of second images, each of the plurality of second images being associated with a first speckle image; 18. A calculation module according to claim 17.

19. 1. A hardware module for an imaging device, the hardware module comprising: a first light source for exposing a target area to coherent first light of a first wavelength, wherein the target area comprises biological tissue; at least one image sensor system having one or more image sensors for capturing at least one sequence of a plurality of images, wherein the at least one sequence of a plurality of images includes a plurality of first speckle images, the plurality of first speckle images being captured during exposure with the first light; and Calculation module according to claim 17 The hardware module comprises:

20. the hardware module further comprising a second light source for illuminating the target area with light of at least a second wavelength, different from the first wavelength, simultaneously or alternately with the first light source; the at least one image sensor is further configured to capture a sequence of a plurality of second images, wherein the plurality of second images are captured during exposure with the second light; and the registration parameters are based on a sequence of the plurality of second images, each of the plurality of second images being associated with a first speckle image; 20. The hardware module of claim 19.

21. 20. A medical imaging device comprising the hardware module of claim 19, wherein the medical imaging device is one of an endoscope, a laparoscope, a surgical robot, a handheld laser speckle contrast imaging device, or an open surgery laser speckle contrast imaging system.

22. A non-transitory computer-readable storage medium storing a computer program or suite of computer programs comprising at least one software code portion, the at least one software code portion performing the following when executed on a computer system: receiving at least one sequence of a plurality of images, wherein the at least one sequence of a plurality of images includes a plurality of first speckle images captured during exposure of a target area to coherent first light at a first wavelength, the target area including biological tissue; determining, based on the captured at least one sequence of the plurality of images, one or more registration parameters of an image registration algorithm for registering the plurality of first speckle images with one another; wherein determining the one or more registration parameters includes determining transformed images by applying a transformation to the images in at least one sequence of images, and determining the registration parameters based on a comparison of the transformed images; or wherein determining the one or more registration parameters comprises: selecting one or more first groups of pixels in a first image of the at least one sequence of the plurality of images based on pixel coordinates; selecting one or more second pixel groups based on pixel coordinates in a second image of the at least one sequence of the plurality of images, wherein the one or more first pixel groups have a different size than the one or more second pixel groups; and determining the registration parameters based on a similarity measure of pixel values ​​in the one or more first pixel groups and the one or more second pixel groups; and determining a registered spatial speckle contrast image, wherein determining the registered spatial speckle contrast image comprises: determining a plurality of registered first speckle images by registering the plurality of first speckle images based on the one or more registration parameters and the image registration algorithm, and determining a plurality of registered spatial speckle contrast images based on the registered plurality of first speckle images; or determining a plurality of first spatial speckle contrast images based on the plurality of first speckle images, and determining a plurality of registered spatial speckle contrast images by registering the plurality of first spatial speckle contrast images based on the one or more registration parameters and the image registration algorithm; and determining a combined spatial speckle contrast image, wherein determining the combined spatial speckle contrast image comprises calculating an average of the registered spatial first speckle images, wherein a weight of an image, a weight of an image region, or a weight of a pixel in the combined spatial speckle contrast image is based on one or more of the registration parameters or differences or changes in speckle contrast; Including, configured to perform the method steps of The non-transitory computer-readable storage medium.

Images