Motion-compensated laser speckle contrast imaging

Abstract

Methods and systems for laser speckle contrast imaging with motion correction and motion compensation are disclosed. The method includes exposing a target region to coherent first light of a first wavelength, the target region including living tissue, and capturing at least one image sequence including first speckle images, and determining one or more transformation parameters of an image registration algorithm for registering the first speckle images to each other. The transformation parameters can be based on a similarity measure of pixel values of a group of pixels in a plurality of images of the at least one image sequence, the images in the plurality of images being selected from or associated with the first speckle images.

Description

Technical Field

[0001] This disclosure relates to motion-compensated laser speckle contrast imaging, and in particular, although not exclusively, to methods and systems for motion-compensated laser speckle contrast imaging, motion compensation modules in laser speckle contrast imaging systems and methods, and computer program products that enable a computer system to perform the methods. Background Technology

[0002] Laser speckle contrast imaging (LSCI) provides a rapid, full-field, in vivo imaging method for determining two-dimensional (2D) perfusion maps of living biological tissues. Perfusion can serve as an indicator of tissue viability, thus providing valuable information for diagnosis and surgery. For example, in intestinal surgery, selecting a highly perfused interventional site can reduce anastomotic leakage.

[0003] The principle of LSCI is as follows: due to the difference in optical path length, backscattered light from tissue irradiated with a coherent laser forms a random interference pattern on the detector. The resulting interference pattern is called a speckle pattern and can be imaged in real time using a digital camera. The movement of particles within the tissue causes fluctuations in this speckle pattern, resulting in speckle blurring of the perfused areas in the image.

[0004] For example, if the ripples are caused by the movement of red blood cells, then the aforementioned blurring may be related to blood flow. Thus, blood perfusion can be imaged in living tissue in a relatively simple manner. W. Heeman et al. describe a recent example of a clinical perfusion imaging protocol using LSCI in the following review article: Clinical applications of laserspeckle contrast imaging: a review, J. Biomed. Opt. 24:8 (2019). Perfusions of other bodily fluids, such as lymphatic perfusion, can also be imaged in a similar manner.

[0005] However, LSCI is highly sensitive to any type of motion. Blurring can be caused not only by the movement of blood flow, but also by any other type of motion, such as tissue movement caused by breathing, heartbeat, muscle contraction, or camera (especially handheld camera) movement.

[0006] In many medical applications, such as diagnostics and surgery, it is desirable for LSCI systems to generate accurate, high-resolution blood flow images, particularly microcirculation images, in real time, with significantly reduced motion artifacts. Therefore, measures need to be taken to minimize motion artifacts during the processing of raw speckle images in order to obtain accurate, high-resolution perfusion images. This can improve the identification of well-perfused and poorly perfused areas, thereby enhancing diagnostic and therapeutic outcomes.

[0007] Various methods for reducing motion artifacts in speckle images are known in the prior art. For example, WO2020 / 045015A1 discloses a laser speckle contrast imaging system capable of capturing near-infrared speckle images and white light images of an imaging target. Simple motion detection schemes may include using reference markers on the imaging target, tracking feature points in a visible light image, or detecting changes in the speckle shape in the speckle image. These motion detection methods are used to determine a global motion vector, which indicates the amount of movement of the image target between two subsequent images. The speckle contrast image can be generated based on the speckle image, and the amount of motion can be corrected based on the motion vector.

[0008] For example, motion is even more pronounced in handheld LSCI systems compared to tripod-supported systems. For instance, Lertsakdadet et al. described a motion compensation scheme for laser speckle imaging using reference markers attached to the tissue to be imaged in the following article: Correcting for motion artifacts in handheld laser speckle images, Journal of biomedical optics 23(2), March 2018. In many applications, using markers is not feasible.

[0009] Similarly, P. Miao et al. describe a method for generating high-resolution LSCI images by registering raw speckle images using a convolutional filtering and correlation interpolation scheme in the following article: High resolution cerebral blood flow imaging by registered laser speckle contrast analysis, IEEE transactions on bio-medicalengineering 57(5):1152–1157. The registered images were then retrospectively analyzed using temporal laser speckle contrast analysis. However, this registration of raw speckle images requires significant computational resources and is therefore unsuitable for precise real-time imaging applications.

[0010] Therefore, as can be seen from the above, there is a need in the art to improve motion-compensated laser speckle contrast imaging schemes. In particular, there is a need in the art to improve methods and systems for laser speckle contrast imaging that allow for real-time, robust, label-free, high-resolution perfusion imaging, especially microcirculation imaging, in which motion artifacts are substantially eliminated or at least reduced. Summary of the Invention

[0011] Those skilled in the art will understand that aspects of the present invention can be embodied as a system, method, or computer program product. Therefore, aspects of the present invention can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein as “circuit,” “module,” or “system.” The functionality described in this disclosure can be implemented as an algorithm executed by a computer’s microprocessor. Furthermore, aspects of the present invention can take the form of a computer program product embodied in one or more computer-readable media on which computer-readable program code is embodied (e.g., stored).

[0012] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media include the following: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0013] Computer-readable signal media may include propagated data signals embodying computer-readable program code, for example, in baseband or as part of a carrier wave. The propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium; it is not a computer-readable storage medium, and it may communicate, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0014] The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic, cable, RF, etc., or any suitable combination thereof. The computer program code used to perform the operations of various aspects of this invention may be written in any combination of one or more programming languages, including functional or object-oriented programming languages ​​such as Java™, Scala, C++, Python, etc., and conventional procedural programming languages ​​such as “C” or similar programming languages. The program code may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer, a server, or a virtualization server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the aforementioned connection may be to an external computer (e.g., via the Internet through an Internet service provider).

[0015] The following description of various aspects of the present invention is based on flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor, particularly to a microprocessor or central processing unit (CPU) or graphics processing unit (GPU) of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, to generate a machine, such that the instructions, when executed by a computer processor, other programmable data processing apparatus, or other device, create means for implementing the functions / actions specified by one or more blocks in the flowchart illustrations and / or block diagrams.

[0016] The aforementioned computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture, the article of manufacture including instructions that implement the function / action specified by one or more blocks in a flowchart and / or block diagram.

[0017] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide a process for implementing the function / action specified by one or more blocks in a flowchart and / or block diagram.

[0018] The flowcharts and block diagrams in the figures illustrate the structure, function, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may appear in the order shown in the figures. For example, in fact, two blocks shown consecutively may be executed substantially simultaneously, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs the specified function or action, or by a combination of dedicated hardware and computer instructions.

[0019] The purpose of the embodiments disclosed herein is to reduce or eliminate at least one of the disadvantages known in the prior art.

[0020] In a first aspect, the present invention relates to a method for motion-compensated laser speckle contrast imaging. The method includes: exposing a target region to coherent first light of a first wavelength, the target region comprising living tissue, and capturing at least one image sequence. The at least one image sequence includes a first speckle image captured during exposure to the first light. The method further includes: determining one or more transformation parameters for an image registration algorithm to register the first speckle images to each other. The transformation parameters may be based on a similarity measure of pixel values ​​of groups of pixels in a plurality of images of the at least one image sequence, the images being selected from or associated with the first speckle image. The method further includes: determining a registered first speckle image by registering the first speckle image based on the one or more transformation parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or determining a first speckle contrast image based on the first speckle image, determining a registered speckle contrast image by registering the first speckle contrast image based on the one or more transformation parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast image.

[0021] Therefore, sequences of speckle images or sequences of speckle contrast images can be registered or aligned without the use of markers. The registered images can then be combined into a combined speckle contrast image with high resolution and high accuracy. The combined image can include information from multiple speckle images, such as pixel-wise averaging, preferably a weighted average, or the combined image can be, for example, the single most reliable image within a moving window of the images, such as the image with the least transformation relative to subsequent images in the image sequence (i.e., the image closest to the identity transformation using a suitable metric).

[0022] Using the above method, the image is less sensitive to noise caused by motion. Motion can be caused by, for example, the movement of a camera (especially in a handheld system) or by the movement of a patient (e.g., caused by muscle contraction). Therefore, the embodiments in this disclosure can realize or improve speckle contrast imaging in a wide range of applications, including, for example, perfusion imaging of the large intestine (which requires the camera to move along the entire surface to be imaged), or perfusion imaging of skin burns (where the patient may be unable to remain still due to pain).

[0023] One advantage of this method is that it can be executed (essentially) in real time using generally available hardware. In some embodiments, for example, there may be a small delay based on multiple frames (e.g., 20 frames, or 20 frames of sufficient quality), a fixed amount of time (e.g., 1 second), or a time based on physiological characteristics (e.g., the time for one or two heartbeats, or the time for one or two breaths). This delay generally does not impair clinical use. Physiological characteristics may be based on image analysis of speckle images, speckle contrast images, and / or images from multiple images; or physiological characteristics may be based on predetermined constants of knowledge about physiological phenomena, or physiological characteristics may be based on external input, for example, from a heart rate monitor.

[0024] Another advantage of this method is that it does not require placing reference markers in the field of view. This is especially important for areas where it is undesirable to place reference markers, such as when imaging internal organs, burn tissue, or brain tissue, or for relatively large image targets, where otherwise a large number of reference markers or repeated marker replacements would be necessary.

[0025] Because this method corrects and / or compensates for the motion of the target region relative to the camera, more reliable perfusion images can be created. In some cases, reliable images can be obtained faster or more easily, for example, when imaging targets with irregular motion, where the operator would otherwise have to wait for periods of minimal motion to acquire an image. For example, motion correction may include transforming one or more images based on detected significant motion. Motion compensation may include combining multiple images to improve the signal-to-noise ratio.

[0026] By registering a first speckle image or a first speckle contrast image, multiple first speckle images from a sequence of first speckle images can be used to determine a combined speckle contrast image, which has improved contrast and / or spatial resolution compared to individual first speckle contrast images. The number of registered speckle images to be combined may depend on clinical requirements and / or the quality parameters of the first speckle images.

[0027] Speckle contrast images can be computed based on unregistered and therefore untransformed speckle images. Subsequently, the speckle contrast images can be transformed and then combined. Transformations, especially those more general than simple translation or rotation, can distort the speckle pattern or parts thereof; for example, the speckles can be enlarged, reduced, or deformed. Therefore, computed speckle contrast images based on untransformed speckle images prevents the introduction of noise due to transformation into the speckle contrast images.

[0028] Optionally, the speckle image can be registered and thus transformed first, and a speckle contrast image can then be calculated. This allows for the calculation of temporal speckle contrast or temporal-spatial speckle contrast based on two or more registered speckle images. Using temporal speckle contrast or spatial-temporal speckle contrast can result in higher spatial resolution. In this embodiment, the image is preferably registered with sub-pixel precision.

[0029] Another advantage of this embodiment is that only one image sequence is needed: the same image sequence can be used to determine the transform parameters for image registration, to determine the average weights, and to determine the combined laser speckle contrast image or perfusion image. However, in other embodiments, a second image sequence can be used to determine the transform parameters for image registration and to determine the average weights.

[0030] In one embodiment, the method further includes exposing the target region to one or more second wavelengths of second light, preferably coherent light of a second wavelength or light comprising a plurality of second wavelengths of the visible spectrum, wherein exposure to the second light is alternated with or simultaneous with exposure to the first light. In this embodiment, the at least one image sequence includes a second image sequence, the second images being captured during exposure to the second light; and the plurality of images are selected from the second image sequence, each of the second images being associated with a first speckle image.

[0031] Therefore, in one embodiment, the laser speckle contrast imaging method includes alternately or simultaneously exposing a target region to coherent first light of a first wavelength and one or more second light of a second wavelength, wherein the second light is preferably coherent light of a second wavelength or preferably light comprising a plurality of second wavelengths of the visible spectrum, and the target region preferably comprises living tissue. The method may further include: capturing a sequence of first speckle images during exposure with the first light, capturing a sequence of second images during exposure with the second light, wherein the speckle images of the first speckle image sequence are associated with the images of the second image sequence. One or more transformation parameters of a registration algorithm for registering at least a portion of the first speckle image sequence can be determined based on a similarity measure of pixel values ​​of pixel groups in at least a portion of the second image sequence associated with the first speckle image. The method may further include: registering at least a portion of the first speckle image sequence based on one or more transformation parameters and a registration algorithm to determine a registered first speckle image, and determining a combined speckle contrast image based on the registered first speckle image. Optionally, the method may include: determining a first speckle contrast image based on at least a portion of a first speckle image sequence; registering the first speckle contrast image based on one or more transform parameters and a registration algorithm to determine a registered speckle contrast image; and determining a combined speckle contrast image based on the registered first speckle contrast image.

[0032] In the first embodiment described above, the first light and the second light are identical light having the same wavelength, and the first speckle image sequence and the second image sequence are identical image sequences. However, in an advantageous embodiment, at least one of one or more second wavelengths differs from the first wavelength; therefore, the second image differs from the first speckle image. The first speckle image and the associated second image may also be different portions of a single image. The images in the image sequence may be frames in a video stream or frames in a multi-frame snapshot. In this disclosure, the second image may also be referred to as a corrected image. Typically, at least one group of pixels from each of at least a portion of the second image sequence is used to determine one or more transform parameters.

[0033] Therefore, the acquisition of the first wavelength and the first speckle image sequence can be optimized for speckle contrast imaging based on the number of samples to be measured (e.g., optimized exposure time); while the acquisition of one or more second wavelengths and the second image sequence can be optimized to obtain images that are easy to register accurately, for example by ensuring high contrast between anatomical features such as blood vessels and normal tissue. A typical combination could be a speckle image acquired using infrared light and a second image acquired using white light, or a speckle image acquired using red light and a second image acquired using green or blue light; of course, other combinations are also possible.

[0034] In one embodiment, the plurality of pixel groups may represent predetermined features in a plurality of images, the predetermined features preferably being associated with a plurality of objects (preferably anatomical structures) in the target region. Pixel groups may be selected using a feature detection algorithm.

[0035] These features can be features related to physical objects in the target region, such as features related to blood vessels, rather than image features that are not directly related to the object (e.g., overexposed image portions or speckle). Predetermined features can be determined, for example, by belonging to a class of features, such as corners or regions with large intensity differences. They can be further determined by, for example, quality indices, constraints on the distances between features, etc. In some embodiments, the neighborhood of one or more determined features can be used to determine the displacement vector and / or the transformation.

[0036] Determining displacement based on a relatively small number of features (compared to the total number of pixels) can significantly reduce computation time while still yielding accurate results. This is especially true for relatively simple motions (e.g., the displacement of an entire target area due to camera movement).

[0037] In one embodiment, the method may further include filtering the plurality of images with a filter adapted to increase the probability that a group of pixels represents a feature corresponding to an anatomical feature. For example, the filter may identify overexposed and / or underexposed areas and / or other image artifacts, and a mask may be created based on these areas or artifacts. Thus, the identification of features associated with these areas or artifacts can be prevented.

[0038] In one embodiment, determining the transformation parameters based on a similarity metric of pixel values ​​for pixel groups may include: for each pixel group in the images from the plurality of images, determining a convolution or cross-correlation with at least a portion of different images from the plurality of images. The method may further include determining a polynomial expansion to model pixel values ​​in the pixel neighborhood of the plurality of images, comparing the expansion coefficients, and determining a displacement vector based on the comparison.

[0039] In one embodiment, determining one or more transformation parameters may include: determining a plurality of associated pixel groups based on the similarity metric, each pixel group belonging to a different image from the plurality of images; determining a plurality of displacement vectors based on the position of the pixel groups relative to the respective images in the plurality of images, the displacement vectors representing the motion of the target region relative to an image sensor; and determining the transformation parameters based on the plurality of displacement vectors.

[0040] By determining the displacement vectors of multiple features or corresponding feature pairs, arbitrary movement of the camera relative to the imaging target can be determined and corrected. The displacement vectors can be used to determine, for example, affine transformations, projective transformations, or homography transformations, which can correct for image translation, rotation, scaling, and cropping based solely on image data. Therefore, this method does not require information such as the distance between the camera and the target, the angle of incidence, etc.

[0041] The determination of the displacement vector and / or one or more transformations can be based on optical flow parameters, which are determined using a suitable optical flow algorithm (e.g., dense optical flow algorithm or sparse optical flow algorithm).

[0042] In one embodiment, determining the combined speckle contrast image may further include: calculating the average value of the registered first speckle image and the average value of the registered first speckle contrast image, wherein the average value is preferably a weighted average value, and the image weights are preferably based on the transform parameters or on the relative magnitude of the speckle contrast. Optionally, the combined speckle image or speckle contrast image may include: filtering at least a portion of the first speckle image and at least a portion of the first speckle contrast image, respectively, using, for example, a median filter, a minimum or maximum filter, etc. The weights of the weighted average may be based on, for example, a quantity obtained from the speckle contrast or a quantity obtained from the displacement vector.

[0043] In one embodiment, calculating the combined laser speckle contrast image may include: calculating the average value of the registered speckle image sequence, which is preferably a weighted average value.

[0044] In one embodiment, the method may further include, for each first speckle image or each first speckle contrast image associated with an image among the plurality of images, determining a transformation magnitude associated with each first speckle image or each first speckle contrast image based on a plurality of displacement vectors, preferably based on the lengths of the plurality of displacement vectors, and / or based on parameters of a defined transformation. A weighted average may be determined using weights based on the determined transformation magnitudes associated with each first speckle contrast image, preferably the weights being negatively correlated with the determined transformation magnitudes.

[0045] Weights based on the magnitude or amount of displacement, or on the magnitude or amount of transformation, can be quickly determined for each image independently of other images. For example, the transformation magnitude can be based on the norm of the matrix representing the transformation, or the norm of the matrix representing the difference between the transformation and the identity transformation. Alternatively, the transformation magnitude can be based on statistical measures of the displacement vectors, such as the mean, median, nth percentile, or maximum displacement vector length. Images with large displacements are often noisier and can therefore be assigned smaller weights, thus improving the quality of the combined images.

[0046] In one embodiment, the method may further include, for each first speckle image, determining a normalized amount or a change in speckle contrast relative to one or more previous and / or subsequent images in the first speckle contrast image sequence. A weighted average may be determined using weights based on the determined normalized amount or determined change in speckle contrast associated with each of the first speckle contrast images. Alternatively or additionally, the weights may be determined based on the normalized amount or change in speckle contrast of a second speckle contrast image.

[0047] Image quality can be indicated by weights based on differences or variations (especially abrupt changes) in speckle contrast. Typically, speckle contrast, and therefore these weights, can be influenced by various factors throughout the system, such as camera motion relative to the target area, fiber motion, or other factors affecting optical path length or causing fluctuations in illumination. Therefore, using speckle contrast-based weights can yield higher-quality composite images. Typically, speckle contrast is determined in arbitrary units, so weights can be determined by analyzing a sequence of speckle images. Since speckle contrast is negatively correlated with perfusion, perfusion units based on speckle contrast can be used similarly.

[0048] In one embodiment, the algorithm can be applied to a predefined region of interest (ROI) within the camera's field of view. This ROI can be determined by the user or can be predetermined. For example, the outer boundaries of the image can be ignored and / or hidden from the view, e.g., to prevent the transformed image boundaries from being visible. Applying the algorithm only to a portion of the image, or only a portion of the algorithm, may be faster. The ROI can be transformed based on the determined transformation. In different embodiments, the algorithm can be applied to the entire image.

[0049] In one embodiment, the multiple images may be a sequence of first speckle images. In this embodiment, the first wavelength is preferably a wavelength in the green or blue portion of the electromagnetic spectrum. In this way, a balance can be struck between good speckle signal and good visual distinguishability (i.e., high contrast of anatomical features, which will be distinguishable from high speckle contrast), which is advantageous for feature identification. Therefore, no preprocessing step is required to increase the contrast of the speckle images.

[0050] Alternatively, a first wavelength in the red portion of the electromagnetic spectrum (preferably in the range of 600–700 nm, more preferably in the range of 620–660 nm) can be used, or a first wavelength in the infrared portion of the electromagnetic spectrum, preferably in the range of 700–1200 nm. Depending on the tissue type and imaging parameters (such as exposure time), visual distinctiveness may be sufficient to adequately identify the features. Since red and infrared light are primarily reflected by red blood cells, these wavelengths result in speckle patterns with relatively high intensity, making them well-suited for speckle contrast imaging of blood flow.

[0051] In these embodiments, the first speckle image and the images from multiple images are the same image, and the images can be acquired using a relatively simple system that requires only a single light source and a single camera.

[0052] In one embodiment, the light of at least a second wavelength can be light of at least a second wavelength different from the first wavelength. The light of at least a second wavelength is preferably coherent light of a predetermined second wavelength, preferably in the green or blue portion of the electromagnetic spectrum, preferably in the range of 380–590 nm, more preferably in the range of 470–570 nm, and even more preferably in the range of 520–560 nm. Blue light, or especially green light, may result in high contrast or visual distinctiveness because it is absorbed more in blood vessels than in normal tissue. Therefore, features associated with blood vessels, such as edges or corners, can be used to determine transformation parameters.

[0053] Since the first speckle image inherently contains noise (in terms of imaging anatomical features), it is preferable to use a second image based on a different wavelength to determine the displacement vector. Thus, the first speckle image can be acquired based on light selected to optimize the speckle contrast signal, while the second image can be acquired based on light selected to optimize visual discriminability. This system is particularly advantageous for imaging tissues with relatively deep blood perfusion (e.g., skin). In such tissues, most green or blue light does not penetrate deeply enough to interact with blood cells, resulting in a relatively noise-free image.

[0054] In one embodiment, the first wavelength is preferably a wavelength in the red portion of the electromagnetic spectrum (preferably in the range of 600–700 nm, more preferably in the range of 620–660 nm), or a wavelength in the infrared portion of the electromagnetic spectrum (preferably in the range of 700–1200 nm). Red and infrared light, primarily reflected by red blood cells, are well-suited for speckle contrast imaging of blood flow. Infrared light has a greater depth of penetration than red light. Red light may be easier to integrate into existing systems, for example, by using the red channel of an RGB camera to acquire speckle images.

[0055] Preferably, the first wavelength is selected based on what the fluid of interest scatters or reflects; for example, red or near-infrared light can be used to image blood in blood vessels. Preferably, the first wavelength can be selected based on the desired depth of penetration. Light with a relatively high depth of penetration allows the light scattered by the fluid of interest to be detected with a sufficient signal-to-noise ratio, even at some depth in the imaged tissue.

[0056] Preferably, a second wavelength is selected to provide an image with high visual discriminative power, thereby producing consistent features on the tissue surface in the image. For example, green light can be used to image blood vessels in viscera because green light is generally absorbed more in blood than in tissue. The second wavelength can be coherent or incoherent. The second image can also be based on multiple wavelengths, for example, white light can be used. The second wavelength can be generated by, for example, a second coherent light source. Optionally, the first and second wavelengths can be generated by a single coherent light source configured to generate coherent light of multiple wavelengths.

[0057] In one embodiment, the second image sequence may be a second speckle image sequence, and the method may further include: determining a second speckle contrast image based on the second speckle image sequence, and adjusting or correcting a first speckle contrast image based on changes in speckle contrast magnitude in the second speckle contrast image sequence.

[0058] Multispectral coherence correction, also known as dual-laser correction, can eliminate or reduce noise in a first speckle contrast image by adjusting the speckle contrast in a determined first speckle contrast image based on changes in speckle contrast in a determined second image sequence. This adjustment can be based on a predetermined correlation between the speckle contrast of the first and second speckle contrast images.

[0059] By using a second image based on a second wavelength for multispectral coherence correction and image registration, it is advantageous to combine multispectral coherence correction with image registration using the second image. In such an embodiment, the second wavelength preferably has a relatively small penetration depth. Thus, the second image can include information primarily related to the surface of the target region. This is especially true for tissues with little perfusion near the surface, such as skin, scar tissue, and certain tumor types.

[0060] In one embodiment, the method may further include: dividing each image in at least a portion of the first speckle image sequence, each image in at least a portion of the first speckle contrast image, and each image in the plurality of images into a plurality of regions, preferably into non-overlapping regions. Preferably, the regions in the images from the plurality of images correspond to the regions in the associated first speckle image and the regions in the first speckle contrast image, respectively. Determining the transformation parameters may include determining the transformation parameters for each region; and determining the sequence of registered first speckle images and the sequence of first speckle contrast images includes: registering each region of the first speckle image and each region of the first speckle contrast image based on the transformation, wherein the transformation is based on the corresponding regions in the images from the plurality of images.

[0061] The aforementioned regions can be determined based on factors such as the geometry of the image (e.g., a grid of rectangular or triangular regions) or image characteristics (e.g., light intensity, or a group of pixels that appear to belong to an anatomical structure).

[0062] This allows for the correction of localized motion within a portion of an image, resulting in a higher-quality composite image. Localized motion is typically caused by the movement of a target, such as due to human movement, breathing, heartbeat, or muscle contractions (e.g., the peristaltic movement of the lower abdomen). These regions can be as small as a single pixel. If a weighted average is used to combine two or more images, weights can be assigned individually to each region and / or to the entire image. Composite images can also be based on regions or on a per-image basis.

[0063] In one embodiment, the target area may include a perfused organ (preferably perfused with body fluids, more preferably with blood and / or lymph), and / or may include one or more blood vessels and / or lymphatic vessels. The method may further include: calculating the perfusion intensity based on the combined speckle image, wherein the perfusion intensity is preferably blood perfusion intensity or lymphatic perfusion intensity.

[0064] The method may also include post-processing the image (e.g., thresholding, false coloring, overlaying on other images such as white light images), and / or displaying the combined image or a derivative thereof.

[0065] In a second aspect, the present invention also relates to a hardware module for an imaging device, preferably a medical imaging device. The hardware module includes: a first light source for exposing a target region to coherent first light of a first wavelength, the target region including living tissue. The hardware module may further include: an image sensor system having one or more image sensors for capturing at least one image sequence, the at least one image sequence including a first speckle image captured during exposure to the first light. The hardware module may further include: a computer-readable storage medium and a processor, wherein the computer-readable storage medium has computer-readable program code, the processor is coupled to the computer-readable storage medium, the processor is preferably a microprocessor, more preferably a graphics processing unit, wherein, in response to executing the computer-readable program code, the processor is configured to: determine one or more transformation parameters for an image registration algorithm for registering the first speckle images to each other, the transformation parameters being based on a similarity measure of pixel values ​​of pixel groups in a plurality of images selected from or associated with the first laser speckle image, the transformation parameters preferably being defined as one of: homography transformation, projection transformation, or affine transformation; and determine a registered first speckle image by registering the first speckle image based on the one or more transformation parameters and the image registration algorithm, and determine a combined speckle contrast image based on the registered first speckle image; or determine a first speckle contrast image based on the first speckle image, determine a registered speckle contrast image by registering the first speckle contrast image based on the one or more transformation parameters and the image registration algorithm, and determine a combined speckle contrast image based on the registered first speckle contrast image.

[0066] In one embodiment, the hardware module may include: a second light source for illuminating the target area with light of at least a second wavelength, different from the first wavelength, simultaneously or alternately with the first light source. The at least one image sensor is further configured to capture a second image sequence, the second images being captured during exposure to the second light. The plurality of images may be selected from the second image sequence, each of the second images being associated with a first speckle image.

[0067] An image sensor system may include a first image sensor for capturing a first image sequence and a second image sensor for capturing a second image sequence, or a single image sensor for capturing both the first image sequence and the second image sequence.

[0068] In one embodiment, the hardware module may further include a display for displaying the combined speckle image and / or its derivatives, preferably displaying a perfusion intensity image. Alternatively or additionally, the hardware module may include video output for outputting the combined speckle image and / or its derivatives.

[0069] The image sensor system may include a first image sensor for capturing an image at a first wavelength and a second image sensor for capturing an image at least at a second wavelength. The first and second image sensors may be the same image sensor, different portions of a single image sensor (e.g., the red and green channels of an RGB camera), or different image sensors. The module may also include optics for directing light from a first light source and an optional second light source to a target area, and / or directing light from the target area to the first and second image sensors.

[0070] The present invention also relates to a medical imaging device, comprising the hardware modules of any of the above, wherein the device is preferably one of the following: an endoscope, a laparoscope, a surgical robot, a handheld laser speckle contrast imaging device, or an open surgical laser speckle contrast imaging system.

[0071] In another aspect, the present invention relates to a computing module for a laser speckle imaging system, the computing module comprising a computer-readable storage medium and a processor, the computer-readable storage medium embodying at least a portion of a program, the processor being coupled to the computer-readable storage medium, the processor preferably being a microprocessor, more preferably a graphics processing unit, wherein, in response to executing the computer-readable storage code, the processor is configured to perform executable operations, the executable operations including:

[0072] The method includes receiving at least one image sequence, the at least one image sequence including a first speckle image captured during exposure to coherent first light of a first wavelength, the target region including living tissue; determining one or more transformation parameters for an image registration algorithm to register the first speckle images to each other, the transformation parameters being based on a similarity measure of pixel values ​​of pixel groups in a plurality of images in the at least one image sequence, the plurality of images being selected from or associated with the first speckle image, the transformation parameters preferably being defined as one of: homography transformation, projection transformation, or affine transformation; and determining a registered first speckle image by registering the first speckle image based on the one or more transformation parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle image; or determining a first speckle contrast image based on the first speckle image, determining a registered speckle contrast image by registering the first speckle contrast image based on the one or more transformation parameters and the image registration algorithm, and determining a combined speckle contrast image based on the registered first speckle contrast image.

[0073] For example, the aforementioned computational module can be added to existing or new medical imaging devices (e.g., laparoscopes or endoscopes) to improve laser speckle contrast imaging, particularly perfusion imaging. In one embodiment, the method steps described in this disclosure can be performed by a processor in a device for coupling coherent light to an endoscopic system. Such a device can be coupled between the light source and video processor of the endoscopic system and the endoscope (e.g., a laparoscope). Thus, the coupling device can add laser speckle imaging capabilities to the endoscopic system. The aforementioned coupling device is described in more detail in Dutch patent application NL 2026240, which is incorporated herein by reference.

[0074] In one embodiment, the at least one image sequence includes a second image sequence, the second images being captured during exposure to second light having one or more second wavelengths, the second light preferably being coherent light of a second wavelength, or the second light including multiple second wavelengths of the visible spectrum, wherein the exposure to the second light is alternated with or simultaneous with the exposure to the first light. The multiple images may be selected from the second image sequence, each of the second images being associated with a first speckle image.

[0075] The present invention may also relate to a computer program or computer program suite comprising at least one software code portion or computer program product, the computer program product storing at least one software code portion, which, when run on a computer system, is configured to perform any of the above method steps.

[0076] The present invention may also relate to a non-transitory computer-readable storage medium that stores at least one portion of software code that, when executed or processed by a computer, is configured to perform any of the above-described method steps.

[0077] The invention will be further described with reference to the accompanying drawings, which schematically illustrate embodiments according to the invention. It should be understood that the invention is not limited in any way to these specific embodiments. Attached Figure Description

[0078] The following description of the accompanying drawings of specific embodiments of the present invention is merely exemplary in nature and is not intended to limit the teachings, applications, or uses of the invention.

[0079] Figure 1A The diagram schematically illustrates a system for motion-compensated laser speckle contrast imaging according to an embodiment of the present invention. Figures 1B to 1D A flowchart of laser speckle contrast imaging according to an embodiment of the present invention is shown;

[0080] Figures 2A to 2C The original speckle image, the laser speckle contrast image based on the original speckle image, and the perfusion image based on the laser speckle contrast image are shown.

[0081] Figure 3A and Figure 3B A flowchart of laser speckle contrast imaging according to an embodiment of the present invention is shown;

[0082] Figure 4A and Figure 4B A flowchart of laser speckle contrast imaging according to an embodiment of the present invention is shown;

[0083] Figure 5 A method for determining a transformation according to an embodiment of the present invention is shown;

[0084] Figures 6A to 6D A method for determining a transformation according to an embodiment of the present invention is shown;

[0085] Figure 7A and Figure 7B A flowchart illustrating laser speckle contrast imaging combining more than two original speckle images according to an embodiment of the present invention is shown;

[0086] Figure 8 A flowchart illustrating a computationally corrected laser speckle contrast image according to an embodiment of the present invention is shown;

[0087] Figure 9 This schematically illustrates a motion-compensated speckle contrast image determined by weighted averaging according to an embodiment of the present invention; and

[0088] Figure 10 A block diagram of an exemplary data processing system that can be used to perform the methods and software products described in this application is shown. Detailed Implementation

[0089] Laser speckle contrast imaging can be based on spatial contrast, temporal contrast, or a combination thereof. Generally, using spatial contrast results in high temporal resolution but relatively low spatial resolution. Furthermore, individual images may suffer quality loss, for example, due to motion or illumination artifacts, leading to image quality variations from image to image. On the other hand, using temporal contrast is associated with relatively high spatial resolution and relatively low temporal resolution. However, the quality of temporal contrast can be strongly affected by the motion of the target relative to the camera, potentially leading to incorrect pixel combination. Hybrid methods may have some of the advantages and disadvantages of both approaches mentioned above.

[0090] In this disclosure, a speckle image may also be referred to as a raw speckle image to better distinguish it from a speckle contrast image. Therefore, the term "raw speckle image" can refer to an image representing a speckle pattern, where pixels have pixel values ​​representing light intensity. A raw speckle image can be an unprocessed image or a (pre)processed image. The term "speckle contrast image" can be used to refer to a processed speckle image, where pixels have pixel values ​​representing the magnitude of speckle contrast, typically a relative standard deviation over a predefined neighborhood of the pixel.

[0091] Figure 1A A system 100 for motion-compensated laser speckle contrast imaging according to an embodiment of the present invention is schematically illustrated. The system may include a first light source 104 for generating coherent light, such as laser light, at a first wavelength for illuminating a target region 102. The target region is preferably living tissue, such as skin, intestines, or brain tissue. The first wavelength may be selected to interact with bodily fluids (e.g., blood or lymph) that can pass through the target. The first wavelength may be in the red or (near) infrared portion of the electromagnetic spectrum, for example, in the range of 600–700 nm, preferably in the range of 620–660 nm or in the range of 700–1200 nm. The first wavelength may be selected based on the bodily fluid of interest and / or the tissue being imaged. It may also be selected based on the characteristics of the imaging sensor. In particular, different quantities of interest may be imaged, such as flow in a single large or small blood vessel, or microvascular perfusion of the target region, depending on the exposure time, the size of the imaging region, the speckle size, and the image resolution.

[0092] In one embodiment, the system may further include a second light source 106 for generating light of at least a second wavelength for illuminating the target region 102, the at least second wavelength preferably comprising the green portion of the electromagnetic spectrum. The at least second wavelength can be selected to include wavelengths that create images with high visual discriminative power (i.e., high contrast of anatomical features). The second wavelength can be selected based on the tissue in the target region. Typically, the light of the at least second wavelength can be coherent or incoherent light, and can be monochromatic (e.g., narrow-band blue or green imaging light) or polychromatic (e.g., white light). In other embodiments, only the first light source is used. Figure 1C In the illustrated embodiment, at least the second wavelength of light is monochromatic coherent light. For example, the second wavelength can be generated by a second coherent light source. Alternatively, the first wavelength and the second wavelength of light can be generated by a single coherent light source configured to generate coherent light of multiple wavelengths.

[0093] The system may also include one or more image sensors 108 for capturing images associated with light of a first wavelength and, where applicable, images associated with light of at least a second wavelength, both of which have interacted with a target in the target region. In various embodiments, the system may include multiple cameras, such as a first camera for acquiring a first raw speckle image associated with the first wavelength and a second camera for acquiring a corrected image associated with the second wavelength. The system may also include additional optics, such as optical fibers, lenses, or beam splitters, to direct light from one or more light sources to the target region and from the target region to one or more image sensors.

[0094] When a target is illuminated with coherent light, a laser speckle pattern can be formed through self-interference. The image can be received and processed by processing unit 110. (Refer to...) Figures 1B to 1D The example will be described in further detail. The processing unit can output the processed image to, for example, a monitor or computer 112, essentially in real time. The processing unit can be a separate unit or part of a computer. The processed image can be displayed by the monitor or computer.

[0095] In one embodiment, an endoscope or laparoscope can be used to direct light to a target area and acquire an image. For example, one or more light sources, one or more image sensors, a processing unit, and a display can be part of an endoscopic system.

[0096] Figures 1B to 1D A flowchart of motion-compensated laser speckle contrast imaging according to an embodiment of the present invention is shown. Alternatives mentioned in one of these embodiments may also be applied to other embodiments, unless such combinations would lead to contradictions in the description.

[0097] Figure 1B A flowchart of motion-compensated laser speckle contrast imaging according to an embodiment of the present invention is shown. In this embodiment, only coherent light of a first wavelength generated by a first light source 104 is used, which may be, for example, green light, or preferably red light. One or more image sensors 108 are single image sensors, typically optimized or at least suited to the wavelength used, and are monochrome image sensors. As described above, and will be discussed below... Figure 1C and Figure 1D The example shown may be used in other embodiments with different configurations.

[0098] In the first step 120, a sequence of raw speckle images is acquired, for example, the raw speckle image sequence is captured or received from an external source. These speckle images will also be used as correction images. Based on each first raw speckle image, 126 first speckle contrast images can be calculated. The speckle contrast can be determined in any suitable manner, for example, by convolving with a predetermined convolution kernel, or by determining the relative standard deviation of pixel value intensity within a sliding window. Since speckle contrast is perfusion-related, the perfusion unit weight can be determined based on the speckle contrast value of the first speckle contrast images.

[0099] In a single-wavelength embodiment, the original speckle image or a speckle contrast image can be used as the correction image. In each correction image, the location 130 of a predetermined object feature can be determined. Based on the locations of the predetermined object features in two or more correction images, for example using an optical flow algorithm, the displacement vector of the motion of the identified target region can be determined. This will refer to... Figure 5 A more detailed description is provided in Figure 6. A (sparse) optical flow algorithm can be used to determine the displacement vector based on selected features. Other embodiments may use, for example, a dense optical flow algorithm; in this case, feature determination is not required.

[0100] Based on the displacement vector, the transformation used to register the images in the second image sequence can be determined. Preferably, the transformation is selected from a class of homography transformations, a class of projection transformations, or a class of affine transformations. Optionally, the optical flow weights can be determined based on the displacement vector or based on parameters defining the transformation.

[0101] The determined transformation can then be used to register the speckle contrast images to each other (134 or geometric alignment) to produce a registered first speckle contrast image. In an alternative embodiment, the original speckle images can be registered before calculating the speckle contrast. However, such an embodiment is less preferred because image registration can affect pixel values ​​and thus contrast.

[0102] Subsequently, temporal filtering can be used, for example, to combine the first speckle contrast images registered in 136. Temporal filtering may include averaging multiple first speckle contrast images. The averaging may be a weighted average, where the weights are based on optical flow weights and / or perfusion unit weights. In an alternative embodiment, temporal filtering can be used to combine the registered original speckle images, and a speckle contrast image can be determined based on the combined speckle images. This produces a motion-compensated speckle contrast image.

[0103] The combined first raw speckle image can then be post-processed, for example, converted to perfusion values, thresholded to indicate areas with high or low perfusion, overlaid on a white light image of the target area, etc. This post-processing can be performed by, for example, processing unit 110 or computer 112. Figure 1C A flowchart of motion-compensated laser speckle contrast imaging according to an embodiment of the present invention is shown. In the illustrated embodiment, light of a first wavelength generated by a first light source 104 is red light, and light of at least a second wavelength generated by a second light source 106 is coherent green light. One or more image sensors 108 are single sensors including red, green, and blue channels. The first and second wavelengths are selected to minimize crosstalk. The signal in the red channel caused by green light is significantly smaller than the signal caused by red light; and the signal in the green channel caused by red light is significantly smaller than the signal caused by green light. As described above, other embodiments may use different configurations.

[0104] In the first step 140, a sequence of RGB images is received. A first sequence of first raw speckle images (142) can be extracted from the red channel of the RGB image sequence, and a second corrected image sequence (144) can be extracted from the green channel of the RGB images. The corrected image sequence can be a second sequence of second raw speckle images. Each first raw speckle image can be associated with a corrected image extracted from the same RGB image. In other embodiments, the first raw speckle images and corrected images can be obtained by different cameras, different sensors of a multi-sensor camera (e.g., a 3CCD camera), other (color) channels of a single camera (e.g., a YUV camera); or, if the target is alternately illuminated with light of a first wavelength and at least a second wavelength, the images can be alternately obtained by a single monochrome camera.

[0105] Based on each first original speckle image, 146 first speckle contrast images can be calculated. Optionally, based on each second original speckle image, 148 second speckle contrast images can be calculated. The speckle contrast can be determined in any suitable manner, such as by convolving with a predetermined convolution kernel, or by determining the relative standard deviation of pixel value intensity within a sliding window. Since speckle contrast is related to perfusion, the perfusion unit weights can be determined based on the speckle contrast values ​​of the first speckle contrast images. Optionally, as referenced... Figure 8 In more detail, the second speckle contrast image can be used to correct the first speckle contrast image. In this case, the corrected speckle contrast value can be used to determine the perfusion unit weight.

[0106] In each corrected image (i.e., in this embodiment, in each second original speckle image), the location of 150 predetermined object features can be determined. Based on the locations of the predetermined object features in two or more corrected images, displacement vectors for identifying the motion of the target region can be determined, for example using an optical flow algorithm. This will refer to... Figure 5 The following is a more detailed description, as shown in Figure 6. A (sparse) optical flow algorithm can be used to determine the displacement vector based on selected features. Other embodiments may use, for example, a dense optical flow algorithm; in this case, feature determination is not required.

[0107] Based on the displacement vector, the transformation used to register the images in the second image sequence can be determined. Preferably, the transformation is selected from a class of homography transformations, a class of projection transformations, or a class of affine transformations. Optionally, the optical flow weights can be determined based on the displacement vector or based on parameters defining the transformation.

[0108] The determined transformation can then be used to register or geometrically align the first speckle contrast image associated with the corrected image.

[0109] Subsequently, temporal filtering can be used, for example, to combine the first speckle contrast images registered in 156. Temporal filtering may include averaging multiple first speckle contrast images. The averaging may be a weighted average, where the weights are based on optical flow weights and / or perfusion unit weights.

[0110] In the final step 158, the combined first raw speckle image can be post-processed, for example, converted to perfusion values, thresholded to indicate areas with high or low perfusion, overlaid on a white light image of the target area, etc. This post-processing can be performed by, for example, processing unit 110 or computer 112.

[0111] Figure 1DA flowchart of motion-compensated laser speckle contrast imaging according to an embodiment of the present invention is shown. In the illustrated embodiment, the first wavelength of light generated by the first light source 104 is infrared light, but other colors such as red, green, or blue are also possible. The light of at least a second wavelength generated by the second light source 106 is (incoherent) white light. The advantage of using infrared and white light is that both can be used simultaneously without significantly interfering with each other.

[0112] As shown in the figure, one or more image sensors 108 are two image sensors from two cameras (an infrared camera and a color camera). This is feasible if laser speckle imaging is added to a system that already includes a color camera, for example, in an open surgical environment. Furthermore, having dedicated image sensors allows for individual optimization of hardware and / or device parameters, such as exposure time. As mentioned above, other embodiments can use different configurations. For example, in some embodiments, a single camera can be used to capture both an infrared image and a (white light) color image. The advantage of using a single camera is that the infrared and color images are automatically aligned and correlated with each other.

[0113] In the first step 160, a first sequence of first images is captured by an infrared camera. This first sequence can be stored by an image processor as a sequence of raw speckle images. In the second step 161, a second sequence of second images is captured by a color camera. This second sequence can be stored as a sequence of corrected images. Each raw speckle image can be associated with one or more corrected images. Preferably, each raw speckle image is associated with at least one corrected image that is temporally closest to, and preferably captured simultaneously with, the raw speckle image. Preferably, the frame rates of the first and second cameras are selected to allow direct association, for example, by selecting one frame rate as an integer multiple of the other frame rate.

[0114] Based on each original speckle image, 166 speckle contrast images can be computed. The speckle contrast can be determined in any suitable manner, such as by convolving with a predetermined kernel, or by determining the relative standard deviation of pixel value intensity within a sliding window. Since speckle contrast is perfusion-related, perfusion unit weights can be determined based on the speckle contrast values ​​of the first speckle contrast image.

[0115] In each corrected image (i.e., in each white light image in this embodiment), the position of 170 predetermined object features can be determined. Based on the positions of the predetermined object features in two or more corrected images, displacement vectors for identifying the motion of the target region can be determined, for example using an optical flow algorithm. This will refer to... Figure 5 The following is a more detailed description, as shown in Figure 6. A (sparse) optical flow algorithm can be used to determine the displacement vector based on selected features. Other embodiments may use, for example, a dense optical flow algorithm; in this case, feature determination is not required.

[0116] Based on the displacement vector, the transformations used for registering and correcting images in the image sequence can be determined. Preferably, the transformations are selected from classes of homography, projection, or affine transformations. Optionally, the optical flow weights can be determined based on the displacement vector or on parameters defining the transformation.

[0117] The determined transformation can then be used to register or geometrically align the first speckle contrast image associated with the corrected image. Since two cameras are used in this embodiment, and these cameras do not share a field of view or a reference frame, registering the speckle contrast image based on the transformation parameters determined using the corrected image can include applying a transformation to the transformation parameters to account for this variation in the reference frame. If the two cameras are positioned at fixed locations relative to each other, the transformation can be predetermined. Otherwise, the transformation can be determined based on image processing of, for example, the calibration image or markers. The markers can be natural or artificial.

[0118] Subsequently, temporal filtering can be used, for example, to combine the first speckle contrast images registered in 176. Temporal filtering may include averaging multiple first speckle contrast images. The averaging may be a weighted average, where the weights are based on optical flow weights and / or perfusion unit weights.

[0119] In the final step 178, the combined first raw speckle image can be post-processed, for example, converted to perfusion values, thresholded to indicate areas with high or low perfusion, overlaid on a white light image of the target area, etc. This post-processing can be performed by, for example, processing unit 110 or computer 112.

[0120] In one embodiment, the method steps described herein can be executed by a processor in a device for coupling coherent light to an endoscope system. Such a device can be coupled between the light source and video processor of the endoscope system and the endoscope (e.g., a laparoscope) of the endoscope system. Thus, the coupling device can add laser speckle imaging capabilities to the endoscope system. The aforementioned coupling device is described in more detail in Dutch patent application NL 2026240, which is incorporated herein by reference.

[0121] In alternative embodiments, the method steps described in this disclosure may be combined with a pre-existing imaging system, thereby enabling application in open surgical settings.

[0122] Figure 2AThe original speckle image and laser speckle contrast images based on the original image are shown, depicting a target with low perfusion and a target with high perfusion. Images 202 and 204 are original speckle images of a human fingertip including the nail bed. When image 202 was obtained, blood flow through the finger was restricted, resulting in low blood perfusion (artificial ischemia). When image 204 was obtained, blood flow was unrestricted, resulting in much higher blood perfusion compared to the previous case. The difference in speckle pattern associated with the perfusion difference is difficult for a human observer to see. A magnified portion 210 of image 204 is also shown in the figure, revealing the speckle structure in more detail.

[0123] Images 206 and 208 are speckle contrast images based on images 202 and 204, respectively. Lighter colors indicate lower contrast, thus indicating higher perfusion, while darker colors indicate higher contrast, thus indicating lower perfusion. In these images, the differences in perfusion are immediately and clearly visible, especially in the nail bed where blood flow is relatively close to the surface.

[0124] Figure 2B A series of laser speckle contrast images of a low-perfusion target exhibiting motion before and after motion correction, according to an embodiment of the present invention, are shown. Image 220 1-5 This is a speckle contrast image of the human fingertip (including the nail bed). In image 220... 2-4 During the acquisition process, finger movement causes a loss of contrast due to finger motion. If the user is interested in blood flow, the image 220... 2-4 The low speckle contrast represented by the lighter colors in the image 222 may be considered motion artifacts. 1-5 Based on image 220 respectively 1-5 The same original speckle image, but it has been corrected using a motion correction algorithm according to an embodiment of the present invention.

[0125] Figure 2C A series of speckle contrast images and graphs representing perfusion levels based on this series of speckle contrast images are shown. Graph 230 shows perfusion measurements of the nail bed, showing a first curve 232 representing perfusion determined based on uncorrected measurements, and a second curve 234 representing perfusion determined based on measurements corrected using a motion correction algorithm as described in this disclosure. Within a timeframe of approximately 10 seconds, blood flow to the finger is artificially restricted, and the restriction is lifted after approximately 30 seconds. Particularly within the timeframe of 23 to 38 seconds, the uncorrected perfusion measurements show numerous motion artifacts, where perfusion appears to rise sharply and then fall again.

[0126] The figure further illustrates the process before processing using a single-wavelength-based motion correction and compensation algorithm (Image 236). 1-3 ) and afterwards (Image 238)1-3 Three exemplary speckle contrast images are provided. In these images, lighter colors represent low speckle contrast, thus indicating high perfusion, while darker colors represent high speckle contrast, thus indicating low perfusion. The three uncorrected images look more or less identical, making it difficult for the user to identify the time with low or high perfusion (or, in other applications, time could be replaced by region). In contrast, the motion-corrected images show a clear difference between the (intermediate) image acquired during restricted blood flow and other images with unrestricted blood flow, allowing the user to select the time or place with high perfusion.

[0127] Furthermore, anatomical structures, such as the edges of fingers and nails, are more easily identified in motion-compensated images, whereas details may be difficult to discern in uncorrected images due to their granularity. This further facilitates the user's interpretation of the images.

[0128] Figure 3A A flowchart of laser speckle contrast imaging according to an embodiment of the present invention is shown. In a first step 302, a first raw speckle image can be obtained at a first time instance t = t1. A light source can illuminate a target area with coherent light of a predetermined wavelength, and an image sensor can capture the first raw speckle image based on the predetermined wavelength. Based on the first raw speckle image, a first laser speckle contrast image can be calculated 304. For example, the laser speckle contrast image can be determined by determining the relative standard deviation of pixel values ​​in a sliding window (e.g., a 3×3 window, a 5×5 window, or a 7×7 window). Typically, a (2n+1)×(2n+1) window can be chosen for the natural number n according to the speckle size. Alternatively, a convolution with a kernel can be used, where the kernel size can be chosen based on the speckle size. The relative standard deviation can be determined by calculating the standard deviation of pixel intensity values ​​in a region and dividing it by the average pixel intensity value in that region. Alternatively, the value of the laser speckle contrast can be determined in any other suitable manner.

[0129] In the next step 308, the processor may determine a first plurality of first features in the first original speckle image. Optionally, the first plurality of first features may be determined in the first speckle contrast image. Preferably, an image with the most clearly defined anatomical features is used; whether anatomical structures have higher visual distinguishability in the original speckle image or the speckle contrast image may depend on imaging parameters such as wavelength and exposure time. Figure 3A In the remaining description, the term "speckle image" can refer to either the original speckle image or a speckle contrast image. (See below for reference.) Figure 5 The steps related to feature detection are described in more detail in Figure 6.

[0130] Steps 312-318, performed at the second time instance t=t2, are similar to steps 302-308, respectively. Therefore, a second original speckle image (312) can be obtained at the second time instance t=t2. The target area can be illuminated by coherent light of a predetermined wavelength, and the image sensor can capture the second original speckle image based on the predetermined wavelength. Based on the second original speckle image, a second laser speckle contrast image (314) can be calculated.

[0131] In the next step 318, the processor can determine a second plurality of second features in the second speckle image. At least a portion of the second plurality of second features should correspond to at least a portion of the first plurality of first features. Because in typical applications, the target region does not change significantly between t = t1 and t = t2, the second speckle image is usually similar to the first speckle image. Therefore, when using a deterministic algorithm to detect features, in the sense that the features detected in the image represent the same or virtually identical anatomical features in the imaging target, most of the features detected in the second speckle image usually correspond to the features detected in the first speckle image. Therefore, a plurality of second features can be associated with a plurality of first features.

[0132] In the next step 320, the processor may determine multiple displacement vectors based on the first feature and the corresponding second feature, the displacement vectors describing the displacement of the feature relative to the image. For example, the processor may determine a feature pair including a first feature and a second feature, determine a first position of the first feature relative to a first speckle image, determine a second position of the second feature relative to a second speckle image, and determine the difference between the first position and the second position. Typically, the corresponding feature pair may be a pair of first features and associated second features representing the same anatomical feature.

[0133] In the next step 322, the processor can determine a transformation, such as an affine transformation or a more general homography transformation, for registering corresponding features to each other based on multiple displacement vectors. For example, the transformation that minimizes the distance between corresponding feature pairs can be found by selecting from a class of transformations. Based on this transformation, the processor can register or align the first laser speckle contrast image and the second laser speckle contrast image to each other 324 by transforming the first laser speckle contrast image and / or the second laser speckle contrast image. Typically, the older image can be converted to be registered with the newer image.

[0134] In other embodiments, steps 308 and 318 can be omitted, and the displacement vector can be determined based on the first speckle image and the second speckle image. For example, dense optical flow algorithms, such as the Pyramid Lucas-Kanade algorithm or the Farneback algorithm, can be used to determine the displacement vector. These algorithms typically convolve the pixel neighborhood from the first speckle image with a portion or all of the second speckle image, thereby matching the neighborhood of each pixel in the first speckle image with the neighborhood in the second speckle image. Such methods may also include determining a polynomial expansion to model the pixel values ​​in the pixel neighborhoods of the first and second speckle images, comparing the expansion coefficients, and determining the displacement vector based on the comparison.

[0135] In this way, the displacement vector of a single pixel or a group of pixels can be determined based on the pixel values ​​in the pixel group of the speckle image.

[0136] In some embodiments, step 320 may be omitted, and a transformed image may be determined, for example, using a trained neural network that receives the first and second images as input and provides the transformed image as output to register the first image with the second image, or alternatively, to register the second image with the first image.

[0137] The processor can then calculate a combined (e.g., averaged) laser speckle contrast image based on the registered first and second laser speckle contrast images. Calculating the combined image may include: calculating a weighted average, where the weights are preferably based on a normalized amount of speckle contrast, relative changes in speckle contrast, a determined displacement vector, and / or parameters associated with the determined transform. Calculating the combined image may further include: applying one or more filters, such as a median filter or an outlier filter.

[0138] In other embodiments, these steps may be performed in a different order. For example, a laser speckle contrast image may be calculated after the original speckle image has been registered. In this way, a temporal speckle contrast image or a temporal-spatial speckle contrast image may be calculated. However, if the transformation is more general than translation and rotation (e.g., including scaling or shearing), the transformation may distort the speckle pattern, thereby introducing noise sources. In some embodiments, a single laser contrast original speckle image may be calculated based on a combination (e.g., average) of the original speckle images. In this case, the images are preferably registered with subpixel accuracy.

[0139] Figure 3BA flowchart of laser speckle contrast imaging according to an embodiment of the present invention is shown. In the first step 332, a first raw speckle image can be obtained at a first time instance t = t1. A first light source can illuminate the target area with coherent light of a first wavelength, and a first image sensor can capture the first raw speckle image based on the first wavelength. Based on the first raw speckle image, a first laser speckle contrast image can be calculated 334. The laser speckle contrast image can be determined with reference to step 304 above.

[0140] In the next step 336, a first corrected image associated with the first speckle image can be obtained at the first time instance t = t1. The first corrected image may include pixels having pixel values ​​and pixel coordinates, where the pixel coordinates identify the position of the pixels in the image. Preferably, the first corrected image is obtained simultaneously with the first original speckle image, but in alternative embodiments, for example, the first corrected image may be obtained before or after the first original speckle image.

[0141] The second light source can illuminate the target area with light of at least a second wavelength different from the first wavelength, and the second image sensor can capture the first calibrated image based on the at least second wavelength. The second light source can use coherent or incoherent light. The second light source can produce monochromatic or polychromatic light, such as white light. The second image sensor can be the same sensor as the first image sensor, or a different sensor.

[0142] Refer to the above Figure 3A In the described embodiment, the second wavelength is the same as the first wavelength, and the first corrected image is the same as the first original speckle image.

[0143] In the next step 338, the processor may determine a first plurality of first features in the first corrected image. (Refer to...) Figure 5 The steps related to feature detection are described in more detail in Figure 6.

[0144] Steps 342–348 are respectively similar to steps 332–338 performed at the second time instance t = t2. Therefore, a second original speckle image 342 can be obtained at the second time instance t = t2. The first light source can illuminate the target area with coherent light of a first wavelength, and the first image sensor can capture the second original speckle image based on the first wavelength. Based on the second original speckle image, a second laser speckle contrast image 344 can be calculated.

[0145] In the next step 346, a second corrected image associated with the second original speckle image can be obtained at the second time instance t = t2. The second light source can illuminate the target area with light of at least a second wavelength, and the second image sensor can capture the second corrected image based on at least the second wavelength.

[0146] In the next step 348, the processor can determine a second plurality of second features in the second corrected image. At least a portion of the second plurality of second features should correspond to at least a portion of the first plurality of first features. Because in typical applications, the target region does not change significantly between t = t1 and t = t2, the second corrected image is usually similar to the first corrected image. Therefore, when using a deterministic algorithm to detect features, in the sense that the features detected in the image represent the same or virtually identical anatomical features in the imaging target, most of the features detected in the second corrected image usually correspond to the features detected in the first corrected image. Therefore, a plurality of second features can be associated with a plurality of first features.

[0147] In the next step 350, the processor may determine multiple displacement vectors based on the first feature and the corresponding second feature, the displacement vectors describing the displacement of the feature relative to the image. For example, the processor may determine a feature pair including a first feature and a second feature, determine a first position of the first feature relative to a first corrected image, determine a second position of the second feature relative to a second corrected image, and determine the difference between the first position and the second position. Typically, the corresponding feature pair may be a pair of first features and associated second features representing the same anatomical feature.

[0148] In the next step 352, the processor can determine a transformation, such as an affine transformation or a more general homography transformation, for registering corresponding features to each other based on multiple displacement vectors. For example, the transformation that minimizes the distance between corresponding feature pairs can be found by selecting from a class of transformations. Based on this transformation, the processor can register or align the first laser speckle contrast image and the second laser speckle contrast image to each other 354 by transforming the first laser speckle contrast image and / or the second laser speckle contrast image. Typically, the older image can be converted to be registered with the newer image. In embodiments with more than one image sensor, the transformation parameters determined based on the corrected image can be adjusted to account for differences between the fields of view of more than one image sensor.

[0149] In other embodiments, steps 338 and 348 may be omitted, and the displacement vector may be determined based on the first and second corrected images. For example, dense optical flow algorithms, such as the Pyramid Lucas-Kanade algorithm or the Farneback algorithm, may be used to determine the displacement vector. These algorithms typically convolve the pixel neighborhood from the first corrected image with a portion or all of the second corrected image, thereby matching the neighborhood of each pixel in the first corrected image with its neighborhood in the second corrected image. Such methods may also include determining a polynomial expansion to model the pixel values ​​in the pixel neighborhoods of the first and second corrected images, comparing the expansion coefficients, and determining the displacement vector based on the comparison.

[0150] In this way, the displacement vector of a single pixel or a group of pixels can be determined based on the pixel values ​​in the pixel group in the first corrected image and the pixel values ​​in the associated pixel group in the second corrected image.

[0151] In some embodiments, step 350 may be omitted, and a transformed image may be determined, for example, using a trained neural network that receives the first and second images as input and provides the transformed image as output to register the first image with the second image, or alternatively, to register the second image with the first image.

[0152] As described above with reference to step 326, the processor can calculate 356 combined (e.g., averaged) laser speckle contrast images based on the registered first laser speckle contrast image and the second laser speckle contrast image.

[0153] In other embodiments, these steps may be performed in a different order. For example, a laser speckle contrast image may be calculated after the original speckle image has been registered. In this way, a temporal speckle contrast image or a temporal-spatial speckle contrast image may be calculated. However, if the transformation is more general than translation and rotation (e.g., including scaling or shearing), the transformation may distort the speckle pattern, thereby introducing noise sources. This is particularly true for embodiments with more than one camera. In some embodiments, a single laser contrast original speckle image may be calculated based on a combination (e.g., average) of the original speckle images. In this case, the images are preferably registered with subpixel accuracy.

[0154] Figure 4A A flowchart of a motion-compensated speckle contrast imaging method according to an embodiment of the present invention is shown. In a first step 402, the method may include exposing a target region to coherent light of a predetermined wavelength, the target region comprising living tissue. Preferably, the target region comprises living tissue, such as skin, burn tissue, or viscera, such as intestinal or brain tissue. Preferably, the living tissue is perfused and / or comprises blood vessels and / or lymphatic vessels. The predetermined wavelength may be a wavelength in the visible spectrum, such as the red, green, or blue portions of the visible spectrum, or the predetermined wavelength may be a wavelength in the infrared portion of the spectrum, preferably in the near-infrared portion.

[0155] In the next step 404, the method may include, for example, capturing at least one image sequence by an image sensor, the at least one image sequence including a (raw) speckle image captured during exposure to the first light.

[0156] Each raw speckle image can include pixels, which are defined by pixel coordinates and have pixel values. Pixel coordinates define the position of a pixel relative to the image and are typically associated with sensor elements of an image sensor. Pixel values ​​can represent light intensity.

[0157] Image sensors can include 2D image sensors, such as CCDs, or monochrome or color cameras. Images in an image sequence can be frames from a video stream or frames from multi-frame snapshots.

[0158] In the next step 406, the method may further include determining one or more transformation parameters for an image registration algorithm used to register the speckle images to each other. These transformation parameters may be based on a similarity measure of pixel values ​​of pixel groups in a plurality of images selected from the speckle images. Alternatively, the images in the plurality of images may include other images associated with the original speckle image besides the original speckle image. The transformation parameters preferably define one or more transformations from a group of homography, projection, or affine transformations.

[0159] The determination of transformation parameters based on pixel groups is described in more detail below with reference to step 416. It should be understood that in this embodiment, the (original) speckle image is used as the correction image.

[0160] The method may further include: using step 408 or step 410, determining a combined laser speckle contrast image based on at least a portion of the first speckle image sequence and one or more determined transformation parameters.

[0161] In step 408, the method may further include: registering the speckle image based on the one or more transform parameters and the image registration algorithm to determine a registered speckle image, and determining a combined speckle contrast image based on the registered speckle image. In such an embodiment, the algorithm may first use the determined transform to register the original speckle image sequence, then calculate the speckle contrast image sequence, and then combine the registered speckle contrast image.

[0162] In alternative step 410, the method may further include: determining a speckle contrast image based on the speckle image; registering the speckle contrast image based on the one or more transform parameters and the image registration algorithm to determine a registered speckle contrast image; and determining a combined speckle contrast image based on the registered speckle contrast image. In other words, the algorithm may first calculate a sequence of speckle contrast images, then use the determined transform to register the speckle contrast images, and then combine the registered speckle contrast images.

[0163] Further alternatives will be discussed below with reference to steps 418 and 420.

[0164] Figure 4B A flowchart of a motion-compensated speckle contrast imaging method according to an embodiment of the present invention is shown. In a first step 412, the method may include alternately or simultaneously exposing a target region to coherent first light of a first wavelength and one or more second wavelengths of second light, preferably, the one or more second wavelengths being at least partially different from the first wavelength. The second light may be, for example, coherent light of a second wavelength, narrowband light, or light comprising multiple second wavelengths of the visible spectrum, such as white light. Preferably, the target region comprises living tissue, such as skin, burn tissue, or viscera, such as intestinal or brain tissue. Preferably, the living tissue is perfused and / or comprises blood vessels and / or lymphatic vessels.

[0165] In the next step 414, the method may include, for example, capturing a first (raw) speckle image sequence during exposure to a first light and capturing a second (corrected) image sequence during exposure to a second light using an image sensor system having one or more image sensors in a fixed relationship with each other. The speckle images of the first speckle image sequence may be associated with images of the second image sequence.

[0166] In the case of simultaneous exposure and acquisition, each second image can be associated with a first speckle image acquired simultaneously. In embodiments where the second image and the first speckle image are the same image, each image can be considered to be associated with itself, and depending on the image's function in the algorithm (e.g., determining transformation parameters or providing perfusion information), the image can be referred to as either the first speckle image or the second image.

[0167] In the case of alternating acquisition, a first speckle image can be associated with, for example, a second image acquired immediately before or after the first speckle image, or with both (i.e., a second image acquired immediately before and after the first speckle image). When the first speckle image is acquired at a higher rate than the second image, multiple first speckle images can be associated with a single second image.

[0168] Therefore, a first sequence of raw speckle images and a sequence of corrected images of the target region can be obtained, with each corrected image associated with one or more first raw speckle images. Each corrected image may include pixels, defined by pixel coordinates and having pixel values. Pixel coordinates define the position of a pixel relative to the image and are typically associated with sensor elements of an image sensor. Pixel values ​​may represent light intensity.

[0169] An image sensor system may include one or more 2D image sensors, such as CCDs. One or more image sensors, such as grayscale cameras or color (RGB) cameras, may be used to acquire a first raw speckle image and a corrected image. Images in an image sequence may be, for example, frames from a video stream or a multi-frame snapshot.

[0170] In the next step 416, the method may further include determining one or more transformation parameters for a registration algorithm used to register at least a portion of the first speckle image sequence, based on a similarity measure of pixel values ​​in pixel groups of at least a portion of the second image sequence associated with the first speckle image. The transformation parameters preferably define one or more transformations from the group consisting of homography, projection, or affine transformations.

[0171] Determining the transformation parameters may include selecting a first corrected image from at least a portion of a corrected image sequence and determining a plurality of first pixel groups within the first corrected image. The first corrected image may be a reference corrected image. In one embodiment, for example, when generating a single output image based on user input, the first corrected image may be a first image. In other embodiments, for example, when generating a continuous stream of output images, the first corrected image may be the most recent corrected image in time.

[0172] The first pixel group can be associated with features (e.g., edges or corners) in the first corrected image. Preferably, the features are associated with physical or anatomical features (e.g., blood vessels, or more specifically, sharp corners or bifurcations within blood vessels). Image features not associated with physical features, such as overexposed image portions or the edges of speckle, can exhibit large inter-frame variations and may therefore be less useful for the registered image. Features can be predetermined features, such as features belonging to a class of features, such as corners or regions with large intensity differences. Features can be further determined by, for example, quality metrics, constraints on the mutual distances between features, etc.

[0173] Alternatively, a group of pixels can be associated with a region in the first corrected image, such as a neighborhood of a predetermined set of pixels, for example, each pixel in the image, or a selection of pixels that are uniformly distributed across the image.

[0174] Determining the transformation parameters may further include selecting one or more second corrected images different from the first corrected image. For each of the selected one or more second corrected images, a plurality of second pixel groups may be determined. The second pixel groups may be associated with features in the second corrected images. If feature-based (image) registration is used, such as using a sparse optical flow algorithm, preferably, the same algorithm is used to determine the first pixel group and the second pixel group.

[0175] If the first pixel group is determined based on pixel coordinates, the second pixel group can be determined by convolving or cross-correlating the first pixel group with the second corrected image, and, for example, by selecting the pixel group most similar to the first pixel group based on a suitable similarity metric. The convolution may be spatially restricted, for example, by searching only for matching second pixel groups located close to the first pixel group. Alternatively or additionally, the second pixel group can be restricted to maintain the mutual orientation of the first pixel group, for example, to prevent anatomically impossible combinations.

[0176] The second pixel group can then be associated with the first pixel group at least based on the similarity of pixel values. In embodiments where the second pixel group is determined by matching or convolution with the first pixel group, this association can be performed as part of determining the second pixel group. If feature-based registration is used, the second pixel group can be associated with the first pixel group based on the similarity between the features associated with the second pixel group and the features associated with the first pixel group.

[0177] The transformation for registering the second corrected image and the first corrected image can be determined based on the pixel coordinates of pixels in the associated first and second pixel groups, and thus the transformation for registering the associated first speckle image or the resulting first speckle contrast image can be determined. Determining the transformation may include determining the 3D motion of the image sensor system relative to the target region, or the transformation may be determined by the effect of the 3D motion on the acquired image.

[0178] As an intermediate step, a displacement vector can be determined based on the positions of a first pixel group and an associated second pixel group, for example, based on the positions of features in a first corrected image and a second corrected image. The displacement vector can represent the motion of a target region relative to an image sensor or image capturing device. In some embodiments, the displacement vector and / or transformation can be determined using the neighborhood of one or more defined object features.

[0179] The determination of the displacement vector and / or the transformation can be based on optical flow parameters determined using any suitable sparse or dense optical flow algorithm. Determining the displacement vector may include: determining corresponding or matching feature pairs, where one feature in the feature pair is determined in a corrected image associated with a first time instance, and the other feature in the feature pair is determined in subsequent corrected images in a sequence of corrected images associated with subsequent time instances.

[0180] The following is for reference. Figure 5 The methods for determining the transformation parameters and, optionally, the features are discussed in more detail in Figure 6.

[0181] The method may also include using step 418 or step 420 to determine a combined laser speckle contrast image based on at least a portion of the first speckle image sequence and one or more determined transformation parameters.

[0182] In step 418, the method may further include: determining a registered first speckle image by registering at least a portion of a first speckle image sequence based on one or more transform parameters and a registration algorithm; and determining a combined speckle contrast image based on the registered first speckle image. In such an embodiment, the algorithm may first use the determined transform to register the original speckle image sequence, then calculate the speckle contrast image sequence, and then combine the registered speckle contrast images.

[0183] In alternative step 420, the method may further include: determining a first speckle contrast image based on at least a portion of a first speckle image sequence; registering the first speckle contrast image based on one or more transform parameters and a registration algorithm to determine a registered speckle contrast image; and determining a combined speckle contrast image based on the registered first speckle contrast image. In other words, the algorithm may first compute a speckle contrast image sequence, then use the determined transform to register the speckle contrast image, and then combine the registered speckle contrast image.

[0184] In another alternative, determining the combined laser speckle contrast image may include: determining a sequence of registered first original speckle images based on a first original speckle image and a determined transform; determining a combined speckle contrast image based on two or more registered first original speckle images in the registered first original speckle image sequence; and determining the combined speckle contrast image based on the combined original speckle image. In such an embodiment, the algorithm may first register the original speckle image sequence using the determined transform, then combine the registered speckle contrast images, and finally calculate the speckle contrast image.

[0185] In a further alternative, the combination and calculation of speckle contrast can be a single step, for example, by calculating temporal or temporal-spatial speckle contrast based on a first original speckle image sequence registered.

[0186] Combining the original speckle image or speckle contrast image can include averaging, weighted averaging, filtering using, for example, a median filter, etc. The weights used for weighted averaging can be based on, for example, quantities derived from speckle contrast, quantities derived from transform parameters, or quantities derived from displacement vectors. See below for reference. Figure 9 The methods for combining original speckle images or speckle contrast images are discussed in more detail.

[0187] Figure 5A method for determining a transformation according to an embodiment of the present invention is illustrated. A first plurality of first features 506 can be determined from a first corrected image 502 obtained at a first time instance t = t1. 1-3 Each of the plurality of first features can be associated with a group of pixels in the first corrected image. Preferably, the first features relate to anatomical structures, such as blood vessels 504. 1–2 Alternatively, other stable features that do not move between subsequent frames can be assumed. Therefore, the first corrected image is preferably obtained using light that makes this anatomical structure clearly visible. For example, green light can be used, which is strongly absorbed by blood vessels but not by most other tissues. Thus, blood vessels can appear as dark structures in a lit environment, resulting in high visual distinguishability. However, other embodiments may use one or more other wavelengths of light, such as blue or white light.

[0188] Feature 506 1-3 Any suitable feature detection algorithm can be used to determine the feature detector, such as a feature detector based on the Harris detector or the Shi-Tomasi detector, for example, the goodFeaturesToTrack function from the OpenCV library. Other examples of suitable feature detectors and descriptors include Speeded-Up Robust Features (SURF), Features from Accelerated Segment Test (FAST), Binary Robust Independent Elementary Features (BRIEF), and combinations thereof, such as ORB. Various suitable algorithms have been implemented in generally available image processing libraries such as OpenCV. Preferably, depending on the application, the algorithm should be fast enough to allow real-time image processing.

[0189] Typically, sharp corners (e.g., feature 506) 1,3 Bifurcation (e.g., feature 5062) and bifurcation are good features. Preferably, a deterministic feature detection algorithm is used, i.e., a feature detection algorithm that detects the same features in the same image. Preferably, the features should be distributed over most of the image region. A good distribution of feature points on the image can be achieved by requiring the minimum distance between selected feature points. In some embodiments, such as BRIEF- or ORB-based implementations, descriptors identifying feature attributes can be assigned to features, thereby facilitating feature discrimination and feature matching.

[0190] The minimum number of feature points depends on the type of transformation; for example, an affine transformation has six degrees of freedom, while a homography transformation has eight. Therefore, the first plurality of first features can include at least 5 features, preferably at least 25, more preferably at least 250, and even more preferably at least 1000. The number of features can depend on the number of pixels in the image; a larger number of features is used for images with more pixels. Generally, a larger number of features can produce a more accurate transformation because random errors can be averaged out.

[0191] However, the number of features may be limited for various reasons. For example, only a limited number of features may meet predetermined quality metrics, such as the magnitude of local contrast or the sharpness of corners. Furthermore, computation time increases with the number of features; therefore, the number of features may be limited to allow for real-time image registration, for example, for a 50fps video feed, the entire algorithm should ideally run for less than 20ms per frame.

[0192] A second plurality of second features 516 associated with the second pixel group can be determined in the second corrected image 512 obtained at the second time instance t = t2. 1-3 Preferably, the field of view of the second corrected image substantially overlaps with that of the first corrected image, preferably more than half. Preferably, the second feature relates to the same anatomical structure as the first feature, such as blood vessels 514. 1-2 Preferably, the same feature detection algorithm is used to detect features in both the first and second corrected images.

[0193] The determined first and second features may each include positional information relative to the first and second corrected images, respectively. Based on the first plurality of first features 506 1–3 And the second and multiple second features 516 1–3 Multiple displacement vectors can be determined. (524) 1–2 The displacement vector describes the displacement of a feature relative to the image. In an intermediate step, corresponding feature pairs can be determined; for example, feature 5061 can be associated with feature 5161, feature 5062 can be associated with feature 5162, and feature 5063 can be associated with feature 5163. For example, corresponding feature pairs can be determined based on feature parameters such as local contrast or angular sharpness, or based on the distance between features in the first and second corrected images. In some embodiments, not all features in the first corrected image can be paired with features in the second corrected image. In some embodiments, displacement vector 524 is determined. 1-3 The determination of corresponding feature pairs can be performed in a single step.

[0194] For example, if a minimum distance between features is mandated and the displacement is assumed to be less than the minimum distance, an algorithm that minimizes the distance between point clouds formed by the first and second features can implicitly determine the corresponding feature pairs and the displacement vector for each pair. In a typical implementation, the typical inter-frame displacement is only a few pixels. In one embodiment, multiple displacement vectors can be filtered to exclude potential outliers, such as displacement vectors that deviate from displacement vectors originating from nearby features by more than a predetermined amount.

[0195] In other embodiments, the displacement vector can be determined based on associated pixel groups, or based on pixel values ​​in the first and second corrected images. As explained above with reference to Figure 3, feature detection can be omitted in such embodiments. Instead, a dense optical flow algorithm (e.g., the Pyramid Lucas-Kanade algorithm or the Farneback algorithm) can be used to determine the displacement vector. In principle, any method based on pixel values ​​of corresponding pixel groups can be used to determine the displacement vector.

[0196] Based on multiple displacement vectors 524 1-3 The transformation can be determined. In one embodiment, the transformation can be defined by an average displacement vector or a median displacement vector, or by another displacement vector that statistically represents multiple displacement vectors. In different embodiments, the transformation can be an affine transformation, a projection transformation, or a homography transformation that combines, for example, translation, rotation, scaling, and shearing transformations. Preferably, the transformation maps features in the first corrected image to corresponding features in the second corrected image. This transformation can then be applied to the first original speckle image to register the first original speckle image with the second original speckle image.

[0197] In one embodiment, the first and second corrected images can be preprocessed before feature determination. For example, overexposed and / or underexposed regions can be identified based on pixel values. These regions can then be masked so that features cannot be detected in them. The mask can be slightly larger than the overexposed or underexposed regions, for example, by growing the identified regions with a predetermined number of pixels. Masking overexposed and / or underexposed regions can improve feature quality because masking prevents features from being associated with, for example, the edges or corners of the overexposed regions.

[0198] Figure 6A An example of determining a transformation according to an embodiment of the present invention is shown, wherein the determined transformation is a translation. See reference... Figure 5 The explanation is that the first plurality of first features 602 can be determined in the first corrected image obtained at t=t1. 1-3 Furthermore, a second plurality of second features 604 can be determined in the second corrected image obtained at t = t2. 1-3Based on the corresponding feature pairs, multiple displacement vectors 606 can be determined. 1-3 For clarity, only features and displacement vectors are shown, not (anatomical) structures.

[0199] Under normal circumstances, the determined displacement vector is 606. 1–3 The values ​​will not be exactly the same. In the example shown, displacement vector 6061 is slightly shorter than the average, while displacement vector 6063 is slightly larger than the average. Similarly, the directions of the displacement vectors show some variation. Based on the displacement vectors, the average displacement vector 608 can be determined. Translation can be defined by a single vector. For example, all pixels of the first original speckle image obtained at t = t1 can be offset by an amount equal to the average displacement vector. In principle, translation can be determined based on a single displacement vector. However, by determining multiple displacement vectors, the accuracy of the transformation can be improved.

[0200] In one embodiment, the average displacement vector 608 and the determined displacement vector 606 can be calculated, for example, based on the change in the displacement vector. 1-3 The similarity between them. In this way, an indication can be obtained, that is, how much the transformation compensates for the detected displacement of a single feature pair. Alternatively, the average distance between features in the transformed first corrected image and corresponding features in the second corrected image can be determined.

[0201] Figure 6B An example of determining a transformation according to an embodiment of the present invention is shown, wherein the determined transformation is an affine transformation. Similar to... Figure 6A It is possible to determine the first or multiple first features 612 1-3 , second multiple second features 614 1-3 and multiple displacement vectors 616 1-3 However, in this example, the average displacement vector 618, which has a length of almost zero, does not represent the determined displacement vector, which is typically longer and points in a different direction.

[0202] Therefore, to compensate for this motion, more general transformations, such as affine transformations, are needed. Affine transformations include translation, rotation, mirroring, scaling, and shearing transformations, as well as combinations thereof. Transformations can be selectively excluded by restricting the values ​​of the transformation parameters. For example, mirroring might be excluded as a possible transformation because it is generally impossible to mirror in reality.

[0203] Typically, an affine transformation can be computed using a transformation matrix with six degrees of freedom, as described in equation (1), which acts on points expressed in homogeneous coordinates. By restricting the potential of the affine transformation matrix, the affine transformation can be limited to only predefined operations. For example, more specific transformation matrices can be obtained, such as rotation matrices, which may be more suitable for certain applications.

[0204]

[0205] In the formula, A is the transformation matrix used to transform a point p with coordinates x and y (usually pixel coordinates) to a transformed point p' with coordinates x' and y'. Matrix A includes six free parameters, where t x and t y Translation is defined, while submatrices are defined. Reflection, rotation, scaling, and / or shearing can be defined. In this case, the transformation size can be based, for example, on the norm of the transformation matrix A, or the norm of the matrix AI, where I is the identity matrix.

[0206] To solve equation (1), at least three displacement vectors can be used to provide a solvable system of six equations and six unknowns. Such a linear system can be solved in a deterministic manner known in the art. In a typical embodiment, many displacement vectors can be determined, each of which may include a small error. Therefore, a more robust approach could be to use multiple displacement vectors and employ an appropriate fitting algorithm, such as least-squares fitting as shown in equation (2).

[0207] [A] = arg(min) A (∑ i ||p' I -Ap i || 2 (2)

[0208] The reliability of the determined transformation can be re-determined, as referenced above. Figure 6A As explained.

[0209] Figure 6C An example of determining a transformation according to an embodiment of the present invention is shown, wherein the determined transformation is a projection transformation. Projection transformations include and are more general than affine transformations. Projection transformations include, for example, tilt transformations. They may be needed to compensate for, for example, angular changes between a camera and a target region.

[0210] Similar to Figure 6A and 6B It is possible to determine the first or multiple first features 622 1-3 , second multiple second features 624 1-3and multiple displacement vectors 626 1-3 In this example, the translation on the left side of the image is much smaller than the translation on the right side. Therefore, applying an average translation would over-transform the pixels on the left and under-transform the pixels on the right. This displacement can be corrected by a projection transformation. Various methods are known in the art for determining a projection transformation based on four or more translation vectors.

[0211] Generally, projection transformation can be calculated using a projection matrix, as described in equation (3), using non-uniform coordinates (x1, y1, z1) and (x2, y2, z2).

[0212]

[0213] The non-uniform coordinates (x1, y1, z1) can be correlated with the pixel coordinates (x, y) of the feature by x1 = x, y1 = y, z1 = 1, while the transformed pixel coordinates (x', y') can be obtained from the non-uniform coordinates by x' = x2 / z2, y' = y2 / z2. In some embodiments, the displacement vector can be determined by (x' - x, y' - y). In other embodiments, the displacement vector is not explicitly constructed.

[0214] Therefore, equation (3) can be rewritten as two independent equations, as shown in equation (4).

[0215] x'(h 31 x+h 32 y+h 33 ) = h 11 x+h 12 y+h 13 (4)

[0216] y'(h 31 x+h 32 y+h 33 ) = h 21 x+h 22 y+h 23

[0217] Solving for H, equation (4) can be rewritten as equation (5).

[0218] v x h = 0

[0219] v y h = 0 (5)

[0220] in,

[0221] h=(h 11 ,h 12 ,h 13 ,h 21 ,h22 ,h 23 ,h 31 ,h 32 ,h 33) T

[0222] v x =(-x,-y,-1,0,0,0,x'x,x'y,x') (6)

[0223] v y =(0,0,0,-x,-y,-1,x',y,x')

[0224] Using a set of N feature points, a system of equations can be established, as shown in equation (7):

[0225] Vh=0 (7)

[0226] in,

[0227] V = (v x1 ,v y1 ,v x2 ,v y2 ,…,v xN ,v yN ) T (8)

[0228] By definition, matrix H is homogeneous and can be scaled by any constant; therefore, the projection transformation matrix contains eight degrees of freedom. Thus, equation (7) can be solved using only four sets of coordinates provided by the four features. The projection transformation matrix H can be normalized using, for example, equation (9) or equation (10):

[0229] h 33 =1 (9)

[0230] h 11 2 +h 12 2 +h 13 2 +h 21 2 +h 22 2 +h 23 2 +h 31 2 +h 32 2 +h 33 2 =0 (10)

[0231] Equation (7) can be solved deterministically using at least four points, but similar to the explanation above regarding affine transformations, using more points yields more robust results. In the case of projection transformations or homography transformations, singular value decomposition (SVD) can be used to solve equation (7). In some embodiments, the step of determining the displacement vector or point pair is combined with the step of determining homography to find the most accurate projection transformation, which can be done using algorithms such as RANSAC.

[0232] In one embodiment, the algorithm may first compute a relatively simple transformation, such as translation. The algorithm can then determine whether the computed transformation reproduces the determined displacement vector with sufficient accuracy. If not, the algorithm may try a more general transformation, such as an affine transformation, and repeat the same process. If translation is sufficient in enough cases, this can reduce the required computation time. In different embodiments, the algorithm may always compute a general transformation, such as always computing a general homography transformation. This can lead to more accurate registration of the original speckle image or speckle contrast image.

[0233] Figure 6D An example of determining multiple transformations according to an embodiment of the present invention is shown. Each corrected image 650 in a series of corrected images can be divided into multiple regions 652. 1–n Preferably, multiple non-intersecting regions, such as a rectangular grid, are combined to cover the entire image. Subsequently, each region 6541, 6542, ..., 654 can be designated as a separate region for the entire image. n Determine transformations 6541, 6542, ..., 654 n Preferably, it is in accordance with the above reference. Figures 6A to 6C The same approach is used. Subsequently, each region can be transformed using the transformation determined for that region. Alternatively, the determined transformation can be assigned to, for example, the center pixel in the region, and the remaining pixels can be transformed according to an interpolation scheme based on the transformation of the region including that pixel and the transformations of neighboring regions.

[0234] In one embodiment, each region can be a single pixel. In such an embodiment, the transformation can be determined based on the pixel value and the pixel values ​​of pixels in the region surrounding the pixel.

[0235] Figure 7A and 7BA flowchart illustrating laser speckle contrast imaging combining two or more original speckle images according to an embodiment of the present invention is shown. At a first time instance t = t1, a first original speckle image 7021 based on light of a first wavelength can be obtained, and the first original speckle image 7021 can be used to calculate a first laser speckle contrast image 7041. A first corrected image 7061 based on light of at least a second wavelength and associated with the first original speckle image can be obtained, and a first plurality of first features 7081 can be detected in the first corrected image. Items 7021–7081 can be obtained by performing step 7101, which can be similar to steps 302–308 described with reference to FIG. 3. In some embodiments, the first corrected image and the first original speckle image can be the same image.

[0236] At the second time instance t = t2, step 7101 can be repeated as step 7102 to generate a second original speckle image 7022 based on light of the first wavelength, a second laser speckle contrast image 7042, a second corrected image 7062 based on light of at least the second wavelength, and a plurality of second features 7082. Based on the plurality of first features 7081 and the plurality of second features 7082, a plurality of first displacement vectors 7121 can be calculated. Based on the plurality of first displacement vectors, a first transformation 7141 can be determined, which can be used to transform the first laser speckle contrast image 7041 to register the first laser speckle contrast image with the second laser speckle contrast image 7042, thereby generating a first registered laser speckle contrast image 7181. As described above, in some embodiments, the transformation can be determined without explicitly detecting features and / or displacement vectors.

[0237] Optionally, a first weight 7161 can be determined based on a plurality of first displacement vectors 7121. The weight can be related to the length of a representative displacement vector (e.g., the maximum, average, or median displacement vector), preferably inversely related, or related to, for example, the average or median length of the plurality of first displacement vectors. (Refer to the above...) Figure 6D As explained, in embodiments where an image is divided into multiple regions and a transformation is determined for each region, weights can be determined for each region based on transformation parameters associated with that region, or a single weight can be determined for the entire image, for example, based on representative parameters such as maximum, average, or median shifts. Reference will be made below. Figure 9 The determination of the weighted average will be discussed in more detail.

[0238] Then, the first registered laser speckle contrast image 7181 can be combined with the second laser speckle contrast image 7042 to generate a first combined laser speckle contrast image 7201. The combined laser speckle contrast image can be, for example, the pixel average or maximum value of the first registered laser speckle contrast image 7181 and the second laser speckle contrast image 7042. Optionally, a weighted average can be determined using a first weight 7161 and / or a second weight 7162.

[0239] At the third time instance t = t3, step 7101 can be repeated as step 7103 to generate a third original speckle image 7023 based on light of the first wavelength, a third laser speckle contrast image 7043, a third corrected image 7063 based on light of at least the second wavelength, and a plurality of second features 7083. Based on the plurality of second features 7082 and the plurality of third features 7083, a plurality of second displacement vectors 7122 can be calculated. Based on the plurality of second displacement vectors, a second transformation 7142 can be determined. The second transformation can be used to transform the first combined laser speckle contrast image 7201 to register the first combined laser speckle contrast image with the third laser speckle contrast image 7043, thereby generating a first registered combined laser speckle contrast image 7221. Then, the first registered laser speckle contrast image 7181 can be combined with the second laser speckle contrast image 7042 to generate the first combined laser speckle contrast image 7201.

[0240] Then, the first registered combined laser speckle contrast image 7201 can be combined with the third laser speckle contrast image 7043 to produce a second combined laser speckle contrast image 7202. Therefore, the second combined laser speckle contrast image can include information from the first laser speckle contrast image 7041, the second laser speckle contrast image 7042, and the third laser speckle contrast image 7043. By repeating these steps, the nth image can include information from all the previous n-1 images. Preferably, the weighting can then be skewed to give higher (compared to older images) weights to the temporally most recent images. This embodiment is particularly useful for streaming video, where each captured frame is processed and output with minimal latency. Another advantage of this method is that relatively small motion can be assumed between two subsequent frames, which can speed up processing and allow for the use of a wider range of algorithms, as some algorithms may not work well for larger motions. Generally, feature-based algorithms may be more reliable for relatively large displacements.

[0241] Figure 7BA flowchart illustrating an alternative method for laser speckle contrast imaging by combining two or more original speckle images according to an embodiment of the present invention is shown. In this embodiment, a predetermined number of images are combined into a single combined image. Step 752 relates to image acquisition, calculation of the speckle contrast image, and determination of features. 1–n -760 1–n See above for reference Figure 7A Explanation of step 702 1–n -710 1–n same.

[0242] However, with Figure 7A The methods shown are different; all displacement vectors are 762. 1,2 These are all determined relative to a single reference image (e.g., the first or last image in a sequence of n images). In the example shown, the image is registered with the nth image. Therefore, by transforming 764... 1、2 The first and second speckle contrast images are applied respectively, and then registered with the nth speckle contrast image. Therefore, the weight is 766. 1,2 It is also determined based on transformation parameters (e.g., the average displacement relative to the nth image). In the final step, all n registered speckle contrast images can be combined into a single combined speckle contrast image 770.

[0243] Figure 7B The method shown may result in combined images having a higher resolution than those shown. Figure 7A The method shown offers higher image quality, but at the cost of a larger latency between image capture and display. Therefore, this method is particularly advantageous for recording snapshots.

[0244] In one embodiment, Figure 7B The method shown can be applied to sliding groups of images, such as the last n images of a video feed. To maintain low latency, n should ideally not be too large in this case; for example, when the frame rate is, for example, 50–60 fps, n can be approximately 5–20. Of course, the number of frames that can be processed also depends on the hardware and algorithm, so a larger number of frames is still feasible. Therefore, some advantages of the two methods can be combined. The advantage of using a relatively small number of frames is that dynamic phenomena (such as the effects of heartbeats) can be imaged. The advantage of a larger number of frames is that such transient effects can be filtered out, especially when the number of frames is chosen to cover an integer multiple of the heartbeat and / or respiratory cycle.

[0245] Figure 8A flowchart illustrating the computational correction of a laser speckle contrast image according to an embodiment of the present invention is shown. Typically, one might be interested in the relative motion of one or more objects in a target region with respect to one or more other objects in the target region (e.g., the motion of bodily fluids or red blood cells relative to tissue). In this case, noise in the signal obtained from the moving object (the desired signal) can be compensated for by a signal obtained from a reference object (the reference signal). The basic principle is that the desired signal can include a first component based on the motion of the quantity of interest relative to the reference object and a second component based on the motion of the entire target region relative to the camera. The reference signal may include only or primarily the component based on the motion of the entire target region relative to the camera. Therefore, the reference signal can be correlated with the second component of the desired signal. This correlation can be used to correct or compensate for the desired signal. For example, a correction term based on the signal strength of the reference signal can be added to the desired signal.

[0246] exist Figure 8 In the illustrated embodiment, target region 802 is illuminated with coherent light of a first wavelength (e.g., red or infrared light), and target region 812 is illuminated with coherent light of a second wavelength (e.g., green or blue light). Preferably, the light of the first wavelength is mostly scattered by the object of interest or fluid (e.g., blood). Preferably, the light of the second wavelength is mostly scattered by the surface of the target region and / or mostly absorbed by the object of interest or fluid. Preferably, the second wavelength is selected such that the second wavelength is reflected by blood at least 25%, at least 50%, or at least 75% less than the second wavelength is reflected by tissue. Preferably, the target region is illuminated simultaneously with light of the first wavelength and light of the second wavelength.

[0247] Scattered light of a first wavelength can generate a first raw speckle image, which can be captured by a first image sensor 804. Scattered light of a second raw speckle image can be captured by a second image sensor 814, the second image sensor preferably being different from the first image sensor. The first raw speckle image can be referred to as the desired signal raw speckle image, while the second raw speckle image can be referred to as the reference signal image or the correction signal image.

[0248] Based on the first original speckle image, a first speckle contrast image (806) can be calculated. Based on the second original speckle image, a second speckle contrast image (806) can be calculated. Preferably, the speckle contrast is calculated in the same manner as the first and second original speckle images. For example, it can be calculated according to the above-mentioned reference Figure 1 and... Figure 3A The method explained in step 304 is used to calculate speckle contrast.

[0249] In the next step 808, a corrected speckle contrast image can be calculated based on the first speckle contrast image and the second speckle contrast image. Calculating the corrected speckle contrast image may include, for example, adding a correction term or multiplying by a correction factor. The correction term or correction factor may be based on an amount of speckle contrast determined in the speckle contrast image compared to a reference amount of speckle contrast. The reference amount of speckle contrast may be, for example, predetermined, or may be dynamically determined based on, for example, the amount of speckle contrast in multiple previous second contrast images, or dynamically determined based on a speckle contrast image with very little motion determined by, for example, a motion correction algorithm as described above.

[0250] Then, the corrected speckle contrast image can be stored 810 for further processing, for example, as a reference. Figure 3A and Figure 3B The reregistration or realignment and time averaging are explained. The second original speckle image and / or the second speckle contrast image can also be stored 818 for further processing, such as determining displacement vectors in multiple second original speckle images to reregister or realign multiple simultaneously captured corrected speckle contrast images. In one embodiment, steps 802–818 can replace… Figure 3B Steps 332–336 in the text.

[0251] Therefore, one advantage of the method in this disclosure is that the second wavelength image can be obtained as described in the reference. Figure 8 The explanation provided, used for multispectral coherence correction (or "dual-laser correction"), can also be found in the reference. Figure 3A and Figure 3B The explanation is used for registering speckle contrast images.

[0252] Figure 9 The diagram schematically illustrates motion-compensated speckle contrast images determined based on a weighted average according to an embodiment of the present invention. For each corrected image 902 1–N The transformation magnitude can be determined based on the parameters of the transformation as defined and / or based on multiple displacement vectors. For example, the transformation magnitude can be based on the lengths or statistical representations of multiple displacement vectors (e.g., the average value of multiple displacement vectors). 1–N ), or based on the matrix norm of the matrix representing the transformation.

[0253] The combined speckle contrast image 906 can be the weighted average of the speckle contrast images in the registered speckle contrast image sequence, with each image weighted by a weight parameter w'.

[0254] The weight parameter w' can be based on the displacement vector ||p' i -p i || to determine that a high displacement corresponds to a small weight, and vice versa. For example, the weight parameter w' is defined using equation (11):

[0255]

[0256] In the formula, the displacement vector can be represented by a total of P points p i 'and point p i Definition: P points p i 'Based on the coordinates (x) of the reference image i ',y i ') Define point p i The coordinates (x, y) on the image to be transformed and registered to the reference image i ,y i )definition.

[0257] The weighting parameter w' can also be determined based on dense or sparse optical flow parameters, which define the optical flow between the first image (typically a reference image) and the second image. For example, the weights can be inversely correlated with the average optical flow over all pixels in the image or over the region of interest in the image, as defined in equation (12):

[0258]

[0259] In the formula, v ij ' is the optical flow of pixel (i, j) including the x and y components of the optical flow, and W and H are the width and height of the pixels in the reference image or the reference region of interest, respectively.

[0260] Alternatively, for example, weight parameters can be determined for each pixel based on the optical flow of each pixel as defined in equation (13):

[0261]

[0262] Instead of temporarily determining weight parameters for each pixel or image, weight parameters can also be determined for predefined regions of the image (such as rectangular or triangular grids). For example, weight parameters can be based on the average optical flow within the corresponding predefined region.

[0263] The weighting parameter can also be based on the speckle contrast value s ij To determine, preferably, the weighting parameter is proportional to the magnitude of the speckle contrast, for example, as defined in equation (14):

[0264]

[0265] In the formula, s ij' is the speckle contrast of the reference image, and W and H are the width and height of the pixels in the reference image or the region of interest, respectively. The advantage of using speckle contrast is that any noise appearing within a small time window of the speckle contrast will blur it. This noise could be due to, for example, motion of the image object or camera, or other sources such as loose fiber optic connections. The advantage of using weights based on displacement vectors or optical flow is that they can have higher LSCI temporal resolution, while speckle contrast-based weights may lag, especially with increased perfusion.

[0266] Alternatively, the weighting parameters can be determined based on the speckle contrast of each pixel.

[0267]

[0268] Alternatively, the weighting parameters can be determined based on the average speckle contrast within a predefined region, such as a square or triangular grid. Weighting parameters can also be determined for dynamically defined regions, where the regions can be determined, for example, based on detected motion.

[0269] In one embodiment, various weights among these weights can be combined. For example, weights can be normalized and added, multiplied, or compared (e.g., selecting the lowest weight). Alternatively, a first weight, such as based on the value of speckle contrast, can be used to filter out images that do not meet predefined quality criteria; for example, images whose contrast decreases by more than a predetermined threshold receive a weight of 0, and all other images receive a weight of 1. Subsequently, a weighted average of the unfiltered images can be determined using weights based on optical flow or displacement.

[0270] By using N images Img k The buffer and the corresponding N weight factors w k A buffer, where k = 1, 2, ..., N, can be used to determine the weighted average using a single weight factor for each image. For each new image acquired, the following steps can be performed:

[0271] 1) Add the new image to the buffer as Img N+1 It can be selected as a reference image;

[0272] 2) Remove the first weight factor w1 and the first image Img1 from the buffer;

[0273] 3) Apply geometric transformations to the other images Img2, Img3, ..., Img in the buffer. N To register them to the reference image Img N+1 If the image has been previously registered to, for example, a previous reference image, the same transformation can be applied to all images.2–N If unregistered images are stored in a buffer, a transformation can be determined for each image in the buffer and applied to each image in the buffer.

[0274] 4) Calculate the weighting factor w as described above. N+1 And add it to the buffer;

[0275] 5) Normalize the weighting factor w2 to w N+1 For example, according to equation (16):

[0276]

[0277] Or according to equation (17):

[0278]

[0279] In the formula, β1 is a positive constant or a negative constant, and w min Defined as the minimum weight in the buffer, β2 should be a constant greater than zero. To avoid negative weights and division by zero, any negative or zero weight can be set to a small positive number. (Using w...) min The second advantage of normalization is that it increases the influence of the weighting factors. Increasing β1 or β2 will reduce the influence of the weighting factors and make the algorithm behave more like an averaging algorithm, while decreasing β1 or β2 will increase the influence of the weighting factors. These constants can be predetermined according to the application.

[0280] 6) Calculate a weighted average using the previously calculated weights to compute the high-quality composite image Img. N+1 For example, as defined in equation (18):

[0281]

[0282] In the formula, i and j are used to index the pixels of the image.

[0283] In an alternative embodiment, the image buffer may consist only of the combined image. In such an embodiment, this can be achieved by using N weighting factors w. k The buffer and the combined image Img based on N most recent images in time N To determine the weighted average, we use k = 1, 2, ..., N, which corresponds to the N most recent images in time. For each new image Img... N+1 The following steps can be taken:

[0284] 1) Add the new image to the buffer as Img N+1 It can be selected as a reference image;

[0285] 2) Remove the first weighting factor w1 from the buffer;

[0286] 3) For the stored image Img N Apply geometric transformations to register it to the reference image Img N+1 .

[0287] 4) Calculate the weighting factor w as described above. N+1 And add it to the buffer;

[0288] 5) Normalize the weighting factor w2 to w N+1 For example, according to equation (16) or (17);

[0289] 6) Calculate the stored image Img N 'and reference image Img N+1 The weighted average of the values ​​is used to calculate the high-quality composite image Img. N+1 For example, as defined in equation (19):

[0290] Img' N+1 (i,j)=(1-w N+1 )·Img N (i,j)+w N+1 ·Img N+1 (i,j) (19)

[0291] In the formula, i and j are used to index the pixels of the image.

[0292] 7) Remove 1 mg from the buffer N 'and Img N+1 Add to buffer.

[0293] The advantage of this algorithm is its faster computation speed, as it only requires applying a single geometric transformation and involves fewer processing steps. The size of the buffer storing the weight factors determines how much influence the historical image has compared to the new image. When the buffer is small, the new image has a stronger presence in the final image, while a large buffer results in a weaker presence of the new image and a stronger presence of images with high weight factors.

[0294] As mentioned above, weights can be determined for the entire image, for each pixel in the image, or for a predetermined or dynamically determined region in the image. Similarly, a single transformation can be applied to each image as a whole, each pixel can be transformed individually, or the transformation can be determined to be applied to a predetermined or dynamically determined region. Therefore, weights can be defined as scalars, matrices, or different formats.

[0295] The advantage of using algorithms based on geometric transformations such as rectangular or triangular meshes and using weights determined for each mesh segment is that such algorithms are more robust to registration errors while still being able to locally correct for noise such as local motion.

[0296] Weights based on displacement or transformation can be quickly determined for each image independently of other images. Images with large displacements are often noisy and can therefore be assigned lower weights, thus improving the quality of the combined image.

[0297] Alternatively or additionally, for each first original speckle image, a normalized amount or a change in speckle contrast relative to one or more previous and / or subsequent images in the first speckle contrast image sequence can be determined. In this case, a weighted average can be determined based on the weights of the determined normalized amount or the determined change in speckle contrast associated with each of the first speckle contrast images.

[0298] Weights based on differences or variations (especially abrupt changes) in speckle contrast can indicate image quality. Generally, speckle contrast, and therefore these weights, can be influenced by various factors throughout the system, such as camera motion relative to the target area, fiber motion, or other factors affecting optical path length, loose connections, or fluctuations in illumination conditions. Therefore, using speckle contrast-based weights can yield higher-quality composite images. Typically, speckle contrast is determined in relative units, so weights can be determined by analyzing the original speckle image sequence. Since speckle contrast is negatively correlated with perfusion, perfusion units based on speckle contrast can be used similarly.

[0299] Preferably, the image can be normalized in a linear manner where the relationship between speckle contrast and weights is constant. In this case, since normalization can be performed directly on each image without referring to the image's time window, speckle contrast-based correction can be more real-time. Alternatively, incremental averaging can also be used.

[0300] Figure 10This is a block diagram illustrating an exemplary data processing system described in this disclosure. The data processing system 1000 may include at least one processor 1002 coupled to a storage element 1004 via a system bus 1006. Thus, the data processing system can store program code within the storage element 1004. Furthermore, the processor 1002 can execute program code accessed from the storage element 1004 via the system bus 1006. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that the data processing system 1000 may be implemented in the form of any system, including processors and memories capable of performing the functions described in this specification.

[0301] Storage element 1004 may include one or more physical storage devices, such as local memory 1008 and one or more mass storage devices 1010. Local memory may refer to random access memory or other non-persistent storage devices typically used during the actual execution of program code. Mass storage devices may be implemented as hard disk drives or other persistent data storage devices. Processing system 1000 may also include one or more cache memories (not shown) that provide temporary storage for at least some program code to reduce the number of times program code must be retrieved from mass storage device 1010 during execution.

[0302] Optionally, the input / output (I / O) devices, shown as button device 1012 and output device 1014, can be coupled to the data processing system. Examples of button devices may include, but are not limited to, keyboards, directional devices such as mice, etc. Examples of output devices may include, but are not limited to, monitors or displays, speakers, etc. The button devices and / or output devices can be coupled to the data processing system directly or through an intervening I / O controller. Network adapter 1016 can also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intervening private or public network. The network adapter may include a data receiver for receiving data sent to the data receiver by the system, device, and / or network, and a data transmitter for transmitting data to the system, device, and / or network. Operating modems, cable-operated modems, and Ethernet cards are examples of different types of network adapters that can be used with the data processing system 1000.

[0303] like Figure 10As shown, storage element 1004 can store application program 1018. It should be understood that data processing system 1000 can further execute an operating system (not shown) that facilitates the execution of the application program. The application program, implemented in the form of executable program code, can be executed by data processing system 1000, for example, by processor 1002. In response to executing the application program, the data processing system can be configured to perform one or more operations, which will be described in further detail herein.

[0304] In one aspect, for example, data processing system 1000 may represent a client data processing system. In this case, application 1018 may represent a client application that, when executed, configures data processing system 1000 to perform the various functions described herein with reference to a "client". Examples of clients may include, but are not limited to, personal computers, portable computers, mobile phones, etc.

[0305] In another aspect, the data processing system may represent a server. For example, the data processing system may represent an (HTTP) server, in which case the application 1018, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function mentioned in this specification.

[0306] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are also intended to include the plural forms. It should be further understood that when the terms “comprise” and / or “comprising” are used in this specification, the presence of the stated features, integritys, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integritys, steps, operations, elements, components, and / or groups thereof is not excluded.

[0307] All the means or steps plus functional elements in the following claims are intended to include any structure, material, action, and equivalent for performing a function in combination with other claimed elements of the specific claim. The description of the invention is for illustrative and descriptive purposes only and is not intended to be exhaustive or limiting to the invention as disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described to best explain the principles and practical application of the invention and to enable others skilled in the art to understand the various embodiments of the invention and to make various modifications according to the particular purpose contemplated.

Claims

1. A method of motion compensated laser speckle contrast imaging, comprising: exposing a target region to coherent first light of a first wavelength, the target region comprising living tissue; capturing at least one image sequence, the at least one image sequence comprising first speckle images, the first speckle images being captured during exposure to the first light; determining one or more transformation parameters of an image registration algorithm for registering the first speckle images to each other, the transformation parameters being based on a similarity measure of pixel values of at least one group of pixels in each of a plurality of images of the at least one image sequence, the images of the plurality being selected from the first speckle images or being associated with the first speckle images; determining first speckle contrast images based on the first speckle images; determining registered speckle contrast images by registering the first speckle contrast images based on the one or more transformation parameters and the image registration algorithm; and determining a combined speckle contrast image by calculating a weighted average of pixel values of the registered first speckle contrast images.

2. The method of claim 1, further comprising: exposing the target region to second light of one or more second wavelengths, wherein exposure to the second light is performed alternately or simultaneously with exposure to the first light; wherein the at least one image sequence comprises a second image sequence, the second images being captured during exposure to the second light; and wherein the plurality of images are selected from the second image sequence, each of the second images being associated with a first speckle image.

3. The method of claim 2, the second light being coherent light of a second wavelength or the second light being light of a plurality of second wavelengths comprising the visible spectrum.

4. The method of claim 2, wherein, the light of the one or more second wavelengths being coherent light of a predetermined second wavelength, and wherein the second image sequence is a sequence of second speckle images; the method further comprising: determining second speckle contrast images based on the sequence of second speckle images; and adjusting the first speckle contrast images based on variations in speckle contrast magnitude in the sequence of second speckle contrast images.

5. The method of claim 4, wherein the predetermined second wavelength is in the green or blue portion of the electromagnetic spectrum.

6. The method of claim 4, wherein the predetermined second wavelength is in the range of 380-590 nm.

7. The method of claim 4, wherein the predetermined second wavelength is in the range of 470-570 nm.

8. The method of claim 4, wherein the predetermined second wavelength is in the range of 520-560 nm.

9. The method of claim 1, wherein, the first wavelength being a wavelength in the red portion of the electromagnetic spectrum or the first wavelength being a wavelength in the infrared portion of the electromagnetic spectrum.

10. The method of claim 9, wherein, the wavelength in the red portion of the electromagnetic spectrum being in the range of 600-700 nm.

11. The method of claim 9, wherein, the wavelength in the red portion of the electromagnetic spectrum being in the range of 620-660 nm.

12. The method of claim 9, wherein, the wavelength in the infrared portion of the electromagnetic spectrum being in the range of 700-1200 nm.

13. The method of claim 1, wherein the weight of an image is based on the transformation parameter or on a relative size of speckle contrast.

14. The method of claim 1, wherein, The pixel groups represent a plurality of predetermined features in the plurality of images.

15. The method of claim 14, wherein, The plurality of predetermined features are associated with an object in the target region.

16. The method of claim 15, wherein, The object in the target region is an anatomical structure.

17. The method of claim 14, further comprising: filtering the plurality of images with a filter adapted to increase a probability that a pixel group represents a feature corresponding to an anatomical feature.

18. The method of claim 1, wherein, Determining one or more transformation parameters comprises: based on the similarity measure, determining a plurality of associated pixel groups, each pixel group belonging to a different image from the plurality of images; based on positions of the pixel groups relative to respective images in the plurality of images, determining a plurality of displacement vectors representing motion of the target region relative to an image sensor; and based on the plurality of displacement vectors, determining the transformation parameter.

19. The method of claim 2, further comprising: dividing each of the first speckle images, each of the first speckle contrast images, and each of the plurality of images into a plurality of regions, respectively; and wherein Determining transformation parameters comprises determining a transformation parameter for each region; and respectively determining a sequence of registered first speckle images and a sequence of first speckle contrast images comprises respectively registering each region of the first speckle images and each region of the first speckle contrast images based on the transformation, wherein the transformation is based on corresponding regions in the second image.

20. The method of claim 19, wherein the plurality of regions are disjoint regions.

21. The method of claim 1, wherein, The target region comprises a perfused organ, and / or the target region comprises one or more blood vessels and / or lymphatic vessels, the method further comprising: based on the combined speckle contrast image, calculating a perfusion intensity.

22. The method of claim 21, wherein, The perfused organ is perfused by a bodily fluid.

23. The method of claim 21, wherein, The perfused organ is perfused by blood and / or lymphatic fluid.

24. The method of claim 21, wherein, The perfusion intensity is a blood perfusion intensity or a lymph perfusion intensity.

25. The method of claim 1, further comprising: displaying the combined speckle contrast image or an image derived from the combined speckle contrast image.

26. A hardware module for an imaging device, the hardware module comprising: a first light source to expose a target region to coherent first light at a first wavelength, the target region comprising living tissue; at least one image sensor system to capture at least one sequence of images, the at least one sequence of images comprising first speckle images, the first speckle images captured during exposure to the first light; a computer readable storage medium having computer readable program code and a processor coupled to the computer readable storage medium, wherein in response to executing the computer readable program code, the processor is configured to: determining one or more transformation parameters of an image registration algorithm for registering the first speckle images with each other, the transformation parameters being based on a similarity measure of pixel values of at least one group of pixels in each of a plurality of images, the images of the plurality being selected from or associated with the first speckle images; determining first speckle contrast images based on the first speckle images; determining registered speckle contrast images by registering the first speckle contrast images based on the one or more transformation parameters and the image registration algorithm; and determining a combined speckle contrast image by calculating a weighted average of pixel values of the registered first speckle contrast images.

27. The hardware module of claim 26, wherein, The imaging device is a medical imaging device.

28. The hardware module of claim 26, wherein, The processor is a microprocessor.

29. The hardware module of claim 26, wherein, The processor is a graphics processing unit.

30. The hardware module of claim 26, further comprising: a second light source for illuminating the target region with light of at least a second wavelength simultaneously or alternately with the first light source, the second wavelength being different from the first wavelength; wherein the at least one image sensor is further configured to capture a second sequence of images, the second images being captured during exposure with the second light; and wherein the plurality of images are selected from the second sequence of images, each of the second images being associated with a first speckle image.

31. The hardware module of claim 26, further comprising: a display for displaying the combined speckle contrast image and / or an image derived from the combined speckle contrast image.

32. The hardware module of claim 31, wherein the image derived from the combined speckle contrast image is a perfusion intensity image.

33. A medical imaging device comprising the hardware module of any one of claims 26-32.

34. The medical imaging device of claim 33, wherein the device is one of: an endoscope, a surgical robot, a handheld laser speckle contrast imaging device, or an open surgery laser speckle contrast imaging system.

35. The medical imaging device of claim 34, wherein the endoscope is a laparoscope.

36. A computing module for a laser speckle imaging system, the computing module comprising a computer readable storage medium having at least a portion of a program and a processor coupled to the computer readable storage medium, wherein, In response to executing the computer-readable storage code, the processor is configured to perform executable operations comprising: receiving at least one sequence of images, the at least one sequence of images comprising a plurality of first speckle images, the first speckle images being captured during exposure of a target region to coherent first light of a first wavelength, the target region comprising living tissue; determining one or more transformation parameters of an image registration algorithm for registering the first speckle images with each other, the transformation parameters being based on a similarity measure of pixel values of at least one group of pixels in each of a plurality of images of the at least one sequence of images, the images of the plurality being selected from or associated with the first speckle images; determining first speckle contrast images based on the first speckle images; determining registered speckle contrast images by registering the first speckle contrast images based on the one or more transformation parameters and the image registration algorithm; and determining a combined speckle contrast image by calculating a weighted average of pixel values of the registered first speckle contrast images. A combined speckle contrast image is determined by calculating a weighted average of pixel values of the registered first speckle contrast image.

37. The computing module of claim 36, wherein, The processor is a microprocessor.

38. The computing module of claim 36, wherein, The processor is a graphics processing unit.

39. The computing module of claim 36, wherein, the at least one image sequence comprises a second image sequence, the second images being captured during exposure to second light having one or more second wavelengths, wherein the exposure to the second light is performed alternately or simultaneously with the exposure to the first light; and wherein the plurality of images are selected from the second image sequence, each of the second images being associated with a first speckle image.

40. The computing module of claim 39, wherein, The second light is coherent light of a second wavelength or the second light comprises a plurality of second wavelengths of the visible spectrum.

41. A computer readable storage medium storing a computer program or suite of computer programs comprising at least one software code portion or computer program product storing at least one software code portion, the software code portion, when run on a computer system, being configured for executing the method according to any one of claims 1-25.

Images