Speckle-based image distortion correction for laser scanning microscopy
By analyzing the speckle characteristics in the image and calculating and adjusting the movement rate of the scanning device, the image distortion problem caused by friction in the laser scanning imaging device is solved, and more robust and economical imaging results are achieved.
Patent Information
- Application Number
- CN202080025783.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-01-28
- Filing Date
- 2020-01-27
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Image distortions caused by non-uniform laser scanning pose challenges to certain laser scanning imaging devices, especially in the non-uniform movement caused by friction during rotation or scanning, resulting in geometric distortions in the image data, which are often unpredictable and unknown. .
By analyzing the shape and size of speckle or other features in the image, calculating the nonlinear relationship between scanning speed and movement rate, performing calibration to adjust the movement rate of the imaging probe, using cross-correlation minimization technology to compensate for multiple distortions Each line is corrected to correct image distortion.
Effective correction of image distortion is achieved, resulting in more robust and cost-effective results, simplifying the design and operation of imaging equipment, and improving the smoothness and regularity of the image.
Smart Images

Figure CN113678054B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 797,548, filed on January 28, 2019, the entire contents of which are incorporated herein by reference in their entirety for all purposes.
[0003] A note about federally funded research
[0004] This invention was made with government support under Grant No. R01DK091923 awarded by the National Institutes of Health. The government has certain rights in this invention. Background Art
[0005] Image distortion caused by non-uniform laser scanning can pose a challenge to certain laser scanning imaging devices, such as catheter endoscopes and capsule endoscopes. For example, components associated with rotation or scanning may have a certain amount of friction, and this friction may change during rotation, for example, depending on the orientation of the imaging device or the waveform of the scan. Since this friction is generally unknown and unpredictable, the distortion caused by friction may ultimately reside in the generated image data, for example, as a geometric distortion of the sample. Summary of the invention
[0006] Methods and systems disclosed herein for correcting distortion of an image from an image probe include techniques for measuring and correcting image distortion based on the shape and / or size of speckle or other features present in the image.
[0007] In various embodiments, the methods and systems may include one or more of the following:
[0008] 1) Determine the scanning speed based on the speckle shape and / or size;
[0009] 2) Calculating the speckle shape and / or size based on the autocovariance of the image (e.g., after the image has been filtered by a high-pass filter);
[0010] 3) performing a calibration to adjust for the non-linear relationship between the imaging probe's movement rate (e.g., rotational or linear) and the speckle shape and / or size; or
[0011] 4) Align each line in multiple distortion compensations by minimizing cross-correlation.
[0012] These correction techniques can be employed by any imaging device with rotational and / or linear scanning components. These techniques can be implemented in software to measure the amount of distortion, so products that include embodiments of the disclosed methods and systems can be simpler, more cost-effective, and produce more robust results.
[0013] In one embodiment, the present invention includes a method of correcting distortion of an image, comprising: analyzing, by a processor, an image segment of the image to identify a speckle artifact, the image segment being obtained from a scanning imaging device; determining, by the processor, an aspect ratio of a shape of the speckle artifact; determining, by the processor, a correction factor for the shape of the speckle artifact based on the aspect ratio; and adjusting, by the processor, a size of the image segment based on the correction factor.
[0014] In another embodiment, the present invention includes an apparatus for correcting distortion of an image, comprising: a processor; and a memory in communication with the processor having a set of instructions stored thereon, which, when executed by the processor, cause the processor to: analyze an image segment of the image to identify a speckle artifact, the image segment being obtained from a scanning imaging device; determine an aspect ratio of a shape of the speckle artifact; determine a correction factor for the shape of the speckle artifact based on the aspect ratio; and adjust a dimension of the image segment based on the correction factor.
[0015] In yet another embodiment, the present invention includes a method of distortion correction, the method comprising: providing an image; dividing the image into a plurality of substantially parallel strips, each strip extending in a first direction; dividing each strip into a plurality of sub-strips along a second direction substantially perpendicular to the first direction; analyzing each of the plurality of sub-strips to locate at least one local bright feature; adjusting at least a portion of each sub-strip to force the at least one local bright feature toward a predetermined shape to create a corrected sub-strip; reassembling the plurality of corrected sub-strips into a corrected strip; and reassembling the plurality of corrected strips into a corrected image.
[0016] In yet another embodiment, the present invention includes an apparatus for correcting distortion of an image, comprising: a scanning imaging device, the scanning imaging device including a coherent light source, the scanning imaging device configured to obtain an image of a sample; a processor, the processor communicating with the scanning imaging device; and a memory, the memory communicating with the processor having a set of instructions stored thereon, which, when executed by the processor, cause the processor to: analyze an image segment of the image to identify a speckle artifact; determine an aspect ratio of a shape of the speckle artifact; determine a correction factor for the shape of the speckle artifact based on the aspect ratio; and adjust a dimension of the image segment based on the correction factor. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Various objects, features, and advantages of the disclosed subject matter may be more fully understood by reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements.
[0018] Figure 1 An example system is shown that may be used to collect data for use with embodiments of the methods and systems disclosed herein.
[0019] Figure 2 shows examples of speckle before (left panel, inset) and after (right panel, inset) distortion compensation according to embodiments disclosed herein;
[0020] Figure 3 represents the relationship found between an image-based distortion estimate based on the speckle aspect ratio (x-axis) and the actual amount of distortion calculated by measuring the aspect ratio of a spherical microbead (y-axis, where the numbers on the axes are unitless ratios), with various curve fitting methods shown superimposed on the data as part of a process for calibrating the relationship between distortion determined using speckle and the actual distortion in an image;
[0021] Figure 4 Indicates Figure 3 Graphical results of the analysis shown, with the results of the bisquare curve-fitting procedure superimposed on the data;
[0022] Figure 5 Indicates Figure 3 The graphical results of the analysis are shown, where the results of the custom curve fitting procedure (based on Figure 4 The formula of the bisquare curve fitting procedure) is superimposed on the data;
[0023] Figure 6 shows how one sub-strip (the red square area in the top image) is adjusted to form a corrected sub-strip (bottom image);
[0024] Figure 7 depicts how to create smoother, more regular images using the distortion correction described herein by applying compensation in the axial scanning direction;
[0025] Figure 8 is a flow chart depicting an example sequence of operations of the present distortion correction algorithm; and
[0026] Fig. 9 A computer system is shown that implements embodiments of the methods and systems disclosed herein.
[0027] Fig.10 An example of a process for correcting image distortion according to some embodiments of the disclosed subject matter is shown.
[0028] Fig.11 An example of a process for distortion correction according to some embodiments of the disclosed subject matter is shown. DETAILED DESCRIPTION
[0029] According to some embodiments of the disclosed subject matter, mechanisms are provided (which may include systems, methods, and media) for image distortion correction.
[0030] Laser speckle (which may be considered an unwanted artifact in some cases) can be used to benefit various disclosed embodiments by using a laser speckle spot (which is expected to be substantially circular) to determine the amount and orientation of any distortion that may occur in an image based on the extent to which the speckle deviates from a substantially circular shape. In various embodiments, features of distortion and features having a particular local brightness (such as laser speckle) are used in the disclosed procedures to compensate for distortion, and these features are easily identified using the automated procedures disclosed herein. Although these features are referred to herein as laser speckle, these features may also have other sources, but are still useful for identifying and correcting image distortion.
[0031] In many imaging use environments (e.g., capsule endoscopy using an illumination laser), undesirable distortion artifacts may occur. For example, in a 360° rotating capsule endoscopy use environment, the motor that rotates the illumination laser and / or the image receiving sensor may stick and / or slip. In this case, distortion may occur in the recorded image because the rotation is not completed smoothly, because the image reconstruction from the scan data is based on the assumption that the scanned data points are equally spaced. That is, a series of data points are collected as the imaging probe rotates and / or translates through the specimen. For a spiral scan obtained when the probe rotates and / or translates through the lumen sample simultaneously, an image of the sample is generated by assembling a series of data points into a two-dimensional or three-dimensional representation based on the assumption that the probe rotates and translates at a uniform rate. However, friction within the device (e.g., friction within the motor itself or friction between the rotating optical fiber and the sheath that houses the optical fiber, among various possible sources of friction) may cause sticking or slipping, resulting in uneven movement, which in turn causes distortion in the image generated from the data stream.
[0032] Therefore, the present distortion correction method and system utilizes the naturally occurring speckle from the illumination laser to determine how much distortion has occurred in the scan direction and modifies the image accordingly to account for and compensate for the distortion.
[0033] Figure 1An example system 100 is shown that can be used to collect data for use with embodiments of the methods and systems disclosed herein. System 100 includes a catheter 110 having an optical probe 120 disposed therein, the optical probe 120 including a reflector 130 (e.g., a ball lens or prism or mirror as shown) located at the end of a waveguide 140 (e.g., an optical fiber). A light source 150 provides illumination to the waveguide 140, which transmits light to the reflector 130 of the optical probe 120, which directs a light beam 122 to a sample 160 (e.g., luminal tissue, such as a blood vessel or a portion of the digestive tract). From the sample 160, the light returns along a path to an image collection and processing system 170, which collects and processes the returned light to form an image or other data. Light source 150 is a coherent light source (such as a laser) that produces speckle artifacts that are used in the present procedure for image correction.
[0034] During data collection, the optical probe 120 can be rotated 124 and / or translated 126 to collect data from a complete circumferential area of the sample 160 (due to the rotation 124) and along the axial length of the sample 160 (due to the translation 126). The waveguide 140 can be rotated 124 and / or translated 126 within the sheath 142 before reaching the catheter 110. To facilitate the rotation and translation of the optical probe 120, a rotation and translation device 180 (the device 180 can also include a rotating optical coupler) is attached to the proximal end of the catheter. The rotation and translation device 180 can include a linear motor for translating the optical probe 120 and a motor (e.g., a hollow motor) for rotating the optical probe 120. In some embodiments, instead of the catheter 110, the optical probe 120 can be contained in a capsule device that is swallowed by the patient, particularly for probing the digestive tract. In various embodiments, the motor for rotating the reflector 130 can be located distally adjacent to the reflector 130, particularly in capsule-based embodiments. In various embodiments, the optical probe 120 may rotate at a rate of 1-30 Hz, 1-100 Hz, 1-1000 Hz, or 1-10000 Hz, and may translate at a rate of 1-10 mm / minute.
[0035] In various embodiments, components that may cause image distortion include sticking or slipping of the waveguide 140 within the sheath 142 or catheter 110 and / or sticking or slipping of components within the rotation and translation device 180 (such as rotational or linear / translational motors). In certain embodiments, the correction procedures disclosed herein can be performed after data is collected, such as at a later date by processing previously stored data. In other embodiments, the correction procedures can be performed as data is collected, such as by processing the data as it is collected and storing a corrected version of the data. In certain embodiments, the correction procedures can be performed at substantially the same rate as data is collected, so that the processing is effectively performed in real time.
[0036] Figure 2 An example of speckle before (left panel, inset) and after (right panel, inset) distortion compensation according to embodiments disclosed herein is shown. In various embodiments, the entire image (in Figure 2 For example, in some embodiments, the image can be divided into strips having a first dimension (e.g., Figure 2 The first dimension may be in the range of 300-500 pixels (and more specifically 400 pixels in certain embodiments), and the second, substantially orthogonal dimension may be determined by a dimension of the original image, e.g., the “width” of the original image; the width of the image (e.g., Figure 2 The vertical dimension in ) can correspond to the direction of movement (e.g., rotation), and in particular can correspond to data collected from a complete rotation of a rotational imaging probe within a luminal sample, where the image is a flat representation of the tubular dataset. Figure 2 Assume a 360° rotation "helical" downward within the lumen, for example, by rotating the imaging probe while linearly advancing the probe to generate a helical data set, where Figure 2 The left side of each panel corresponds to one end of the lumen. As a result, each "strip" represents data from one full rotation of the imaging device (see Figure 2 , which shows how an image may be partitioned into a set of possible image strips). In some embodiments, the number and size of strips in an image may be related to the total height of the strips, e.g., such that each image is partitioned into strips of substantially equal size.
[0037] Next, each "strip" can be traced along its length (e.g., along Figure 2The vertical dimension in a stripe) is subdivided into multiple sub-strips or "frames". For example, each sub-strip can be substantially square, e.g., in the range of 300-500 pixels "long", and more specifically 400 pixels. The number and size of sub-strips in a stripe can be related to the total length of the stripe, e.g., so that each stripe is divided into sub-strips of substantially equal size.
[0038] After forming the sub-strips, each sub-strip is analyzed for areas of greater or lesser illumination, e.g., looking for "bright speckles" or other features indicative of areas of laser speckle. In certain embodiments, an image filtering procedure is used to identify speckle or "bright speckle" in an automated fashion. In one particular embodiment, a high pass filtering scheme may be used, e.g., by taking the difference between the original image and a filtered copy of the image in which "noise" has been reduced or removed (e.g., by using an adaptive filter such as a Wiener filter), which produces a resulting image in which speckle or "bright speckle" is the only feature or the dominant feature.
[0039] Once one or more "bright spots" or other features in a sub-strip are identified, the shape of each of those "bright spots" or features is analyzed. In certain embodiments where the size of the sub-strip or frame has been selected so that at most one feature (such as speckle or "bright spot" (see below)) is contained therein, an automatic procedure may be used to identify the boundaries of such features. This may be accomplished, for example, by mathematically identifying the x and / or y dimensions of any feature based on the feature's full width at half maximum (FWHM). The ratio of the FWHM of the horizontal (or x-dimension) and vertical (or y-dimension) dimensions of a feature (i.e., the aspect ratio) provides an indication of distortion, if any. For example, if the aspect ratio is approximately 1, this indicates little or no distortion, while aspect ratios above or below 1 indicate distortion in the image (e.g., stretching or compression in a given direction).
[0040] In some embodiments, a procedure for identifying the aspect ratio of a feature may be used that is based on calculating the autocovariance of the image data (or the autocovariance of processed image data, as discussed above, in which speckle, "bright speckle," or other features have been emphasized). This procedure may then return feature width values in two orthogonal directions. For example, the SpeckleSize_gray() function in the code provided below processes a sub-strip using autocovariance and returns feature width values in two orthogonal directions, which may take into account multiple speckle or other features within a given sub-strip. In the event that there are two or more such features within a sub-strip, the feature width value returned in each direction takes into account the dimensions of all the features in each respective direction, effectively providing an average of those dimensions. While in some embodiments, feature width is determined by calculating the full width at half maximum (FWHM), in the embodiment indicated by the code below, Gaussian values down to 0.5 and 1 / e are used. 2 This method of determining the characteristic width can provide a better understanding of the size of the speckle even when the magnitude of the Gaussian function is not unity (i.e., even when the function is not normalized so that the magnitude equals 1.0), which would be expected for a good fit to the normalized data, but the procedure may produce unexpected trends.
[0041] After the degree of distortion is identified based on the aspect ratio, the sub-strips are then virtually stretched and / or compressed (i.e., "undistorted") to return each speckle, "bright spot" or other feature to a substantially circular outline, thereby creating a corrected sub-strip or frame. This analysis and distortion is repeated for each sub-strip of a given strip, and the corrected sub-strips are then reassembled (e.g., "stitched together") in their original order to form a corrected strip.
[0042] In some embodiments, the aspect ratio may be further adjusted before the value is used to remove distortion in a sub-strip or frame. For example, the "measured" aspect ratio may be adjusted to determine a "true" aspect ratio, which is then applied to the sub-strip or frame to provide the appropriate adjustment to compensate for the distortion. Figure 3-Figure 5 Adjustment for the measured aspect ratio is performed using one of the curve fitting formulas obtained in the procedures shown in and accompanying text (see below), or using any other suitable adjustment.
[0043] In one embodiment, the strip and sub-strip sizes may be selected so that each sub-strip has, on average, at most one speckle, "bright speckle," or other feature contained therein. By adjusting the strip and sub-strip dimensions to include at most one speckle or other feature, each sub-strip may be adjusted based on a single speckle or other feature, rather than based on two or more features within a single sub-strip. However, in other embodiments, the strip and sub-strip sizes may be selected to include, on average, two, three, four, or more speckles, "bright speckles," or other features. In various embodiments, the size of the sub-strip or frame is selected to be small enough so that if multiple (e.g., 2, 3, 4, or more) speckles, "bright speckles," or other features are contained within a single sub-strip or frame, the features are close enough to each other to experience the same or substantially the same amount of distortion, if any, so that any correction factors applied to the sub-strip or frame have approximately equivalent effects in making all of the features closer to a circular profile. If the sub-strips or frames are too large, they may encompass both distorted areas and other undistorted areas, which will affect the determination of the correction factors and the extent to which image distortion is properly compensated. In some embodiments, the sub-strips may be processed to be square or approximately square with a substantially equal number of pixels in each dimension.
[0044] In some embodiments, motion-based distortion may be relatively local and may be caused, for example, by a slight movement delay when the imaging probe encounters friction or by a slight movement acceleration when the imaging probe is released from friction (this may be referred to as intra-rotational NURD, see below). Given the local nature of the distortion, two adjacent sub-strips may require different amounts of correction, and a given sub-strip that is distorted may be adjacent to a sub-strip that is not distorted and does not require correction at all. Furthermore, distortion in a rotating imaging probe may occur primarily in the rotational direction (which corresponds to the rotational direction). Figure 2 Thus, in some embodiments, resizing (e.g., changing aspect ratio) of a given sub-strip identified based on analysis of speckle, "bright spots," or other features can be accomplished by moving the sub-strip only in the direction of motion of the imaging probe (e.g., Figure 2 The advantage of this method is that the adjusted sub-strips can be reassembled into a single stripe with uniform width.
[0045] In some embodiments, motion-based distortion and / or corrections to the image data used to compensate for the distortion may result in discontinuities in the data between adjacent image strips (this may be referred to as inter-rotation NURD, see below). Various procedures may be applied to the data to properly align the image strips and reconnect the image strips, including procedures described below that involve measuring cross-correlations between adjacent strips (e.g., in combination with Figure 8 Flowchart of the .
[0046] Each strip is segmented, analyzed / warped, and reassembled in a similar manner, and then the multiple corrected strips are reassembled in their original order to form a corrected image. Figure 2 An example of a corrected image is shown in the right panel of In addition, the corrected image can be further used as desired.
[0047] It is contemplated that adjacent sub-strips of a strip and / or adjacent strips of an image may be correlated in any desired manner so that the edges of a portion of an image (e.g., a single laser speckle) are aligned to facilitate combining the corrected sub-strips and / or corrected strips in a precise manner. As a simple example, if one corrected strip depicts a substantially semicircular artifact whose circumference has a known curvature, and an adjacent corrected strip depicts an oppositely opposed substantially semicircular artifact having a very similar or identical circumference, the corrected strip may be "matched" to the semicircular artifact by being "paired" or otherwise having aligned circumferences, provided that they are two portions of a unique substantially circular artifact. The corrected image may be provided in any suitable form, including various user-perceivable forms, for further processing and / or analysis. Corrected images generated using the techniques disclosed herein may be transmitted and / or presented to a user (e.g., a researcher, operator, clinician, etc.) and / or may be stored (e.g., as part of a research database or medical record associated with a subject).
[0048] To calibrate the distortion correction procedure, a series of images of fluorescent beads of different sizes were collected and analyzed ( Figure 3 ). This procedure is used to confirm that naturally occurring speckle is an effective method for identifying and correcting image distortion and adjusting any deviations in the correction determined using speckle to align with the distortion identified using a reference standard (such as a microbead). The actual distortion in the image is determined based on measuring the aspect ratio of each bead, assuming that each bead is a spherical structure with an aspect ratio of 1; therefore, any deviation of the bead from an aspect ratio of 1 is taken as an indication of actual distortion in the image. Image-based distortion is also estimated based on the aspect ratio of the speckle, which is also assumed to be 1. Therefore, Figure 3 ] represents the relationship found between an image-based distortion estimate based on the speckle aspect ratio and the actual amount of distortion calculated by measuring the aspect ratio of spherical microbeads (axes are unitless). This relationship is used to help calibrate the software used to perform the example distortion correction operation.
[0049] exist Figure 3In FIG. 1 , each circle plotted represents an image or portion of an image for which aspect ratio was determined using both beads and speckle, with the calculated aspect ratio for each image being plotted with the speckle-based aspect ratio on the x-axis and the bead-based aspect ratio on the y-axis. It was determined that the aspect ratios (and thus the estimated distortions) generated using the two procedures were not completely consistent, and therefore several functions were used to identify a mathematical relationship between the results obtained using the two procedures. A simple linear fit of the data produced a straight line with a slope of 1.1, while other fitting procedures produced various curves such as Figure 3 In addition to linear fitting, other procedures include least absolute residual (LAR), bisquare (see Figure 4 ) and RobustOff (which means "turn off any robust approximation", i.e., standard least squares polynomial fits, as opposed to LAR and bisquare fits). In addition, based on the bisquare formula (see Figure 5 ) to determine the custom curve fitting formula ("Custom").
[0050] Figure 4 Shown are the results of curve fitting the data set using the bisquare technique with the following parameters:
[0051] f(x)=p1*x 3 +p2*x 2 +p3*x+p4
[0052] where the coefficients (with 95% confidence limits) are:
[0053] p1=07736(0.4771,1.07)
[0054] p2=-1.865(-2.779,-0.951)
[0055] p3=2.739(1.896,3.581)
[0056] p4=-0.427(-0.6545,-0.1994)
[0057] Figure 5 Shown are the results of curve fitting a data set using a custom technique with the following parameters:
[0058] f(x)=p1*x 3 +p2*x 2 +p3*x+p4
[0059] where the coefficients (with 95% confidence limits) are:
[0060] p1=1.619(1.303,1.934)
[0061] p2=-3.786(-4.759,-2.813)
[0062] p3=4.088(3.191,4.985)
[0063] p4=-0.7122(-0.9544,-0.47)
[0064] See also Figure 4 for further information.
[0065] Used to generate Figure 6 and Figure 7 The test sample for the image consists of a set of dry microspheres attached to the inner wall of a vial and imaged using a capsule-based imaging device. The size of the microspheres is 50.2μm ± 0.3μm (4K-50, Thermo Scientific 3K / 4K series particle counter standard). The size is chosen so that the microspheres should be large enough to confirm the round shape of each microsphere in the image, but also small enough to fit within a single imaging field size, i.e., 200μm × 200μm, although other sizes of microspheres smaller than 100μm and larger than 10μm can also be used. Figure 7 The images in the figure were collected at a lower magnification and therefore show a larger Figure 6 A wider field of view.
[0066] Figure 6 shows how to adjust a substrip ( Figure 6 The area within the dashed box on the left side of the top panel of the image is used to form the corrected sub-strip ( Figure 6 ). Figure 6 The top panel in shows an image strip, and the left portion of the top image (enclosed by a dashed box and labeled "Before Correction") is a sub-strip that has been processed using the procedure disclosed herein to determine the degree of distortion and the correction factor used to remove the distortion. Figure 6 The bottom panel of shows the sub-strip after its aspect ratio has been adjusted based on the correction factor identified using the disclosed procedure. While horizontal stretching is the only compensation performed in this example, and thus stretching / compensation in the scan direction is used in the correction in this example, any combination of stretching or compression of the image in any (multiple) directions may help create a corrected sub-strip for a particular use environment. Note the similar pattern of light and dark areas in the "before" and "after" areas within the red outline.
[0067] Figure 7Depicted is how the distortion correction described herein can be used to create smoother, more regular images, not only in the rotational scan direction, but also in the secondary (e.g., axial) scan direction. The left panel shows a test or phantom image before distortion correction is applied, and the right panel shows the same image after distortion correction. The vertical dimension in each panel is the rotational scan direction, while the horizontal dimension in each panel is the linear scan direction. As can be seen in the left panel image before distortion correction, many features in the image are misaligned in the horizontal direction, while in the image after distortion correction in the right panel, the features are more closely aligned from left to right. Although this sample is a phantom for demonstration purposes, a person of ordinary skill in the art will understand how to use the distortion correction procedure disclosed herein to help provide more accurate and useful imaging (e.g., imaging the interior of a patient's body cavity to detect and / or monitor lesions) via the use of a laser speckle-assisted process.
[0068] Figure 8 8 is a flow chart depicting a method 800 for distortion correction according to an embodiment of the present invention. At start 810, a step 812 of loading data on a memory may be followed by a step 814 of segmenting the data via each rotation, followed by initializing 816 a variable i=1. The method 800 continues 818 until i>#rotations, at which point the method 800 reaches end 820.
[0069] The next step of the method 800 may include converting the data segments into rotated image strips 822. Next, the method 800 may include segmenting each image strip into frames or sub-strips 824. A variable j may be initialized 826 and set to j=1. Analysis of each sub-strip or frame will continue 828 until j>the number of frames. For each sub-strip or frame, the method 800 may include: extracting a speckle image for each frame by filtering 830; measuring the aspect ratio of the speckle autocovariance 832; calculating a correction factor based on the aspect ratio 834; and resizing the image in the rotational direction using the correction factor 836. The variable j is incremented 838 so that j=j+1 after analyzing each sub-strip or frame, and control returns to step 828 to determine whether j>the number of frames. The step of compensating for the distortion of each sub-strip or frame provides correction for intra-rotational NURD (non-uniform rotational distortion).
[0070] When all frames for a particular image strip have been analyzed (when j>number of frames at step 828), adjustments are made to correct for inter-rotation NURD, i.e., distortion between adjacent image strips. An initial step may include measuring a cross-correlation between the current and previous image strips 840; a subsequent step may include calculating a phase delay value that maximizes the cross-correlation 842; and a further step may include aligning the current strip with the previous strip using the phase delay 844. Once completed, the variable i may be incremented 846 so that i=i+1, and control returns to step 818 to determine if i>number of rotations. Once i is greater than the number of rotations, the method 800 reaches the end 820.
[0071] In some embodiments, the procedures disclosed herein can be implemented using commercially available data analysis software; a specific example of such code for the Matlab software package is as follows:
[0072] % Speckle-based distortion correction
[0073] data_ND = [];
[0074] for s=1:hori / vert
[0075] ind = 1 + (s-1) * vert: s * vert;
[0076] img = data1(:, ind);
[0077] min_int = mean(min(img));
[0078] max_int = mean(max(img));
[0079] avg_int = mean(mean(img));
[0080] med_int=median(median(img));
[0081] imgf = img-wiener2(img,[2 2]); % Wiener filter
[0082] [size_x,size_y]=size(imgf);
[0083] imgf = imresize(imgf,2);
[0084] [HFWHM,VFWHM]=SpeckleSize_gray(imgf);
[0085] mNURD = VFWHM / HFWHM;
[0086] dNURD=mNURD+0.8109...
[0087] -(-451.6*exp(-0.0007511*avg_int)...
[0088] +0.9899*exp(-3.833e-06*avg_int));
[0089] rNURD=1.619*dNURD^3-3.786*dNURD^2+4.088*dNURD-0.7122;%Custom
[0090] rNURD=max([rNURD 1 / vert]); %To prevent the factor from being negative
[0091] img_ND0=0imresize(data1(:,ind),[size_xround(size_y*rNURD)]);
[0092] data_ND=[data_ND,img_ND];
[0093] logdata=[int2str(i),″,...
[0094] int2str(s),″,...
[0095] int2str(min_int),″,...
[0096] int2str(max_int),″,...
[0097] int2str(avg_int),″,...
[0098] int2str(med_int),″,...
[0099] num2str(HFWHM),″,...
[0100] num2str(VFWHM),″,...
[0101] '\r\n'];
[0102] fprintf(logID,logdata);
[0103] % Write in TIF file
[0104] maxint = mean(max(img)) / 2;
[0105] minint = mean(min(img));
[0106] img = uint16((img - minint) / maxint * (2^16));
[0107] maxint = mean(max(img_ND)) / 2;
[0108] minint = mean(min(img_ND));
[0109] img_ND = uint16((img_ND - minint) / maxint * (2^16));
[0110] index = ceil(i / Num_steps);
[0111] fname = strcat(bead_folder, c, num2str(index), '-',...
[0112] num2str(s), '(', num2str(VFWHM / HFWHM), ')');
[0113] imwrite(img, strcat(fname, '.tif'), 'tif');
[0114] fname = strcat(nurd_folder, c, num2str(index), '-',...
[0115] num2str(s), '(', num2str(VFWHM / HFWHM), ')_ND');
[0116] imwrite(img_ND, strcat(fname, '.tif'), 'tif');
[0117] end
[0118] data1 = data_ND;
[0119] if size(data1, 2) < hori / 2
[0120] data1 = data0;
[0121] end
[0122] % Background compensation
[0123] hori_cur = size(data1, 2);
[0124] hori_max = max(hori_max, hori_cur);
[0125] dummy = 0; data1_min = min(data1)';
[0126] data1_temp = [data1_min(hori_cur - dummy + 1:hori_cur);
[0127] data1_min; data1_min(1:dummy)];
[0128] ylower = smooth(data1_temp, 200)';
[0129] ylower = ylower(dummy + 1:hori_cur + dummy);
[0130] comp_BG = repmat(mean(ylower). / ylower, vert, 1);
[0131] data1 = data1.*comp_BG;
[0132] % Image realignment based on cross - correlation
[0133] data1 = circshift(data1, [0 preDiff]);
[0134] strip_prev = smooth(mean(data0) - mean(mean(data0)), 2000);
[0135] strip_curr = smooth(mean(data1) - mean(mean(data1)), 2000);
[0136] [acor, lag] = xcorr(strip_prev, strip_curr); length_acor = length(acor);
[0137] wfunc = normpdf(1:length_acor, (length_acor + 1) / 2, (length_acor + 1) / 2 / 16)';
[0138] acor=acor.*wfunc;[~,I]=max(acor);cofactor=(cofactor*(i-1)+I) / i;
[0139] lagDiff=lag(I)*(i>1); preDiff=lagDiff;
[0140] data1=circshift(data1,[0lagDiff]);
[0141] data0=data1;
[0142] Here is the Matlab code for the SpeckleSize_gray() function:
[0143] function[HFWHM,VFWHM]=SpeckleSize_gray(SpeckleImg)
[0144] M = size (SpeckleImg, 1); % height of the area
[0145] N = size (SpeckleImg, 2); % width of the area
[0146] SpeckleImg=double(SpeckleImg);
[0147] E = SpeckleImg(1,:); % Create an array with the values of row 1
[0148] s = size(xcov(E)); %Find the size of the autocovariance array
[0149] D = zeros(s); % Create an empty array of size s
[0150] D = double(D); % Convert the value type to double precision
[0151] for i=1:M
[0152] C = SpeckleImg(i,:);
[0153] D = imadd(D,xcov(C,'coeff')); %Sum all rows of xcov array to D
[0154] end
[0155] H0 = D / max(D); % of the horizontal product completed, H is normalized
[0156] sizeH=size(H0,2);
[0157] H=H0(floor(sizeH / 2)-5:floor(sizeH / 2)+5);
[0158] E1 = SpeckleImg(:,1); % Create an array with the values of column 1
[0159] s1 = size(xcov(E1)); %Find the size of the autocovariance array
[0160] D1 = zeros(s1); % Create an empty array of size s1
[0161] D1=double(D1); %Convert the value to double precision
[0162] for j=1:N
[0163] C1=SpeckleImg(:,j);
[0164] if max(C1)>0
[0165] D1=D1+xcov(C1,'coeff'); %Sum the xcov arrays of all rows to D1
[0166] end
[0167] end
[0168] V0 = D1 / max(D1); % The completed vertical product V is normalized
[0169] sizeV = size(V0);
[0170] V=V0(floor(sizeV / 2)-5:floor(sizeV / 2)+5);
[0171] %H and V were fitted with Gaussian distributions and the speckle size was extracted from the fit.
[0172] helper1 = 1:size(H,2);
[0173] helper2 = 1:size(V);
[0174] gauss1=fittype('gauss1'); %Set Gaussian curve fitting
[0175] excludeLowH=excludedata(helper1',H','range',[.1,1]); %Exclude noise outside the speckle
[0176] excludeLowV=excludedata(helper2',V,'range',[.1,1]);
[0177] optionsH=fitoptions(gauss1);
[0178] optionsV=fitoptions(gauss1);
[0179] optionsH.Exclude=excludeLowH;
[0180] optionsV.Exclude=excludeLowV;
[0181] [HFit,HFitStats]=fit(helper1',H',gauss1,optionsH);
[0182] [VFit,VFitStats]=fit(helper2',V,gauss1,optionsV);
[0183] HFWHM = (2*(HFit.c1)*sqrt(-log(.5 / (HFit.a1)))); % FWHM value (full width when fit = 0.5)
[0184] VFWHM=(2*(VFit.c1)*sqrt(-log(.5 / (VFit.a1))));
[0185] HeSquared = ((HFit.c1)*sqrt(-log((.1353353) / (HFit.a1)))); % 1 / e^2 value (full width when fit = 0.135)
[0186] VeSquared=((VFit.c1)*sqrt(-log((.1353353) / (VFit.a1))));
[0187] end
[0188] Fig. 9A schematic block diagram is shown showing an example computer system 900 including hardware components capable of implementing embodiments of the systems and methods disclosed herein for identifying and correcting distortions, wherein the system 900 may include various systems and subsystems. The system 900 may be a personal computer, a laptop, a workstation, a computer system, an appliance, an application specific integrated circuit (ASIC), a server, a blade server center, a server farm, etc., and may be distributed across one or more computing devices.
[0189] The system 900 may include a system bus 902, a processing unit 904, a system memory 906, storage devices 908 and 910, a communication interface 912 (e.g., a network interface), a communication link 914, a display 916 (e.g., a video screen) or other output device, and an input device 918 (e.g., a keyboard, a mouse, a touch screen, a touch pad, etc.). The system bus 902 may communicate with the processing unit 904 and the system memory 906. Additional memory devices 908 and 910 may include various non-transitory storage devices, such as hard drives, servers, independent databases, or other non-volatile memory, and may also communicate with the system bus 902. The system bus 902 interconnects the processing unit 904, the memory devices 906, 908, 910, the communication interface 912, the display 916, and the input device 918. In some examples, the system bus 902 may also interconnect with one or more additional ports, such as a universal serial bus (USB) port, an Ethernet port, or other communication mechanisms / connections.
[0190] The processing unit 904 may be a computing device and may include an application specific integrated circuit (ASIC) or other processor or microprocessor. The processing unit 904 executes a set of instructions to implement the operations of the examples disclosed herein. The processing unit may include a processing core.
[0191] Additional memory devices 906, 908, and 910 may store data in text or compiled form, programs, instructions, database queries, and any other information necessary to operate the computer. Memory 906, 908, and 910 may be implemented as a computer-readable medium (integrated or removable), such as a memory card, a disk drive, a compact disk (CD), or a server accessible via a network. In some examples, memory 906, 908, and 910 may include text, images, video, and / or audio, portions of which may be available in a human-understandable format.
[0192] Additionally or alternatively, system 900 may access external data sources or query sources through communication interface 912 , which may be in communication with system bus 902 and communication link 914 .
[0193] In operation, system 900 can be used to implement one or more portions of distortion correction according to the present invention. According to some examples, computer executable logic for implementing portions of distortion correction resides on system memory 906 and one or more of memory devices 908 and 910. Processing unit 904 executes one or more computer executable instructions originating from system memory 906 and memory devices 908 and 910. The term "computer-readable medium" as used herein refers to one or more media that participate in providing instructions to processing unit 904 for execution.
[0194] It will thus be appreciated that the computer-readable media is non-transitory and may include multiple discrete media operatively connected to the processing unit via one or more of a local bus or a network connection, for example.
[0195] In some embodiments, any suitable computer-readable medium may be used to store instructions for performing the functions and / or processes described herein. For example, in some embodiments, the computer-readable medium may be transient or non-transient. For example, non-transient computer-readable media may include the following media, such as: magnetic media (such as hard disks, floppy disks, etc.), optical media (such as compressed disks, digital video disks, Blu-ray disks, etc.), semiconductor media (such as RAM, flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.), any suitable medium that is not transient or does not have any persistent appearance during transmission, and / or any suitable tangible medium. As another example, transient computer-readable media may include signals on a network, signals on wires, conductors, optical fibers, circuits, or any suitable medium that is transient and does not have any persistent appearance during transmission, and / or any suitable intangible medium.
[0196] It should be noted that the term mechanism as used herein may cover hardware, software, firmware, or any suitable combination thereof.
[0197] Fig.10 An example 1000 of a process for correcting image distortion according to some embodiments of the disclosed subject matter is shown. Fig.10 As shown, at 1002, process 1000 can analyze an image segment of an image to identify a speckle artifact, the image segment being obtained from a scanning imaging device. At 1004, process 1000 can determine an aspect ratio of a shape of the speckle artifact. At 1006, process 1000 can determine a correction factor for the shape of the speckle artifact based on the aspect ratio. Finally, at 1008, process 1000 can adjust the size of the image segment based on the correction factor.
[0198] Fig.11An example 1100 of a process for distortion correction according to some embodiments of the disclosed subject matter is shown. Fig.11 As shown, at 1102, process 1100 may provide an image. At 1104, process 1100 may divide the image into a plurality of substantially parallel strips, each strip extending in a first direction. At 1106, process 1100 may divide each strip into a plurality of sub-strips along a second direction substantially perpendicular to the first direction. At 1108, process 1100 may analyze each of the plurality of sub-strips to locate at least one local bright feature. At 1110, process 1100 may distort at least a portion of each sub-strip to force the local maximum / minimum feature toward a predetermined shape, thereby creating a corrected sub-strip. At 1112, process 1100 may reassemble the plurality of corrected sub-strips into a corrected strip. Finally, at 1114, process 1100 may reassemble the plurality of corrected strips into a corrected image.
[0199] It should be understood that Fig.10 and Fig.11 The above steps of the process may be performed or carried out in any order or sequence, not limited to the order and sequence shown and described in the accompanying drawings. Fig.10 and Fig.11 Some of the above steps of the process may be performed substantially simultaneously or in parallel where appropriate to reduce delays and processing time.
[0200] As used herein, the singular forms "a", "an", and "the" may also include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprise" and / or "comprising" are used herein, the presence of stated features, steps, operations, elements, and / or components may be specified, but not the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.
[0201] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0202] It should be understood that when an element is referred to as being "on," "attached," "connected," "coupled," "in contact with," etc., the other element may be directly on, attached, connected, coupled to, or in contact with the other element, or intervening elements may also be present. In contrast, when an element is referred to as being, for example, "directly on," "directly attached," "directly connected," "directly coupled," or "in direct contact with" another element, there are no intervening elements.
[0203] As shown in the figures, for ease of description, spatially relative terms such as "under", "below", "lower", "over", "upper", etc. may be used herein to describe the relationship of one element or feature to (multiple) other elements or features. It should be understood that in addition to the orientation depicted in the drawings, spatially relative terms can also cover different orientations of the device in use or operation. For example, if the device in the drawings is inverted, elements described as "under" or "beneath" or "on" other elements or features will be oriented as "above" the other elements or features.
[0204] The present invention comprises, consists of or consists essentially of the features described herein, in any combination.
[0205] Therefore, although the present invention has been described above in conjunction with particular embodiments and examples, the present invention is not necessarily so limited, and many other embodiments, examples, uses, modifications and deviations from the described embodiments, examples and uses are intended to be encompassed by the appended claims.
Claims
1. A method for correcting image distortion, comprising: analyzing, by a processor, image segments of the image to identify speckle artifacts, the image segments being obtained from a scanning imaging device; determining, by the processor, an aspect ratio of a shape of the speckle artifact; determining, by the processor, a correction factor for the shape of the speckle artifact based on the aspect ratio; as well as The processor adjusts, based on the correction factor, the dimensions of the image segment.
2. The method according to claim 1, characterized in that Determining the correction factor for the shape of the speckle artifact based on the aspect ratio further comprises: The correction factor is determined based on a polynomial function relating the aspect ratio to the correction factor.
3. The method according to claim 1, characterized in that The image is obtained from at least one of a rotational scanning device or a linear scanning device.
4. The method according to claim 3, characterized in that deriving the image segments by dividing the image into a plurality of strips, Each of the plurality of stripes is subdivided into a plurality of sub-strips, and the image segment includes a sub-strip of the plurality of sub-strips.
5. The method according to claim 4, characterized in that The image is generated by rotational scanning of the sample, Each of the plurality of strips corresponds to data collected in a rotational scanning direction of the sample.
6. The method of claim 4, further comprising: adjusting a dimension of each of the plurality of sub-strips based on a corresponding correction factor determined for each of the plurality of sub-strips, reassembling each of the adjusted sub-strips into a plurality of corrected strips, and The plurality of corrected strips are reassembled into a corrected image.
7. The method according to claim 6, characterized in that Reassembling the plurality of corrected strips into a corrected image further comprises: determining a cross-correlation between two adjacent stripes of the plurality of corrected sub-strips, calculating a phase delay value that maximizes the cross-correlation, and The two adjacent strips are aligned based on the calculated phase delay.
8. The method according to any one of claims 1 to 7, characterized in that Analyzing the image segments to identify the speckle artifact further comprises: extracting a speckle image of the image segment based on filtering the image segment, and Wherein, determining the aspect ratio of the shape of the speckle artifact further comprises: The aspect ratio is determined based on an autocovariance of the speckle artifact in the speckle image.
9. The method according to any one of claims 1 to 7, characterized in that The speckle artifacts are caused by coherent light sources.
10. The method according to claim 9, characterized in that The coherent light source includes a laser.
11. A device for correcting image distortion, comprising: processor; as well as a memory in communication with the processor having stored thereon a set of instructions that, when executed by the processor, cause the processor to: analyzing an image segment of the image to identify speckle artifacts, the image segment being obtained from a scanning imaging device, determining the aspect ratio of the shape of the speckle artifact, determining a correction factor for the shape of the speckle artifact based on the aspect ratio; and The image segments are resized based on the correction factors.
12. The device according to claim 11, characterized in that In determining the correction factor for the shape of the speckle artifact based on the aspect ratio, the instructions further cause the processor to: The correction factor is determined based on a polynomial function relating the aspect ratio to the correction factor.
13. The device according to claim 11, characterized in that The image from which the image segments are derived is generated by at least one of a rotational scan or a linear scan of a sample.
14. The device according to claim 13, characterized in that deriving the image segments by dividing the image into a plurality of strips, Each of the plurality of stripes is subdivided into a plurality of sub-strips, and the image segment includes a sub-strip of the plurality of sub-strips.
15. The device according to claim 14, characterized in that The image is generated by rotational scanning of the sample, Each of the plurality of strips corresponds to data collected in a rotational scanning direction of the sample.
16. The device according to claim 14, characterized in that The instructions further cause the processor to: adjusting a size of each of the plurality of sub-strips based on a correction factor determined for each of the plurality of sub-strips, reassembling each of the adjusted sub-strips into a plurality of corrected strips, and The plurality of corrected strips are reassembled into a corrected image.
17. The device according to claim 16, characterized in that When reassembling the plurality of corrected strips into a corrected image, the instructions further cause the processor to: determining a cross-correlation between two adjacent stripes of the plurality of corrected sub-strips, calculating a phase delay value that maximizes the cross-correlation, and The two adjacent strips are aligned based on the calculated phase delay.
18. The device according to any one of claims 11 to 17, characterized in that In analyzing the image segments to identify the speckle artifact, the instructions further cause the processor to: extracting a speckle image of the image segment based on filtering the image segment, and Wherein, when determining the aspect ratio of the shape of the speckle artifact, the instructions further cause the processor to: The aspect ratio is determined based on an autocovariance of the speckle artifact in the speckle image.
19. The device according to any one of claims 11 to 17, characterized in that The speckle artifacts are caused by coherent light sources.
20. The device according to claim 19, characterized in that The coherent light source includes a laser.
21. A method for distortion correction, the method comprising: Provide images; dividing the image into a plurality of substantially parallel strips, each strip extending in a first direction; dividing each strip into a plurality of sub-strips along a second direction substantially perpendicular to the first direction; analyzing each sub-strip of the plurality of sub-strips to locate at least one local bright feature; adjusting at least a portion of each of the sub-strips to force at least one local bright feature toward a predetermined shape to create a corrected sub-strip; reassembling the plurality of corrected sub-strips into a corrected stripe; as well as The plurality of corrected strips are reassembled into a corrected image.
22. The method according to claim 21, characterized in that At least one local bright feature includes a laser speckle artifact.
23. The method of claim 21, wherein: Providing an image includes obtaining an image from a rotational imaging sensor.
24. The method according to any one of claims 21 to 23, characterized in that Reassembling the plurality of corrected strips into a corrected image further comprises: performing cross-correlation between pairs of adjacent stripes of the plurality of corrected stripes, re-adjusting the relative positions of the pairs of adjacent strips based on performing the cross-correlation, and The plurality of corrected sub-strips are reassembled into a corrected image based on rescaling the relative positions of the pairs of adjacent strips.
25. A device for correcting image distortion, comprising: a scanning imaging device, the scanning imaging device comprising a coherent light source, the scanning imaging device being configured to obtain an image of the sample; a processor in communication with the scanning imaging device, and a memory in communication with the processor having stored thereon a set of instructions that, when executed by the processor, cause the processor to: analyzing image segments of the image to identify speckle artifacts, determining the aspect ratio of the shape of the speckle artifact, determining a correction factor for the shape of the speckle artifact based on the aspect ratio; and The dimensions of the image segments are adjusted based on the correction factors.
26. The device according to claim 25, characterized in that In determining the correction factor for the shape of the speckle artifact based on the aspect ratio, the instructions further cause the processor to: The correction factor is determined based on a polynomial function relating the aspect ratio to the correction factor.
27. The device according to claim 25, characterized in that The scanning imaging device obtains the image from the sample using at least one of a rotational scan or a linear scan of the sample.
28. The device according to claim 27, characterized in that deriving the image segments by dividing the image into a plurality of strips, Each of the plurality of stripes is subdivided into a plurality of sub-strips, and the image segment includes a sub-strip of the plurality of sub-strips.
29. The device according to claim 28, characterized in that The image is obtained by rotationally scanning the sample, Each of the plurality of strips corresponds to data collected in a rotational scanning direction of the sample.
30. The device according to claim 28, characterized in that The instructions further cause the processor to: adjusting a size of each of the plurality of sub-strips based on a correction factor determined for each of the plurality of sub-strips, reassembling each of the adjusted sub-strips into a plurality of corrected strips, and The plurality of corrected strips are reassembled into a corrected image.
31. The device according to claim 30, characterized in that When reassembling the plurality of corrected strips into a corrected image, the instructions further cause the processor to: determining a cross-correlation between two adjacent stripes of the plurality of corrected sub-strips, calculating a phase delay value that maximizes the cross-correlation, and The two adjacent strips are aligned based on the calculated phase delay.
32. The device according to any one of claims 25 to 31, characterized in that In analyzing the image segments to identify the speckle artifact, the instructions further cause the processor to: extracting a speckle image of the image segment based on filtering the image segment, and Wherein, when determining the aspect ratio of the shape of the speckle artifact, the instructions further cause the processor to: The aspect ratio is measured based on an autocovariance of the speckle artifact in the speckle image.
33. The device according to any one of claims 25 to 31, characterized in that The speckle artifact is caused by the coherent light source.
34. The device according to claim 33, characterized in that The coherent light source includes a laser.
