Phase detection and correction using image-based processing

By using threshold and variance metric algorithms to identify the extreme value sets in MEMS scanners, phase drift is automatically corrected, and the image blurring caused by phase drift in MEMS scanners is solved, and image clarity and system automation capabilities are improved.

CN113015881BActive Publication Date: 2024-11-26THE RGT UNIV OF MICHIGAN
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980074812.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-10-26
Filing Date
2019-10-25
Publication Date
2024-11-26
Estimated Expiration
2039-10-25

AI Technical Summary

Technical Problem

When using MEMS scanners for Lisaru scans, phase drift results in image blur and dual image effects, and the prior art is difficult to effectively track and correct, especially in miniaturized devices such as endoscopes, manual phase adjustment requires a large amount of user experience And there is no automation solution.

Method used

Through a fuzzy metric algorithm based on threshold and variance, the original data set is analyzed to identify the extreme value set, and a reconstructed image is generated to correct phase drift, providing computer-implemented methods and systems, and the automatic focus process tracks and corrects phase delays at regular time intervals.

Benefits of technology

It realizes accurate prediction and correction of phase drift in MEMS scanners, improves image clarity, reduces user experience requirements, is suitable for dynamically induced phase delay drift of compact scanners, improves image reconstruction accuracy and system reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113015881B_ABST
    Figure CN113015881B_ABST
Patent Text Reader

Abstract

A computer-implemented method includes: receiving a raw data set representing an image of a sample; analyzing the raw data set using a metric algorithm to identify a first set of extrema; identifying a second set of extrema by analyzing the first set of extrema; and generating a reconstructed image of the sample based on the second set of extrema. A phase correction scanner includes one or more processors; one or more scanners adapted to sequentially sample an object to generate vectors of raw data representing the object in a Lissajous pattern; and a memory storing instructions that, when executed by the one or more processors, cause a computing system to receive the vectors of raw data, analyze the raw data set using a metric algorithm to identify a first set of extrema, and identify a second set of extrema by analyzing the first set of extrema.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Government Interest Statement

[0002] This invention was made with government support under Grant CMMI-1334340 from the National Science Foundation and Grant EB020644 from the National Institutes of Health. The government has certain rights in this invention. Technical Field

[0003] The present disclosure generally relates to systems and methods for tracking and / or correcting phase shifts during two-dimensional (2D) and / or three-dimensional (3D) scanning using an optical instrument. More specifically, methods and systems for detecting and / or correcting phase shifts during scanning (e.g., Lissajous scanning) using a two-axis micro-electromechanical system micromirror are presented. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. To the extent described in this background section, the works of the presently named inventors and aspects of the description that may not otherwise qualify as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0005] Imaging using beam steering has many applications, including biomedical imaging, scanning probe microscopy, 3D printing, single pixel cameras, scanning electron microscopy, light detection and ranging (LiDAR), etc. Various beam steering modes, such as raster, spiral, and Lissajous, can be selected depending on the imaging application and actuator capabilities. The scanning mode may have a direct impact on image resolution, field of view (FOV), and frame rate (FR). For example, Lissajous scanning is obtained when both axes of motion are operated with constant sinusoidal inputs (possibly with different frequencies and / or phases). For many scanning actuators, a large FOV can be achieved by operating the scanner near the resonant frequency. High FR can also be achieved by carefully designing the resonant frequency of the scanner and the displacement at these frequencies. Lissajous scanning is a popular choice in imaging applications because it can be easily implemented using one or more mirror galvanometers, miniature micro-electromechanical systems (MEMS)-based scanners, etc. Unlike raster scanning, Lissajous does not require the two operating frequencies to be far enough apart, simplifying the MEMS design.

[0006] Lissajous scanning has many optical applications, including biomedical imaging, scanning probe microscopy, single-pixel cameras, etc. In a typical application, a scanner (e.g., MEMS micromirror, mirror galvanometer, multiphoton optical probe, etc.) is used to sequentially sample points on the object plane. The light intensity from different points on the object plane is recorded as time series data, called raw data. The sampling trajectory can be determined by the Lissajous pattern, which in turn depends on the scanning frequency (f) in the corresponding motion axis (e.g., x and y axis). x ,f y ), displacement amplitude (θ x ,θ y ) and phase delay An image of the object plane can be obtained by placing the raw data in the image in the same order and position as they were scanned on the object plane. Therefore, in order to reconstruct an image from the raw data, accurate knowledge of the scanning trajectory may be required.

[0007] However, when using MEMS scanners, the actual scanning trajectory may deviate from the phase coordinates used in image reconstruction. Drifting scan trajectory. MEMS scanners can operate at the resonant frequency of the device with natural oscillations. Such oscillations can occur when the scanner is placed in an instrument (e.g., an endoscope), and when the endoscope is placed inside a living organism. The oscillations can be small, but can be enough to cause image registration problems. Images can be blurry and may have double image effects because the properties of the mirror subtly change once inside the endoscope and / or the living organism.

[0008] Phase drift can be attributed to many factors, such as environmental conditions, changes in scanner properties over time, etc. Even a small phase drift can cause large deviations in the scan trajectory, resulting in poor image quality. If the drift is large enough, the image will be unrecognizable. Therefore, the effectiveness of Lissajous scanning and the accuracy of image reconstruction can be affected by the phase difference between the axes. In miniaturized devices such as MEMS scanning mirrors, the resonant frequency of the scanner (i.e., resonant micromirror) can drift by several degrees due to environmental perturbations. This drift, in turn, can cause a change in the phase delay between the mirror motion and the periodic input drive signal.

[0009] Previous approaches to compensating for phase shift in resonant devices include temperature-based calibration, on-chip capacitive sensing, attempts to engineer robust dynamics, and designs that limit temperature sensitivity. However, these conventional approaches suffer from poor repeatability, poor signal-to-noise ratios, and material limitations, especially when using small MEMS devices in applications that are severely space-constrained, such as endoscopy. Various feedback controllers have been proposed, but these controllers are also susceptible to sensing limitations in small in vivo instruments, and these controllers increase system complexity.

[0010] Furthermore, some devices (e.g., endoscopes) may require miniaturized MEMS-based scanners to achieve compact packaging. As mentioned above, the phase of such scanning May be prone to drift. The drift of phase can be attributed to many factors, such as changes in environmental conditions (e.g., temperature), changes in material properties over time, etc. Therefore, the phase It may be a slow function of time, which makes image reconstruction difficult. The phase can be experimentally determined in a controlled environment (such as a laboratory) to reconstruct the image. However, in practical applications, the phase determined at startup in the laboratory will not remain constant indefinitely. The phase can be manually adjusted to compensate for the drift of sparse images, but manual adjustment requires a lot of user experience and is almost impossible to adjust for complex images with unknown structures.

[0011] Generally, clarity-based autofocus processes are known in the art. However, the art does not include clarity-based autofocus processes for tracking and / or correcting drift in phase delay caused by the dynamics of MEMS scanning mirrors and / or other compact scanners that may be used to generate Lissajous patterns. Therefore, there is a need for methods and systems to track and / or overcome phase drift at regular time intervals during Lissajous scanning. Summary of the invention

[0012] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0013] In one aspect, a computer-implemented method for predicting and / or correcting phase drift includes: receiving a raw data set representing an image of a sample; identifying a first set of extreme values ​​by analyzing the raw data set using a metric algorithm; identifying a second set of extreme values ​​by analyzing the first set of extreme values; and generating a reconstructed image of the sample based on the second set of extreme values.

[0014] The phase correction scanner includes one or more processors; one or more scanners adapted to sequentially sample an object to generate a vector of raw data representing the object in a Lissajous pattern; and a memory storing instructions that, when executed by the one or more processors, cause a computing system to receive the vector of raw data, analyze the raw data set using a metric algorithm, identify a first set of extreme values, and identify a second set of extreme values ​​by analyzing the first set of extreme values. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1Adepicts a series of example images demonstrating phase drift effects according to one embodiment,

[0016] Figure 1B depicts a data flow diagram for image reconstruction according to one embodiment,

[0017] Figure 1C Depicted is a set of binarized images showing the intuition behind the threshold measure of the phase drift effect,

[0018] Figure 1D depicts a graph of a threshold metric versus phase x and phase y for a global search and a local search around an initial approximation given by the global search, according to one embodiment and scenario,

[0019] Figure 1E depicts a graph of sharpness (variance) versus phase x and phase y for a global search and a local search of an initial approximation given around the global search, according to one embodiment and scenario,

[0020] Figure 1F depicts a combined graph of a threshold and a clarity metric according to some embodiments,

[0021] Figure 2A Depicts the improvement in image quality of a representative human colon tissue sample,

[0022] Figure 2B depicts a test image used to validate blur performance and sharpness metrics according to some embodiments,

[0023] Figure 2C Describes the Figure 2B A global search of the test images.

[0024] Figure 3A depicts a reconstructed image 302 containing missing pixels of the true phase,

[0025] Figure 3B A graph depicting the effect of missing pixels on the blur metric and a graph depicting the effect of missing pixels on the sharpness metric,

[0026] Figure 4 depicts an exemplary image sequence showing direct binarization of raw data and binarization of grayscale images,

[0027] Figure 5 An example graph depicting the density of an image generated by a scanner.

[0028] Figure 6 A diagram of a system for scanning (eg, endoscopic scanning) according to one embodiment is shown; and

[0029] Figure 7 An example method for phase correction of a scanner is depicted according to one embodiment. DETAILED DESCRIPTION

[0030] Although the following text sets forth a detailed description of many different embodiments, it should be understood that the legal scope of this description is defined by the words of the claims set forth at the end of this document. The detailed description should be construed as exemplary only and does not describe every possible embodiment, as describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments may be implemented using current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.

[0031] It should also be understood that unless a term is expressly defined in this patent using the sentence "As used herein, a term" "is defined herein to mean ..." or similar sentences, no limitation of the meaning of the term, either expressly or by implication, beyond its plain or ordinary meaning is intended, and such term should not be construed as limited in scope based on any statement made in any section of this patent (except the language of the claims). To the extent that any term recited in the claims at the end of this patent is referenced in this patent in a manner consistent with a single meaning, this is done solely for clarity so as not to confuse the reader, and is not intended to limit such claim term by implication or otherwise to that single meaning. Finally, unless a claim element is limited by the recitation of the word "means to" and the function of the recitation without any construction, the scope of any claim element should not be construed based on the application of 35 U.S.C. §112(f).

[0032] The present technology includes a threshold-based blur metric that can binarize the raw data of an image to predict phase errors in a reconstructed image. The present technology includes a sharpness-based method and system for tracking phase shifts during scanning (e.g., Lissajous scanning) using a dual-axis MEMS micromirror. The MEMS mirror can be an electrostatic scanning mirror with two rotating axes to develop an image on one or more image planes. The axes can be scanned at a high frequency to produce a Lissajous scan.

[0033] A variance-based sharpness metric can be used to successfully determine the true phase value. The present technology may include a sharpness metric for autofocus in several applications that can compensate for phase drift during scanning (e.g., Lissajous scanning). The present technology corrects for phase drift caused by dynamic parameter changes in a microscanner using previously unknown methods and systems. According to one embodiment, a problem that arises during image reconstruction using a MEMS mirror can be solved by an algorithm to identify the relative phase of two axes. The effectiveness of a prototype endoscopic imaging probe according to one embodiment is also disclosed herein. In addition, the interaction between sharpness-based phase detection and other trade-offs in scanner design, such as fill factor (FF) relative to frame rate and uneven scan density, and practical methods for managing such trade-offs for MEMS scanners are discussed.

[0034] As mentioned above, the phase Due to the deviation of the scanning trajectory, the MEMS scanner may drift when using it. The scanning trajectory may deviate due to various reasons, including but not limited to environmental disturbances, changes in the material properties of the scanner, etc. To overcome this problem, the present technology discloses a method for accurately predicting the phase Image processing techniques. Specifically, methods and systems are disclosed for identifying phase by identifying when the image is sharpest or least blurry when reconstructed with the exact phase. The present technology discloses various metric algorithms, which may include sharpness and / or blur metrics (e.g., variance, threshold-based, etc.) for accurately predicting phase. In one embodiment, the phase along one axis (e.g., x) is swept by keeping the phase of the y-axis constant equal to an initial guess. For each phase value, an image is constructed and its sharpness (or blurriness) is calculated. The phase corresponding to the sharpest (or most blurry) image is predicted. Similar steps can be followed to obtain the phase The phase along the x-axis is set to The predicted value of .

[0035] Example scan model and phase shift effects in reconstruction

[0036] The displacement of the scanner can be modeled as a function of time. If the scanner is driven along a particular axis with the drive voltage given below, it is within the linear limit of its dynamic behavior.

[0037] V k (t) = A k sin(2πf k t+ψ k ),

[0038] , then the motion of the scanner along the corresponding axis can be modeled as

[0039]

[0040] The subscript k represents the parameter along the corresponding axis (x or y), and f k is the driving frequency close to resonance in the x and y axes, ψ k is the phase of the driving voltage, θ k is the mechanical displacement of the scanner, and is the phase difference between the scanner motion and the input voltage at the current drive frequency.

[0041] In one embodiment, one or more scanners (e.g., two single-axis scanners, a dual-axis scanner, etc.) may be used to sample points from the object plane. The points may be sampled (e.g., sequentially) while the laser beam is manipulated using a single-axis scanner and / or a dual-axis scanner. The sequentially sampled data may include time series raw data. In some embodiments, other configurations of data structures may be used to store intensities (e.g., a dedicated time series data structure).

[0042] If the trajectory of the laser point on the object plane as a function of time is known, the image can be reconstructed from the raw data. The laser motion can be determined by the parameter f k ,ψ k , and where f k , and ψ k It is known that the ratio It can be determined experimentally, but It is usually unknown or not completely known due to environmental interference. In particular, the phase Drift is common in MEMS-based scanners, which are often used in low-power and / or miniaturized applications such as endoscopy. Phase shifts in one or both axes can be attributed to many factors, such as changes in environmental conditions (including temperature), changes in material properties over time, etc. This makes the phase is a slow function of time and makes image reconstruction difficult. The phase can be determined experimentally in a controlled environment (e.g., a laboratory). However, in practical applications, the phase determined in the laboratory does not remain constant indefinitely.

[0043] Phase Can be written as

[0044]

[0045] in is a constant, is the phase drift. The phase can be adjusted manually to compensate for the drift of sparse images. However, manual adjustment requires a lot of user experience and time, and it is almost impossible to adjust the phase of images of structures that are new to the user (e.g., tissue samples of unfamiliar organisms). Therefore, it is necessary to determine the correct phase at regular time intervals with minimal disruption to the endoscopy process.

[0046] Figure 1A Depicted is a series of example images demonstrating the effects of phase drift. The deviations between the individual example images provide visual evidence of the impact of phase inaccuracies on image reconstruction, e.g., autofocus or similar techniques. Figure 1A A raw image 102 depicting raw data is included. The raw data of the raw image 102 may be generated by simulating a MEMS-based scan in a Lissajous style. For example, x =19251Hz,f y =3315Hz, and The example scanning frequencies and phases can be used to simulate the laser scanning pattern. Assuming the sampling rate is 10 7 samples / second, a large number of samples (e.g. 10 6 At any sampling instance, the position of the simulated laser spot can be rounded to the nearest pixel position.

[0047] Figure 1A Including the use of and The reconstructed image 104 corresponds to the original image 102 with an accurate value of . For example, the original image 102 can be a first three-dimensional image and the reconstructed image 104 can be a second three-dimensional image, where either corresponding three-dimensional image includes two or three channels (e.g., grayscale or red, green, and blue), and each channel includes a two-dimensional array of pixels. In some cases, a three-dimensional image can be represented by raw data (e.g., a multi-dimensional array of channels, each channel including a multi-dimensional array of pixels). The original image 102 and / or the reconstructed image 104 can be grayscale (i.e., a two-dimensional image). The effects of phase drift can be seen by introducing errors along one or more axes. For example, the and The exact value of is reconstructed from the original data to reconstruct the reconstructed image 104. If rad error / drift, a reconstructed image 106 may be generated. The reconstructed image 106 may include an overlap of the reconstructed image 104 with itself, with an offset corresponding to This shift can be attributed to the fact that the image registration during the right scan does not overlap exactly with the left scan. Moreover, the shift between the left and right scan images is not is a linear function because it is mapped by a sine function. If for two phases ( and ) has an error of π / 8 rad, the output image may have, for example, four overlapping images, as shown in reconstructed image 108 - one each in the right, left, upper and lower scanning directions. Figure 1A A simple view of the drift problem is provided, but even small phase errors result in image blurring, which significantly affects image resolution and, in turn, the desirability, interpretability, and reliability of the final output image. This potential for undesirability places stringent requirements on the accuracy with which the phase should be predicted and / or detected. Those skilled in the art will appreciate that in some embodiments, accuracy may be parameterized to modulate analysis latency and / or improved accuracy.

[0048] Scan design and image reconstruction

[0049] Figure 1B A data flow diagram 110 is depicted, which shows image construction in an imaging instrument via scanning (e.g., Lissajous scanning). The data flow diagram 110 may include an object H, an imaging instrument (e.g., a single pixel camera), a raw data vector, and an image I. The object H may be any suitable object (e.g., tissue of a biological body). To generate grayscale imaging by single point scanning, a point (x p ,y p ) can be operated at a constant frequency f s Sequential sampling, and the light intensity H(x p ,y p ), stored as data samples r in the original data vector p Through this process, the object H(x p ,y p ) is converted into a one-dimensional time series data r p Therefore, point r p The vector R records the intensity of the object H at regular time intervals Δt (sampling time). The sampling point (x p ,y p ) is determined by a scanning pattern (in one embodiment, a Lissajous pattern). It should be noted that in a Lissajous scan, a point from an object may be scanned multiple times to increase the FF, resulting in a non-uniform scan density. The raw data is depicted as a vector, but the raw data may be stored in any suitable data structure (e.g., a collection, an array, etc.). In addition, it will be appreciated by those skilled in the art that any suitable imaging instrument and / or scanning pattern may be used.

[0050] In some embodiments, the image reconstruction problem can be posed as an inverse problem, where r pis mapped to a two-dimensional image space (e.g., a discrete space) I(i, j), where image I is a gray image (MxN matrix) of object H. Therefore, each ordered pair (i, j) can be represented as a function of p. In embodiments including a discrete image space, r p The mapping to I(i,j) can be many-to-one due to the non-uniform scan density and the positions of sample instances are rounded to the nearest pixel position. The final intensity of the pixel at (i,j) can be the r mapped to the same pixel (i,j). p The process can be summarized by the following calculation:

[0051]

[0052]

[0053] r p =H(x p ,y p )

[0054]

[0055]

[0056] in and is a constant, [] indicates the maximum integer function.

[0057] Phase detection using threshold-based blur metric techniques and / or variance-based sharpness metric techniques

[0058] The present technology includes the use of an image sharpness based autofocus method and system to detect the phase of a two-axis MEMS scanner motion. In general, the sharpness of an image increases as the phase error decreases. In other words, the image is sharpest when reconstructed with the correct phase. In some embodiments, the relationship between image sharpness and phase can be formulated as an optimization problem where the sharpness S of the image is maximized (or the blurriness B is minimized) with respect to the phase. In general, we can write:

[0059]

[0060] The objective function Can be or Or any combination of metrics that make the image sharper and / or less blurry.

[0061] The original phase delay may be continuously adjusted or adjusted using extreme value (i.e., minimum and maximum) calculations until the assumed phase delay is optimized by an amount. In one embodiment, a threshold-based blur metric for threshold prediction / correction may be used. In another embodiment, a variance-based sharpness metric may be used. In some embodiments, by adjusting the initial phase Perform local search nearby, metrics can be used to reduce errors This initial phase can be determined experimentally in a controlled laboratory environment. In one embodiment, the phase It can be predicted by performing a global search in the domain [0,π] and then fine-tuning the phase value by performing a local search.

[0062] Those skilled in the art will appreciate that the threshold metric may be less robust / accurate than the variance metric in the global search. However, it will also be appreciated that the threshold metric can be faster due to binary arithmetic that is more suitable for real-time phase correction. This efficiency and the attendant negligible sacrifice in accuracy may be valuable when analyzing raw data including high frame rates. Nonetheless, in practical implementations of MEMS-based endoscopes, both metrics are very effective in predicting phase error.

[0063] In one embodiment, rather than indicating the blurriness in an image directly, a "shorthand" method is used to represent the amount of repetition in an image with low computational complexity. In this embodiment, a predetermined threshold (σ, a constant given the optimization problem) is used to set the time series raw data R(t) to zero or one. This binarized data is used to reconstruct a black and white (B / W) image, where "binarization" means setting the intensity values ​​to 0 or 1 based on a threshold. The blur metric can be defined as the total number of similar pixels in a B / W image (i.e., all sum to 1 / 0). The normalized blur metric can be defined as

[0064]

[0065] A basic principle behind the above definition of blur is that when an image is generated using inaccurate phases, information is scattered throughout the image. Therefore, similar pixels can be scattered across the image and can therefore increase the value of the blur metric. In general, a combination of phase delays on both axes that includes the "best" image can be selected based on the blur metric and / or the sharpness metric.

[0066] Figure 1C The relationship between blur metric and overlap in a set of binarized images 112A-112D is depicted. Specifically, Figure 1CThe intuition behind the threshold metric for the set of binarized images 112-A to 112-D is illustrated. That is, the threshold metric (i.e., the number of similar pixels) increases as the phase error increases from 0 to 0.075π. The following table describes exemplary values ​​associated with the set of binarized images 112A-112D:

[0067] Table I

[0068] Fuzzy Metrics

[0069]

[0070] In one embodiment, Figure 1C The set of binarized images 112-A to 112-D may correspond to the binarized versions of the original image 102, the reconstructed image 104, the reconstructed image 106, and the reconstructed image 108, respectively. The blur metric may be minimized when all four images completely overlap one another; that is, when the output image is constructed using actual phase values. As the overlap increases, the blur decreases.

[0071] Figure 1D Depicts the graphical results of a global search that can be performed by the above phase prediction algorithm, where the global search independently scans from [0,π] in increments of 0.02 rad and Specifically, Figures 124 and 126 depict the and Graphs 128 and 130 depict the results of fine-tuning by local search around the extreme values ​​(i.e., minimum and / or maximum) obtained from the global search. rad comparison, the phase value predicted by the algorithm can be and rad. Conceptually, these results may be sensitive to the threshold (σ) chosen to binarize the original data. Although the global search has not been proven to be robust and is not guaranteed to provide the correct optimal value for all possible threshold choices, experiments show that the blur metric is always within The local minimum is independent of the choice of threshold.

[0072] As mentioned above, the main benefit of the threshold-based blurring technique is that the computation time for real-time image reconstruction is very short when the approximate phase is known. The algorithm works best when the mean is , because such a choice of mean approximately balances the counts of zeros and ones. Choosing a smaller value may increase the count by one, so that there will be some overlap of one (i.e., similar pixels) in the reconstructed image at any phase setting. Moreover, the sensitivity of the blur metric to the phase changes of the candidates is reduced. If it is too sparse, i.e. Then the ambiguity measure is almost constant with respect to the phase and has a very steep slope near the true phase value. As a result, the discrete steps taken when scanning the candidate phase values ​​may cause the true phase to be missed.

[0073] Example Variance-Based Clarity Metric Embodiment

[0074] In one embodiment, a variance-based sharpness metric among various available metrics may be used to detect and / or predict phase. In a sharp image, a set of pixel intensity values ​​may be well separated, thereby increasing the variance of the pixel intensities. The variance of the pixel intensities may be measured by the sharpness metric.

[0075]

[0076]

[0077] As in the techniques related to threshold-based blur metrics described above, in some embodiments, the phase variables may be independently swept from [0, π] and To perform a global search for the variance measure, such as Figure 1E As depicted by curves 134 and 136 . Figure 1E Depicted is a local search around an initial approximation given by a global search. Figure 1E The image is constructed and the sharpness metric in curves 138 and 140 is evaluated for each phase value. The sharpness metric of curves 138 and 140 can be plotted as a function of phase, which depicts the sharpness. Figure 1E As can be seen from the results, the image is likely to be clearest when it is closest to the true phase value. As noted, the solution can be fine-tuned by searching locally in small discrete steps. The phase values ​​predicted by the algorithm are and The comparison between the phase predicted by the above embodiment and the actual phase is given in the following table:

[0078] Comparison of results

[0079]

[0080] Figure 1FCombined graphs of threshold and negative variance metric versus phase x and y are depicted. Specifically, graphs 144 and 146 depict a global search for phase x to y from 0 to π, respectively. Graphs 148 and 150 depict a local search around an initial approximation given by the global search, respectively. Figure 1F The curve in can correspond to Figure 1D and 1E For example, curve 144 may be the result of combining blur metric graph 124 with sharpness metric graph 134, where sharpness metric graph 134 is negated to flip the vertical axis.

[0081] Example Tissue Imaging Embodiments

[0082] The present technique can be applied to any suitable sample. In some embodiments, the above method can be applied to a scanning mirror (e.g., a two-axis MEMS) mounted in an endoscope (e.g., a single-axis confocal endoscope) during tissue imaging. Prior to mounting, the phase delay of the scanning mirror can be first manually estimated for the x-axis. and for the y-axis When mounted in an endoscope and imaging sample tissue, as described above, the phase compensation algorithm can determine the second measurement as and

[0083] Figure 2A Depicted is an improvement in image quality representing a scan of human tissue (e.g., a tissue sample of a human colon) acquired before phase correction 202 (before image 202) and an image of the same tissue acquired after phase correction 204 (after image 204). For example, Figure 2A The images depicted in FIG. 1 can be produced by a two-axis scanning micromirror in a single-axis confocal imaging instrument, where the instrument corresponds to Figure 1B In some embodiments, Figure 2A The tissue sample depicted in may correspond to Figure 1B , and image 202 and / or image 204 may correspond to an object H depicted in FIG. Figure 1B Image I in .

[0084] Example Photo Example

[0085] This technique can be used in conjunction with images, e.g. Figure 2B Test images 210-218 are depicted in FIG. Figure 2BThe images may include a pirate 210, a mandrill 212, a lake 214, a chemical plant 216, and a photographer 218. The test images 210-218 may be used to demonstrate the effectiveness of the phase detection algorithm and the metrics. The test images 210-218 may be compressed to 256×256 pixels. A scanner as described above may be used, using f x =19251Hz,f y =3315Hz, and The raw data is simulated for each test image 210 - 218 by using the scanning frequency and phase of the rad to simulate the laser scanning pattern of each test image 210 - 218. As described above, global and local search algorithms can be applied to the raw data to predict the phase value. Figure 2C The output of applying blurriness and sharpness metrics to predict phase values ​​is depicted. Graphs 220 and 222 depict a global search from 0 to π for each of the test images 210-218, respectively, scanning about the blurriness metric x and y axes, respectively. Graphs 224 and 226 depict a global search from 0 to π for each of the test images 210-218, respectively, scanning about the negative sharpness metric x and y axes, respectively. In one embodiment, scanning the images 210-218 results in less than 10 -3 The error in radians.

[0086] Example Image Reconstruction

[0087] As described above, the Lissajous scanning trajectory may depend on the driving frequencies and the relative phase difference between them. For example, a repetitive or non-repetitive pattern may be obtained depending on whether the ratio of the driving frequencies is a rational number or an irrational number, respectively. When different pixels are scanned in each cycle (e.g., a cycle of a lower frequency), a non-repetitive curve may be required to cover more area on the object plane and improve FF. FF can be defined as the ratio of the number of pixels scanned at least once in the field of view to the total possible number of pixels. In high-definition (HD) images, the FF can be made high enough by scanning for a long time, which may result in a lower frame rate. Typically, the effect results in a trade-off between FR and FF because it is difficult to maximize FR and FF for HD images. In high FR applications using Lissajous scanning, the reconstructed image may have missing pixels. As a mitigating factor, various algorithms can be used in a post-processing step (e.g., after the initial image reconstruction) to complete these missing pixels. However, in embodiments in which the phase prediction step is prior to other steps, it may be difficult to eliminate missing pixels during phase prediction.

[0088] A large number of samples (e.g., 10 in total) can be taken 6 ) to simulate images (e.g. Figure 1A The original data of the original image 102). 6The number of pixels in the original image may be much larger (e.g., a factor of 20 or more) than the number of pixels in the original image. If the image is reconstructed using the full data set, then there may be no missing pixels. However, in some embodiments, a subset of the original data (e.g., 1 / 4 of the original data) may be used to study the effect of the missing pixels on the blur metric to predict the phase. In high FR imaging applications, pixel loss is unavoidable and will adversely affect both metrics and may cause errors in the results. For example, continuing with the example, only the first 0.25x10 of the original data may be used. 6 samples.

[0089] Go to Figure 3A , depicts a reconstructed image 302, where missing pixels are shown by dashed areas. Image 302 may have a high FR. In some embodiments, in an image reconstructed with true phase information, missing pixels may be marked with a separate color. Figure 3B Various graphs of threshold and variance metrics are depicted relative to phase y, where the effect of missing pixels is also shown. Specifically, curve 304 depicts the effect of missing pixels on the blur metric, while curve 306 depicts the effect of missing pixels on the sharpness metric. Figure 1D and Figure 1F Compared with curve 146, the fuzzy metric is has a high frequency variation. The percentage of missing pixels is given by The function of is plotted on the right y-axis of graphs 304 and 306. The high frequency noise peaks in the blur metric of graph 304 align with the missing pixels. If an image has missing pixels, sharpness-based phase prediction methods may also be affected. In extreme cases, if the contribution of the missing pixels to the sharpness metric exceeds the depth of the global minimum at the true phase, the phase prediction method may fail.

[0090] Examples of overcoming problems in conventional Lissajous scanning

[0091] In one example, techniques for timing of image binarization are provided. Figure 4 Describes the effect of binarization of raw data. For example, Figure 4 Included are images 402-A through 402-E, which respectively illustrate the use of The effect of binarizing the raw data with a threshold of c = 1.3, 1.4, ..., 1.7. Of course, any suitable value of c can be selected. In the threshold-based blur metric embodiment, the raw data itself can be binarized to directly construct a B / W image, as shown in images 402-A to 402-E. The raw data can correspond to Figure 1A and / or Figure 1CIn one embodiment, a grayscale image can be generated from the raw data and then binarized into a B / W image, as shown in images 404-A to 404-E. In some embodiments, binarization is performed directly on the raw data, as shown in Figure 4 As shown, a clearer image can be produced, where the images 402-A to 402-E of the coin are clear, where the images 404-A to 404-E are less clear. Therefore, binarizing the raw data before forming the image can provide better detection of the phase with increased sensitivity. However, in other embodiments, a binarized grayscale image may be preferred.

[0092] Another solution provided by the present technology involves solving the problem of uneven scan density. Lissajous scanning can result in uneven scan density, where the number of data points sampled per unit area in the field of view varies between regions of the scanner. For example, Figure 5 As shown, the density of the image may be lowest in the center of the scan and highest at the edges of the scan. Figure 5 In the example shown, a 240x160 pixel image is displayed, which depicts the uneven scan density, which is represented by f x =19251Hz,f y =3315Hz generated. Figure 5 In the example, the center of the scanner and the edges of the scanner have non-uniform density, respectively, because the scanner speed may be faster in the center and slower at the edges. Scanning the object multiple times may exacerbate the non-uniformity. Scanning density is one aspect of this technology that can be corrected to avoid oversaturating the image. For illustration purposes, the image is intentionally simplified. Figure 5 In some embodiments, the scan density may be represented using a color scale (eg, a color gradient) or in another suitable manner.

[0093] Example Environment

[0094] Figure 66 is an example block diagram 600 showing various components used in an example embodiment for implementing a scanner 602 described herein that allows adjustment of phase delay. The MEMS component 622 discussed previously herein may be positioned near or operably coupled to an object 624 in accordance with the functions of performing the disclosed embodiments. The scanner 602 may have a controller 604 that is operably connected to a database 616 via a link connected to an input / output (I / O) circuit 614. It should be noted that, although not shown, additional databases may be linked to the controller 604 in a suitable manner. The controller 604 includes a program memory 606, a processor 608 (e.g., a microcontroller and / or microprocessor), a random access memory (RAM) 610, a network interface 612, and an input / output (I / O) circuit 614, all of which are interconnected by an address / data bus. It should be understood that, although only one microprocessor 608 is shown, the controller 604 may include multiple microprocessors 608. Similarly, the memory of the controller 604 may include multiple RAMs 610 and multiple program memories 606. Although I / O circuit 614 is shown as a single block, it should be understood that I / O circuit 614 may include a variety of different types of I / O circuits. For example, RAM 610 and program memory 606 may be implemented as semiconductor memory, magnetically readable memory, and / or optically readable memory. An address / data bus may communicatively connect controller 604 to MEMS 622 150 via I / O circuit 614.

[0095] The program memory 606 and / or RAM 610 may store a variety of application programs (i.e., machine-readable instructions) for execution by the microprocessor 608. For example, an operating system may generally control the operation of the MEMS 622 and provide a user interface to implement the processes described herein. The program memory 606 and / or RAM 810 may also store various subroutines for accessing specific functions of the MEMS 622. As examples and not limitations, the subroutines may include, among others: subroutines for controlling the operation of the MEMS 622 or other endoscopic devices as described herein; subroutines for capturing images using the MEMS 622 as described herein; subroutines for predicting and / or correcting phase drift as described herein; and other subroutines, such as, for example, implementing software keyboard functions, subroutines for interfacing with other hardware in the MEMS 622, and the like. The program memory 606 and / or RAM 610 may also store data related to the configuration and / or operation of the MEMS 622, and / or related to the operation of one or more subroutines. For example, the data may be data collected by the MEMS 622, data determined and / or calculated by the processor 608, and the like. In addition to the controller 604, the MEMS 622 may also include other hardware resources. For example, in some embodiments, the MEMS may be part of an endoscope and / or laparoscope (not shown), which may be coupled to various types of input / output hardware, such as a visual display 618 and / or an input device 620 (e.g., a keypad, a keyboard, etc.). Such input and output devices may allow a user to interact with the controller 604 to fine-tune the actuation of the axial and / or lateral scanners. In one embodiment, the display 618 may be touch-sensitive and may cooperate with a software keyboard routine as one of the software routines to accept user input.

[0096] Instance Methods

[0097] Figure 7 A method 700 for correcting phase shift in a scanner is depicted. The method may include receiving a raw data set corresponding to an image of a sample (block 702). The raw data set may correspond to Figure 1B The original data vector shown in . The samples can correspond to Figure 1A The coins shown in Figure 1B The objects shown in and / or Figure 2A The original dataset can be accessed via Figure 6 For example, MEMS 622 can receive the I / O circuit 614 according to the Figure 1BThe scanning and translation process discussed above is used to generate the raw data vector. Method 700 can identify a first set of extreme values ​​by analyzing the raw data set using a metric algorithm (box 704). For example, the method can include a global search along two axes, such as a mirror scan in two directions. When constructed based on a blur metric and / or a clarity metric, method 700 can select a combination of phase shifts that produces the best image. For example, as described above, the blur level in the final constructed image can be minimized, and / or the clarity level can be maximized. Once the global extreme value group is identified, method 700 can search for the global extreme value or search near it to identify a second set of local extreme values ​​(box 706). For example, as described above with respect to Figure 1D As discussed, curves 124 and 126 represent a global search using a fuzzy metric, while curves 128 and 130 represent a local search around an initial approximation. Figure 1E denoted global and local search using a sharpness / variance metric.Once the global and local search has identified a second set of extrema, the minimized and / or maximized values ​​may be used to regenerate the image taking into account the phase drift (block 708).

[0098] Additional Considerations

[0099] In the above, scanning technology is discussed for MEMS scanners related to endoscopic treatment. Any suitable sampling / scanning method can be used to implement the present technology, including but not limited to Lissajous scanning, raster scanning, spiral scanning, etc. In addition, those skilled in the art will understand that the present technology can be applied to other fields / domains for different applications, including but not limited to imaging, projection display technology, 3D printing technology, scanning electron microscopy and atomic force microscopy.

[0100] The following considerations also apply to the foregoing discussion. Throughout the specification, multiple instances may implement operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as individual operations, one or more of the individual operations may be performed simultaneously, and the operations do not need to be performed in the order shown. These and other variations, modifications, additions, and improvements all fall within the scope of the subject matter of this article.

[0101] Unless otherwise stated, discussions herein using words such as "processing," "computing," "calculating," "determining," "presenting," "displaying," and the like may refer to the action or process of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0102] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0103] As used herein, the terms "including", "having" or any other variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, article, or device that includes a list of elements is not necessarily limited to those elements, but may include other elements that are not explicitly listed or that are inherent to such a process, method, article, or device. Further, unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, any of the following satisfies condition A or B: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0104] In addition, the use of "a / an" is used to describe the elements and components of the embodiments herein. This is done only for convenience and to give a general sense of the present invention. The description should be understood to include one or at least one, and the singular also includes the plural, unless otherwise clearly indicated.

[0105] After reading this disclosure, those skilled in the art will understand that additional alternative structures and functional designs that implement the concepts disclosed herein are implemented by the principles disclosed herein. Therefore, although specific embodiments and applications have been depicted and described, it should be understood that the disclosed embodiments are not limited to the precise configurations and components disclosed herein. Various modifications, changes and variations that are obvious to those skilled in the art may be made to the arrangement, operation and details of the methods and devices disclosed herein without departing from the spirit and scope defined by the appended claims.

Claims

1. A computer-implemented method for predicting and / or correcting phase drift, the method comprising: receiving, by a processor, a raw data set representing a sample image, identifying a first set of extrema by analyzing the raw data set using a metric algorithm, the first set of extrema being global extrema corresponding to phase values ​​along orthogonal axes, identifying a second extremum set by analyzing the first extremum set, the second extremum set being local extrema corresponding to phase values ​​along the orthogonal axis, and Based on the second extrema set, a reconstructed image of the sample is generated, the reconstructed image having a corrected phase.

2. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, further comprising: include: The raw data set was acquired using a single-axis confocal endoscope.

3. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, further comprising: include: The raw data set was acquired using a dual-axis confocal endoscope.

4. The computer-implemented method of predicting and / or correcting phase drift of claim 1, wherein receiving a raw data set representing an image of a sample comprises using a micro-electromechanical system mirror or a galvanometer mirror.

5. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, wherein the sample is a tissue sample.

6. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, wherein identifying the first set of extreme values ​​comprises experimentally determining the initial set of extreme values ​​in a controlled environment, and Wherein identifying the second extremum set by analyzing the first extremum set includes performing a local search near the first extremum set.

7. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, wherein identifying the first set of extreme values ​​comprises determining an approximate initial set of extreme values ​​using a global search over a predetermined domain, and Wherein identifying the second extremum set by analyzing the first extremum set includes performing a local search near the approximate initial extremum set. 8 . The computer-implemented method of predicting and / or correcting phase drift of claim 1 , wherein identifying the first set of extreme values ​​comprises using a gradient search algorithm or a Fibonacci search algorithm.

9. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, wherein the metric algorithm is a fuzzy metric, and Wherein identifying the first extreme value set by analyzing the original data set using the fuzzy metric includes binarizing the original data using a predetermined threshold to generate a binarized data set.

10. The computer-implemented method of predicting and / or correcting phase drift according to claim 9, wherein the predetermined threshold is an average of a set of values ​​in the raw data.

11. The computer-implemented method of predicting and / or correcting phase drift according to claim 1, wherein the metric algorithm is a sharpness metric, and Wherein identifying the first extreme value set by analyzing the original data set using the sharpness metric includes calculating the variance of pixel intensities.

12. A computing system, include: Phase correction scanner, one or more processors, Display devices, A memory storing machine-readable instructions that, when executed by the one or more processors, cause the computing system to: sampling an object in an image via the phase correction scanner to generate a one-dimensional time series data set representing the image of the object, the one-dimensional time series data set having intensities corresponding to the image of the object at predetermined sampling intervals, Compute a metric for the one-dimensional time series, Repeat the calculation of the extreme value until the phase delay is optimized. mapping the one-dimensional time series data set to a two-dimensional image space corresponding to the object, and The two-dimensional image is displayed in the display device.

13. The computing system of claim 12, wherein the predetermined sampling interval is a Lissajous pattern.

14. The computing system of claim 12, wherein the predetermined sampling interval is a grid pattern or a spiral pattern.

15. A phase correction scanner, include: one or more processors; one or more scanners adapted to sequentially sample an object to generate vectors of raw data representing the object in a Lissajous pattern, and a memory storing instructions that, when executed by the one or more processors, cause the phase correction scanner to: receiving a vector of said raw data, identifying a first set of extrema by analyzing the raw data set using a metric algorithm, the first set of extrema being global extrema corresponding to phase values ​​along orthogonal axes, and identifying a second set of extrema by analyzing the first set of extrema, the second set of extrema being local extrema corresponding to phase values ​​along the orthogonal axis, wherein identifying the first set of extreme values ​​comprises experimentally determining the initial set of extreme values ​​in a controlled environment, and Wherein identifying the second extremum set by analyzing the first extremum set includes performing a local search near the first extremum set.

16. The phase correction scanner of claim 15, the memory storing further instructions which when executed cause the phase correction scanner to: Determine an approximate initial set of extrema using a global search over a predetermined domain, and A local search is performed near the approximate initial set of extrema.

17. The phase correction scanner of claim 15, wherein the metric algorithm is a blur metric, the memory storing further instructions which when executed cause the phase correction scanner to: A binarized data set is generated by binarizing the original data using a predetermined threshold.

18. The phase correction scanner of claim 17, wherein the predetermined threshold is an average value of a set of values ​​in the raw data.

19. The phase correction scanner of claim 15, wherein the metric algorithm is a sharpness metric, the memory storing further instructions which when executed cause the phase correction scanner to calculate a variance of pixel intensities.