Super-resolution imaging systems and methods including event-based imaging sensors

Abstract

A super-resolution imaging method (400) can include using an event-based imaging sensor (410) having one or more pixels to receive light originating from a first portion of a scene within a focused spot. At least one of the event-based imaging sensor and the scene can be moved (420) relative to one another so that a different second portion of the scene is in the focused spot. The event-based imaging sensor can be used to detect a change in light intensity (430) caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot. The feature can have a dimension smaller than the focused spot. A characteristic of the feature can be estimated (440) based on the change in light intensity and motion of the scene relative to the focused spot. A super-resolution image showing the feature can then be generated (450).

Description

[0001] SUPER-RESOLUTION IMAGING SYSTEMS AND METHODS INCLUDING EVENT-BASED IMAGING SENSORS

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 63 / 560,897, filed March 4, 2024, which is hereby incorporated herein by reference.

[0004] STATEMENT REGARDING FEDERALLY SPONSORED

[0005] RESEARCH OR DEVELOPMENT

[0006] Not applicable.

[0007] NAMES OF THE PARTIES TO A JOINT RESEARCH AGREEMENT

[0008] Not applicable.

[0009] INCORPORATION BY REFERENCE STATEMENT

[0010] Not applicable.

[0011] BACKGROUND

[0012] Event-based imaging sensors (EBIS) respond to changes in intensity of light. Each EBIS pixel can respond to a change in the logarithmic intensity of light (a measure of brightness under uniform lighting conditions). If lighting conditions do not change (as is the case in optical microscopy), then the signal represents a change in local reflectance or fluorescence from the sample. EBIS cameras have been used for macro-imaging such as in robotics, and for tracking macroscopic objects such as vehicles, stars, satellites, and others. Since the data format of EBIS cameras is recorded in events rather than in frames, EBIS can utilize new imaging processing methods referred to as “event processing” rather than conventional image processing. EBIS cameras can deliver high effective framerates when significant care and calibration is utilized to process the acquired data. SUMM RY

[0013] An example super-resolution imaging method can include using an event-based imaging sensor having one or more pixels to receive light originating from a first portion of a scene a within a focused spot. The event-based imaging sensor or the scene can be moved relative to one another so that a different second portion of the scene is in the focused spot. The event-based imaging sensor can be used to detect a change in light intensity caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot. The feature can have a dimension smaller than the focused spot. A characteristic of the feature can be estimated based on the change in light intensity and motion of the scene relative to the focused spot. A super-resolution image can then be generated, showing the feature.

[0014] The present disclosure also describes super-resolution imaging systems. An example super-resolution imaging system can include an event-based imaging sensor having one or more pixels configured to receive light originating from a first portion of a scene within a focused spot. The event-based imaging sensor can be configured to move relative to the scene such that a different second portion of the scene moves into the focused spot. The system can also include a processor configured to detect a change in light intensity using the event-based imaging sensor. The change in light intensity can be caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot. The feature can have a dimension smaller than the focused spot. The processor can also estimate a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot, and generate a super-resolution image showing the feature.

[0015] There has thus been outlined, rather broadly, the more important features of the invention so that the detailed description thereof that follows may be better understood, and so that the present contribution to the art may be better appreciated. Other features of the present invention will become clearer from the following detailed description of the invention, taken with the accompanying drawings and claims, or may be learned by the practice of the invention. BRIEF DESCRIPTION OF THE DR WINGS

[0016] FIG. l is a schematic view of a focused spot moving relative to a scene in an example method in accordance with an example of the present technology.

[0017] FIGs. 2A-2C are schematic views of a focused spot moving relative to a scene containing fluorescent beads in an example method in accordance with examples of the present technology.

[0018] FIGs. 3A-3C are schematic views of a focused spot moving relative to another scene containing fluorescent beads in an example method in accordance with the present technology.

[0019] FIG. 4 is a flowchart illustrating an example super-resolution imaging method in accordance with the present technology.

[0020] FIG. 5 is a schematic view of an example super-resolution imaging system in accordance with the present technology.

[0021] FIG. 6A is a schematic view of another example super-resolution imaging system in accordance with the present technology.

[0022] FIG. 6B is a schematic representation of a two-dimensional grid illustrating the way the line beam is dispersed into a spectrum along the axis perpendicular to the length of the line beam in accordance with an example of the present technology.

[0023] FIG. 7 is a schematic view of yet another example super-resolution imaging system in accordance with the present technology.

[0024] FIG. 8 is a schematic view of another example super-resolution imaging system in accordance with the present technology.

[0025] These drawings are provided to illustrate various aspects of the invention and are not intended to be limiting of the scope in terms of dimensions, materials, configurations, arrangements or proportions unless otherwise limited by the claims.

[0026] DETAILED DESCRIPTION

[0027] While these exemplary embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, it should be understood that other embodiments may be realized and that various changes to the invention may be made without departing from the spirit and scope of the present invention. Thus, the following more detailed description of the embodiments of the present invention is not intended to limit the scope of the invention, as claimed, but is presented for purposes of illustration only and not limitation to describe the features and characteristics of the present invention, to set forth the best mode of operation of the invention, and to sufficiently enable one skilled in the art to practice the invention. Accordingly, the scope of the present invention is to be defined solely by the appended claims.

[0028] Definitions

[0029] In describing and claiming the present invention, the following terminology will be used.

[0030] The singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to “a feature” includes reference to one or more of such elements and reference to “the processor” refers to one or more of such components.

[0031] As used herein with respect to an identified property or circumstance, “substantially” refers to a degree of deviation that is sufficiently small so as to not measurably detract from the identified property or circumstance. The exact degree of deviation allowable may in some cases depend on the specific context.

[0032] As used herein, “adjacent” refers to the proximity of two structures or elements. Particularly, elements that are identified as being “adjacent” may be either abutting or connected. Such elements may also be near or close to each other without necessarily contacting each other. The exact degree of proximity may in some cases depend on the specific context.

[0033] As used herein, the term “about” is used to provide flexibility and imprecision associated with a given term, metric or value. The degree of flexibility for a particular variable can be readily determined by one skilled in the art. However, unless otherwise enunciated, the term “about” generally connotes flexibility of less than 2%, and most often less than 1%, and in some cases less than 0.01%.

[0034] As used herein, a plurality of items, structural elements, compositional elements, and / or materials may be presented in a common list for convenience. However, these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus, no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on their presentation in a common group without indications to the contrary.

[0035] As used herein, the term “at least one of’ is intended to be synonymous with “one or more of.” For example, “at least one of A, B and C” explicitly includes only A, only B, only C, or combinations of each.

[0036] Numerical data may be presented herein in a range format. It is to be understood that such range format is used merely for convenience and brevity and should be interpreted flexibly to include not only the numerical values explicitly recited as the limits of the range, but also to include all the individual numerical values or sub-ranges encompassed within that range as if each numerical value and sub-range is explicitly recited. For example, a numerical range of about 1 to about 4.5 should be interpreted to include not only the explicitly recited limits of 1 to about 4.5, but also to include individual numerals such as 2, 3, 4, and subranges such as 1 to 3, 2 to 4, etc. The same principle applies to ranges reciting only one numerical value, such as “less than about 4.5,” which should be interpreted to include all of the above-recited values and ranges. Further, such an interpretation should apply regardless of the breadth of the range or the characteristic being described.

[0037] Any steps recited in any method or process claims may be executed in any order and are not limited to the order presented in the claims. Means-plus-function or step-plus- function limitations will only be employed where for a specific claim limitation all of the following conditions are present in that limitation: a) “means for” or “step for” is expressly recited; and b) a corresponding function is expressly recited. The structure, material or acts that support the means-plus function are expressly recited in the description herein. Accordingly, the scope of the invention should be determined solely by the appended claims and their legal equivalents, rather than by the descriptions and examples given herein. Example Embodiments

[0038] Microscopy, telescopes, photography, semiconductor metrology, and other optical technologies use various types of cameras, sensors, or imaging devices. Event-based imaging sensors are a type of bio-inspired sensors that serve as event cameras. Event-based imaging sensors differ from other cameras in that instead of capturing images at a fixed rate, they measure per-pixel brightness changes, and the output of a data stream of events that encode time, location, and brightness changes. These event-based image sensors can offer useful properties compared to traditional cameras, such as high temporal resolution, high dynamic range, low power consumption, high pixel bandwidth, and reduced motion blur. Hence, event-based image sensors have a large potential for robotics and computer vision in difficult cases for traditional cameras, such as high speed and high dynamic range. Image processing software for use with event-based imaging sensors can be different than software for traditional cameras. For example, the software can receive a stream of event data, which may be separated by varying small time increments, instead of receiving data for entire image frames at a set frame rate.

[0039] The present technology involves a way of utilizing event-based sensors to allow the capture of super- resolution images. For example, the systems and methods described herein can be used to capture images with a resolution finer than the diffraction limited resolution. All lenses have an upper-performance limit dictated by physics, known as the diffraction limit. Even a perfect lens, with no design flaws, is still limited by diffraction. The diffraction limited resolution is also referred to as the cutoff frequency of a lens. The diffraction limit can be calculated using Equation 1 : ^Diffraction Limit where f / # is the f-number of the lens (also referred to as f-stop, or the ratio of the lens’s focal length to its aperture diameter) and 2 is the wavelength of light in millimeters. The diffraction limit has units of line pairs per millimeter. The diffraction limit is also related to the Airy disk, which is a disk the size of the smallest point to which a beam of light can be focused. The Airy disk is characterized by a diffraction pattern made up of a bright center region and concentric rings around the center. As Airy patterns from different objects approach one another, they begin to overlap, and eventually the patterns from the different objects cannot be distinguished from each other. The Airy disk can represent the smallest possible focused spot due to diffraction. Features that are smaller than this minimum focused spot and closer together than the size of this minimum focused spot cannot be resolved simultaneously using a traditional camera sensor, even with a flawless lens. In fact, the size of the minimum focused spot can often be larger than the size of pixels of current image sensor devices. As an example, the Airy disk diameter with a lens having an f-number of 4 and a light wavelength of 520 nm is 5.08 gm, and the Airy disk diameter with a lens having an f-number of 16 and a light wavelength of 520 nm is 20.3 gm.

[0040] The present technology can utilize the high temporal resolution of the event-based imaging sensor to resolve features in images that are smaller than the focused spot, and which may be smaller even than the minimum diffraction limited focused spot. In particular, the systems and methods described herein can be used to resolve features that would not be possible to resolve using a traditional optical camera because of the diffraction limit. In some cases, the system can be used for microscopy, and the system can be capable of resolving features having a dimension of 5 pm or less, or 1 pm or less, or 500 nm or less, or 200 nm or less, or 150 nm or less, or 100 nm or less, or 75 nm or less, or 50 nm or less, or from 5 nm to 500 nm, or from 5 nm to 200, or from 5 nm to 150 nm, or from 5 nm to 100 nm, or from 5 nm to 75 nm, or from 5 nm to 50 nm.

[0041] The systems described herein can include an event-based image sensor that can be focused on a focused spot. The focused spot can move across a scene, such as by scanning across a scene. The event-based sensor can detect changes in brightness with a high temporal resolution. Based on the movement speed of the focused spot and the times when changes in brightness occur, the features that caused the changes in brightness can be resolved even when these features are smaller or closer together than size of the focused spot. The resolution can be finer than the spot size, even when the spot size is limited by diffraction or by limitations of the lens, such as imperfections in the lens. Software can be used to process the event data to provide a high signal to noise ratio in the recorded images. Thus, images can be produced that can surpass the spatial resolution imposed by diffraction or by limitations of the lens. This can be accomplished by harnessing the temporal resolution capabilities of an event-based sensor and utilizing computational techniques to extract images at a spatial resolution finer than would otherwise be possible. The systems and methods described herein can be applied in various fields, such as microscopy, semiconductor metrology, photography, telescopes, and other imaging systems. The methods can be compatible with almost any signal modality, including fluorescence, scattering, reflection, self-emission, phosphorescence, transmission, phase contrast, and more.

[0042] In certain examples, a super-resolution imaging method can include using an eventbased imaging sensor having one or more pixels to receive light originating from a first portion of a scene a within a focused spot. The event-based imaging sensor or the scene can be moved relative to one another so that a different second portion of the scene is in the focused spot. The event-based imaging sensor can be used to detect a change in light intensity caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot. The feature can have a dimension smaller than the focused spot. A characteristic of the feature can be estimated based on the change in light intensity and motion of the scene relative to the focused spot. A super-resolution image can then be generated, showing the feature.

[0043] In some cases, the characteristic can be a position, a dimension, an orientation, or a combination thereof. In certain examples, the feature of the first portion or the feature of the second portion can be a particle. The feature of the first portion or the feature of the second portion can have a dimension smaller than a diffraction limit. Additionally, the superresolution image can have a resolution finer than a diffraction-limited resolution. In some examples, the focused spot can move a distance that is smaller than the focused spot when detecting the change in light intensity. The distance that the focused spot moves can also be smaller than a diffraction limit. The light that originates from the scene can be from fluorescence, scattering, reflection, self-emission, phosphorescence, transmission, phasecontrast, or a combination thereof.

[0044] It is noted that the methods and systems described herein may not rely on temporal changes of brightness of the scene being imaged, such as in other super-resolution microscopy methods that rely on emitters that emit light at different times. Some other methods utilize fluorophores that change brightness over time in order to resolve these fluorophores in time and thereby resolve features smaller than the diffraction limit. In contrast, the methods and systems described herein can be used to image any features, including features that have a constant or static brightness. The events captured by the eventbased image sensor can arise due to movement of the focused spot, and not by any actual change in brightness of the scene or subject. In further examples, the methods and systems described herein can be used with a combination of features that have constant brightness and also features that have changing brightness. In certain examples, the methods and systems described herein can be combined with other super-resolution microscopy methods that involve the use of flashing fluorophores or other such emitters. FIG. 1 shows a schematic example of a focused spot 1 10. In this figure, the focused spot is represented by a circle. The focused spot can encompass a first portion of a scene, which includes any features of the scene that fall within this circle. The focused spot can then be moved to a second position, depicted by circle 120. In this position, the focused spot encompasses a second portion of the scene. The diameter of the focused spot is shown in this figure as Ax. Although the event-based image sensor can detect changes of brightness of light originating from objects within this focused spot, the sensor may not be capable of resolving multiple objects or features within this focused spot, i.e., smaller than Av. If the focused spot is the minimum diffraction limited focused spot, then Ax can represent the diffraction limit. The distance moved by the focused spot between circle 110 and 120 is shown as dx. This distance can be smaller than the diameter of the focused spot. In some examples, the initial position of the focused spot can be referred to as position xi, and the second position can be xi + <)x. The focused spot can begin at position xi at time A, and then the focused spot can move to position xi + dx at time t + dt.

[0045] FIG. 2A shows an example in which the focused spot scans across a scene that includes two fluorescent beads 240. The focused spot begins at a first positions shown as circle 210. The two fluorescent beads are closely spaced at locations (xi, i) and (X2, y ). The focused spot is scanned over the sample in the horizontal direction from left to right. At time Zi, the left fluorescent bead is illuminated and will emit photons. This will generate an event signal on the event-based sensor with the following co-ordinates: (xi, yi, Zi, 1), where xi is the location along the x-axis, yi is the location along the y-axis, ti is the time of the event, and the last value of “1” is the polarity of the signal (positive one denoting an increasing brightness). FIG. 2B shows the focused spot at time Zi + dt, when the focused spot has moved to the position of circle 220. In this example, the unit of time dt is the smallest time difference between measured events that the event-based imaging sensor is capable of capturing. At time ti + dt, a new event signal is generated with the co-ordinates: (X2, yi, h + dt, 1) denoting the increasing brightness from the fluorophore on the right. At this instant, there is no event at (xi, i) as there is no brightness change (within the threshold determined by the eventbased pixel). As a result, these two fluorophores can be resolved as two separate events even though they are spaced closer than the diffraction limit. Furthermore, at time, Zi + 2dt, the focused spot has moved to the position of circle 230 as shown in FIG. 2C. At this time, a new event is generated at h + 2b / , -1), when the fluorescence from the left fluorophore disappears resulting in a decrease in brightness (denoted by the -1 value in polarity). Subsequent event-processing can then be used to resolve these closely-spaced fluorophores. Redundant data as illustrated here can also be used to enhance the signal -to-noise ratio. For example, if the 3t is smaller, then events can be measured closer together and the position of the fluorophore beads can be determined more precisely.

[0046] FIG. 3A shows another example with a more complex scene that includes five fluorescent beads 340. The focused spot moves from the position of circle 310 to circle 320 in FIG. 3B, and then to circle 330 in FIG. 3C. This can result in a more complex set of event coordinates that can be computationally processed to resolve the image with superresolution. In this example, an event can be recorded at the time of FIG. 3 A, when two of the fluorescent beads enter the focused spot, which registers as an increase of brightness captured by the event-based image sensor. If the brightness of individual fluorescent beads is known, then it can be determined that two beads have moved into the focused spot based on the brightness change. If the next event is measured at the time of FIG. 3B, another brightness change equivalent to two beads can be recorded. Then, at the time of FIG. 3C, a brightness change equivalent to one bead can be measured. If the focused spot is scanning along the x- axis, then this process can provide the x-axis location of the two beads in the focused spot in FIG. 3 A, and the x-axis location of the next two beads that enter the focused spot in FIG. 3B, and the x-axis location of the last bead that enters the focused spot in FIG. 3C. Additional information can be provided by continuing to scan in the x-axis direction and recording events when these beads leave the focused spot. If the time between measured events (dt) can be reduced, then the x-axis locations of the beads can be determined more precisely. Additionally, the scanning can also be performed in another direction, such as along the y- axis, to provide more information about the y-axis location of the beads.

[0047] The pixels in the event-based imaging sensor can be configured to send data when a brightness change above a certain threshold is detected. The threshold can be chosen empirically via experiments with actual samples or scenes. The threshold can also be chosen using simulations if the parameters are well known, such as in the case of inspection of semiconductor chips. The methods described herein can also be applied to photography, machine-vision applications, microscopy, semiconductor metrology (inspection and defect classification), telescopy, satellite imaging, and other applications.

[0048] FIG. 4 is a flowchart illustrating an example super-resolution imaging method 400. This method includes: using an event-based imaging sensor having one or more pixels, receiving light originating from a first portion of a scene a within a focused spot 410; moving at least one of the event-based imaging sensor and the scene relative to one another such that a different second portion of the scene is in the focused spot 420; using the event-based imaging sensor, detecting a change in light intensity caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot, wherein the feature has a dimension smaller than the focused spot 430; estimating a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot 440; and generating a super-resolution image showing the feature 450.

[0049] As used herein, a “feature” can include any object or portion of an object that is of interest in the scene, such as a particle, a shape, a void space, a colored portion, a light portion, a dark portion, and so on.

[0050] As used herein, a “characteristic” of a feature can be a position, a dimension, an orientation, or a combination thereof. For example, if the scene being imaged includes particles that are smaller than the focused spot, then the particles can be the features referred to in the method and the characteristic can be the position of particles, the size of particles, the orientation of particles, and so on.

[0051] The event-based imaging sensor can move in relation to the scene being imaged. This movement can be accomplished by moving the event-based imaging sensor while the scene remains stationary, or by moving the scene while the event-based imaging sensor remains stationary, or by moving both the event-based imaging sensor and the scene. In some examples, the movement can result in the focused spot scanning across the scene at a known rate. Any reference to “scanning the focused spot,” “moving the focused spot,” and the like as used herein can refer to moving the event-based imaging sensor, moving the scene, or any combination thereof which results in a different portion of the scene coming into the focused spot. When the event-based imaging sensor detects changes in brightness due to the presence of features in the scene, the time of detection of these features combined with the known scan rate can allow the position of the features to be calculated. Tn some examples, the method can include scanning the focused spot across the scene in one direction. In other examples, the method can include scanning the focused spot across the scene in multiple directions. For example, the focused spot can be scanned across the scene in a first direction and then scanned across the scene again in a second direction that is orthogonal to the first direction. In a particular example, the focused spot can be scanned from left to right and then from top to bottom. This can allow the position of features to be estimated along the left to the right direction (i.e., the x-coordinates of the features) and then scanning from top to bottom can allow the position to be estimated in the up and down direction (i.e., the y- coordinates of the features). In certain examples, some features can be aligned at the same x- axis location or the same y-axis location, making it more difficult to resolve these features. For example, a pair of fluorescent beads that both have same x-axis location can move into the focused spot at the same time, causing both beads to be measured as a single event. Therefore, it can be useful to scan the scene in the orthogonal y-axis direction. When the two beds are scanned in the y-axis direction, the beads can be more easily resolved because the beads have different y-axis locations. In further examples, it may be useful to scan the scene in more than two directions. For example, a scene can be scanned along the x-axis, and then along the y-axis, and then at an angle 45° with respect to the x-axis, and then at an angle - 45° with respect to the x-axis, and so on. If the brightness of individual beads is known, then it can be determined whether events result from the brightness change of individual beads or combinations of two or more beads. In some examples, the scene can be scanned and rescanned in multiple different directions until it is determined that all beads have been resolved individually. In other examples, a certain fraction can be selected as a threshold, such as 90%, and then the scene can be scanned in different directions until at least that threshold fraction of the beads have been resolved individually. These examples are described as applying to fluorescent beads; however, the same processes can be used with any type of feature that is desired to be resolved using the event-based image sensors.

[0052] An example algorithm that can be used to estimate a characteristic of a feature, such as the position of a particle or other feature, can include the following steps: (1) Capture event data; (2) Cluster events to identify particles. This is easiest to perform when there is prior knowledge of the size of the particles. If no prior information is available, this can be done based on the spatial resolution of the lens used. Clustering events can include grouping positive events (increasing brightness) and negative events (decreasing brightness) that are separated spatially by approximately the width of the focused spot, indicating that the positive and negative events are caused by a particle passing through the focused spot. (3) From the cluster, the centroid vs time can be generated, which will provide tracking information. The centroid can be the center of the cluster of locations where events occur associated with a single particle. Although these examples are described with respect to identifying particles, the same algorithms can be applied to identifying a variety of features other than particles. Alternative approaches include optical flow algorithms, for example.

[0053] In other examples, the event-based imaging sensor can move relative to the scene at an unknown rate or at a known but random rate. In certain examples, random vibrations of the scene or the event-based imaging sensor can provide the movement. In certain examples, the random vibrational motion can be measured so that the motion of the focused spot relative to the scene can be known. An event clustering algorithm as described above can be used to identify particles or other features utilizing the random motion instead of controlled scanning motion.

[0054] In further examples, the methods can be combined with machine-learning algorithms for classification, inferencing, etc. This can be useful for autonomous cars and other applications where high-speed, high-resolution imaging is useful (for example, seeing a pedestrian from a long distance away).

[0055] The methods described herein can be performed using super-resolution imaging systems in some examples. In certain examples, a super-resolution imaging system can include an event-based imaging sensor having one or more pixels configured to receive light originating from a first portion of a scene within a focused spot. The event-based imaging sensor can be configured to move relative to the scene such that a different second portion of the scene moves into the focused spot. The system can also include a processor configured to detect a change in light intensity using the event-based imaging sensor. The change in light intensity can be caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot. The feature can have a dimension smaller than the focused spot. The processor can also estimate a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot, and generate a super-resolution image showing the feature.

[0056] It is noted that references to “a processor” can include a single processor or multiple processors that work together to perform the methods described herein. For example, some event-based imaging sensors can include their own integrated processors that can detect brightness changes of light received by pixels of the sensor. In some examples, an integrated processor can compare brightness changes detected by the sensor to a threshold value. If a brightness change exceeds the threshold value, the integrated processor can be configured to transmit event data from the sensor to a controller, personal computer, or other computing device. This other computing device can also include a processor that can be configured to process the event data from the sensor in order to estimate a characteristic of the feature being imaged. Thus, references to “a processor” or “the processor” can refer to multiple processors in some cases. In other examples, a single processor can be configured to perform all these functions.

[0057] FIG. 5 shows a schematic view of an example super-resolution imaging system 500. The system includes an event-based imaging sensor 510. The event-based imaging sensor receives light from a focused spot 520, as the light originates from the focused spot and then passes through a lens 530 before being received by the event-based imaging sensor. The focused spot encompasses a first portion 540 of a scene 542. In this example, the scene is on a moveable stage 550 that can move the scene so that the focused spot scans across the scene. The system also includes a processor 560 connected to the event-based imaging sensor. The processor is programmed to detect changes in light intensity using the event-based imaging sensor. This can be accomplished by the event-based imaging sensor sending event data to the processor when a pixel of the event-based imaging sensor detects a brightness change greater than a predetermined threshold. The brightness change can be a result of moving the scene so that a feature in the first portion of the scene moves out of the focused spot, or so that a feature from a second portion of the scene moves into the focused spot. The feature can have a dimension smaller than the focused spot. The processor can be programmed to estimate a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot. The characteristic can be a position, a dimension, an orientation, or another characteristic of the feature. The processor can also be programmed to generate a super-resolution image showing the feature. The super-resolution image can be displayed on an electronic display 570.

[0058] The systems and methods can also incorporate spectroscopy to further resolve spectra or color. This can be achieved with color-fdters combined with the event-based sensor or with standard methods for spectroscopy including push-broom, whisk-broom and snapshot methods. This may also be achieved by combining the event-based camera with a separate sensor producing spectral information. Thus, the sensors can capture hyperspectral resolution at super-resolution.

[0059] The movement of the event-based imaging sensor relative to the scene can be scanning in one direction, scanning in multiple directions, random vibrations, or a combination thereof. In some examples, the event-based imaging sensor can include a linear array of pixels. The moving can include scanning the event-based imaging sensor relative to the scene in a direction perpendicular to the linear pixel array. In certain examples, the system can include a moving stage configured to support the scene and move the scene, wherein the event -based imaging sensor is configured to remain stationary. In other examples, the eventbased imaging sensor can include a two-dimensional array of pixels. The system can also include a diffractive filter array positioned to disperse the light onto the two-dimensional array of pixels. The diffractive filter array can be configured to disperse the light in a spatially variant manner, a spectrally variant manner, or a combination thereof. The system can also include a slit configured to admit a linear portion of light from the scene, such that the first portion of the scene and the second portion of the scene are linear portions. The diffractive filter array can disperse the light spectrally in a direction perpendicular to the slit.

[0060] FIG. 6A shows one example super-resolution imaging system 600 in which one line of the sample scene is probed at a time. In this example, light emitted by a light source 680 is reflected by a parabolic mirror 682, then the light passes through a polarizer 684 and a lens 686 that focuses the light into a line beam. The light is represented schematically by dashed lines that converge to a point on the stage 650, but the line beam can actually have the form of a focused line 620. Thus, the line beam can illuminate a narrow strip of a sample on the stage. The light is reflected off the sample to a second lens 630, then through an analyzer 632 and a third lens 634. The third lens focuses the light to pass through a slit 636 and then through a diffraction grating 690. The diffraction grating is configured to disperse the line beam into a spectrum of various wavelengths present in the line beam. The wavelengths are dispersed along an axis perpendicular to the long axis of the line beam. The dispersed light is then recorded with an event-based imaging camera 610. FIG. 6B shows a two-dimensional grid that schematically represents the way the line beam is dispersed into a spectrum along the axis perpendicular to the length of the line beam. In this figure, the vertical axis represents the length of the line beam, and at each point along the length of the line beam the light is split into a spectrum that extends along the horizontal axis. In these figures, the line beam is represented as being subdivided into 8 portions, and the spectrum is represented as being split into 8 portions, to make an 8-by-8 grid in FIG. 6B. However, in reality there are not set subdivisions, and the resolution of the line beam and the data captured by the event-based imaging camera can be much greater. The data recorded by the camera in this system can be referred to as hyperspectral images, since a combination of visual image data and spectral image data are captured. Referring again to FIG. 6A, the stage can be moved (or the light source and camera assembly can be moved while the stage is stationary) to scan the line beam across a sample on the stage. In some cases, the line beam can scan across the sample in a direction perpendicular to the length of the line beam. Any changes in brightness or spectrum of the light reflected by the sample, due to particles or features of the sample that pass in or out of the line beam, can be detected by the event-based imaging camera. Thus, an event can be recorded at any location along the length of the line beam and the event can include data of changes in the intensity of various wavelengths of light reflected at each point along the line beam.

[0061] In some examples, the line beam can scan across a sample multiple times in multiple different directions. In a particular example, the line beam can scan across the sample once in the way described above, with the line beam moving relative to the stage in a direction perpendicular to the length of the line beam. Then, the stage can be turned (or the line beam and camera assembly can be turned) such as by turning the stage 90°. The line beam can then scan across the sample again so that the scanning direction is 90° off from the scanning direction used for the first scan. This can provide hyperspectral event-based image data in two perpendicular directions. As explained above, in some cases it may be useful to perform additional scans in more than two different directions. The system can scan the line beam across a sample in a roll-to-roll fashion in some examples. In other examples, the system can include the event-based image camera coupled to a vehicle such as a car or plane and the scanning movement can be provided by the movement of the vehicle relative to a scene being imaged. In certain examples, such a system can include an event-based image camera configured to capture image data from a line beam of light collected from obstacles or terrain near a vehicle. The line beam can be oriented perpendicular to the direction that the vehicle is moving so that the scene is scanned perpendicular to the line beam. In a particular example, the event-based imaging camera can be oriented downward from a plane, and configured to scan the ground beneath the plane while the plane flies forward. In another example, the event-based image camera can be installed on a satellite and oriented downward to scan the ground while the satellite moves over the ground. In yet another example, the event-based imaging camera can be incorporated into a telescope that scans across and area of the sky to collect astronomical data.

[0062] Spectral information can also be obtained computationally as illustrated in FIG. 7. This figure shows another example super-resolution imaging system 700, that includes a specially engineered diffractive-filter array 790 (DFA) that disperses incident light locally, which creates a point-spread function (PSF) for the system that is both spatially- and spectrally-variant. In this case, spatial super-resolution may be obtained in the similar manner as in FIGs. 2A-2C, but with a PSF that is not a simple focused spot, but more spatially structured. Light 702 from the sample or scene being imaged is diffracted by the DFA and then the light is captured by an event-based image sensor 710 according to the PSF. In some examples, the event-based image sensor can include a one-dimensional array of pixels or a two-dimensional array of pixels. The data received by the event-based image sensor can be processed using the linear inverse problem according to the PSF of the DFA. The data captured by the event-based image sensor can include spatial image data, spectral data, and temporal data from the times at which the events are generated. Thus, the data can be processed to assemble a spatio-spectral -temporal image. In some examples, the DFA can be in close proximity to the sensor, such as at a distance much less than 0.5 mm. However, in other examples, the DFA can be placed farther away, for instance in the Fourier plane (with a lens system in between the DFA and the sensor). The example system shown in this figure also includes a processor 760 in electronic communication with the event-based image sensor. The processor can process the data captured by the event-based image sensor to generate the spatio-spectral-temporal image. The processor is also in electronic communication with a memory device 762. The memory device can include stored instructions as software programming the processor to perform the processing. In some examples, the processor can also be in electronic communication with an electronic display (not shown) to display the generated image.

[0063] The systems and methods described herein can be combined with other superresolution schemes including photoactivated localization microscopy, stochastic optical reconstruction microscopy, stimulated emission depletion microscopy, etc. The systems and methods described herein can also include polarization resolved cameras to further obtain polarization information.

[0064] In further examples, the system can include a beam splitter and at least one additional sensor, wherein the beam splitter is positioned to split the light and direct the light to the event-based imaging sensor and the at least one additional sensor. The at least one additional sensor can be a polarization-resolved camera, a spectrum-resolved camera, or a combination thereof.

[0065] FIG. 8 shows an example super-resolution imaging system 800 that includes the event-based image sensor 810 and a polarization-resolved camera 812 and a spectrum- resolved camera 814. Beam splitters 820 are used to direct light 802 to all the cameras. Image processing algorithms as described above can then be used to obtain spatial super-resolution in addition to polarization and spectrum information.

[0066] The systems and methods described herein can have significant commercial potential with various applications across various industries. In the field of microscopy, the ability to achieve high-speed imaging with superior spatial resolution and signal -to-noise ratio opens up new possibilities for biological and medical research. Scientists and researchers can delve into intricate cellular structures and dynamic processes with unprecedented detail, leading to advancements in genetics, pharmacology, and pathology. The semiconductor industry can also benefit from this innovation. Semiconductor metrology, which involves precisely measuring and characterizing semiconductor materials and devices, is used to ensure high- quality production. By leveraging high temporal resolution and sub-diffraction-limited spatial resolution capabilities of the methods described herein, manufacturers can achieve more accurate and comprehensive semiconductor metrology, improving product performance and yield. Photography is another area that stands to gain from these systems and methods. The ability to capture high-resolution images with an enhanced signal-to-noise ratio can revolutionize the field, enabling photographers to capture intricate details, vibrant colors, and sharpness. From professional photographers seeking the utmost precision to everyday consumers looking for better image quality in their smartphones or digital cameras, this technology can cater to a wide range of photography applications. Furthermore, the versatility of this technology makes it suitable for integration into various imaging systems. Whether telescopes for astronomy, industrial inspection equipment, or surveillance systems, the ability to achieve sub-diffraction-limited spatial resolution and high-speed imaging can significantly enhance the capabilities of these systems. The technology's compatibility with different signal modalities, including fluorescence, scattering, reflection, and more, further expands its potential applications.

[0067] The devices described herein may also contain communication connections or networking apparatus and networking connections that allow the devices to communicate with other devices. Communication connections are an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules and other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. A “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection and wireless media such as acoustic, radio frequency, infrared and other wireless media. The term computer readable media as used herein includes communication media.

[0068] Reference was made to the examples illustrated in the drawings and specific language was used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is thereby intended. Alterations and further modifications of the features illustrated herein and additional applications of the examples as illustrated herein are to be considered within the scope of the description. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more examples. In the preceding description, numerous specific details were provided, such as examples of various configurations to provide a thorough understanding of examples of the described technology. It will be recognized, however, that the technology may be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the technology.

[0069] Although the subject matter has been described in language specific to structural features and / or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. Numerous modifications and alternative arrangements may be devised without departing from the spirit and scope of the described technology.

[0070] Examples

[0071] The present technology can include the following enumerated examples.

[0072] Example 1 : A super-resolution imaging method, comprising: using an event-based imaging sensor having one or more pixels, receiving light originating from a first portion of a scene within a focused spot; moving at least one of the event-based imaging sensor and the scene relative to one another such that a different second portion of the scene is in the focused spot; using the event-based imaging sensor, detecting a change in light intensity caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot, wherein the feature has a dimension smaller than the focused spot; estimating a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot; and generating a super-resolution image showing the feature.

[0073] Example 2: The method of any of Examples 1 through 17, wherein the characteristic is a position, a dimension, an orientation, or a combination thereof. Example 3: The method of any of Examples 1 through 17, wherein the feature of the first portion or the feature of the second portion is a particle.

[0074] Example 4: The method of any of Examples 1 through 17, wherein the feature of the first portion or the feature of the second portion has a dimension smaller than a diffraction limit.

[0075] Example 5: The method of any of Examples 1 through 17, wherein the super-resolution image has a resolution finer than a diffraction-limited resolution.

[0076] Example 6: The method of any of Examples 1 through 17, wherein the moving comprises moving the focused spot a distance that is smaller than the focused spot.

[0077] Example 7: The method of any of Examples 1 through 17, wherein the moving comprises moving the focused spot a distance that is smaller than a diffraction limit.

[0078] Example 8: The method of any of Examples 1 through 17, wherein the light originates from the scene through fluorescence, scattering, reflection, self-emission, phosphorescence, transmission, phase-contrast, or a combination thereof.

[0079] Example 9: The method of any of Examples 1 through 17, wherein the moving comprises scanning in one direction, scanning in multiple directions, random vibrations, or a combination thereof.

[0080] Example 10: The method of any of Examples 1 through 17, wherein the event-based imaging sensor comprises a linear array of pixels.

[0081] Example 11 : The method of any of Examples 1 through 17, wherein the moving comprises scanning the event-based imaging sensor relative to the scene in a direction perpendicular to the linear pixel array. Example 12: The method of any of Examples 1 through 17, wherein the event-based imaging sensor comprises a two-dimensional array of pixels.

[0082] Example 13: The method of any of Examples 1 through 17, further comprising dispersing the light onto the two-dimensional array of pixels using a diffractive filter array.

[0083] Example 14: The method of any of Examples 1 through 17, wherein the diffractive filter array disperses the light in a spatially variant manner, a spectrally variant manner, or a combination thereof.

[0084] Example 15: The method of any of Examples 1 through 17, wherein the first portion of the scene and the second portion of the scene are linear portions, and wherein the diffractive filter array disperses the light spectrally in a direction perpendicular to the linear portions.

[0085] Example 16: The method of any of Examples 1 through 17, further comprising splitting the light using a beam splitter, wherein the beam splitter directs the light to the event-based imaging sensor and at least one additional sensor.

[0086] Example 17 : The method of any of Examples 1 through 17, wherein the at least one additional sensor comprises a polarization-resolved camera, a spectrum-resolved camera, or a combination thereof.

[0087] Example 18: A super-resolution imaging system, comprising: an event-based imaging sensor having one or more pixels configured to receive light originating from a first portion of a scene within a focused spot, wherein the event-based imaging sensor is configured to move relative to the scene such that a different second portion of the scene moves into the focused spot; and a processor configured to: detect a change in light intensity using the event-based imaging sensor, wherein the change in light intensity is caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot, wherein the feature has a dimension smaller than the focused spot, estimate a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot, and generate a super-resolution image showing the feature.

[0088] Example 19: The system of any of Examples 18 through 35, the characteristic is a position, a dimension, an orientation, or a combination thereof.

[0089] Example 20: The system of any of Examples 18 through 35, wherein the feature of the first portion or the feature of the second portion is a particle.

[0090] Example 21 : The system of any of Examples 18 through 35, wherein the feature of the first portion or the feature of the second portion has a dimension smaller than a diffraction limit.

[0091] Example 22: The system of any of Examples 18 through 35, wherein the super-resolution image has a resolution finer than a diffraction-limited resolution.

[0092] Example 23: The system of any of Examples 18 through 35, wherein the moving comprises moving the focused spot a distance that is smaller than the focused spot.

[0093] Example 24: The system of any of Examples 18 through 35, wherein the moving comprises moving the focused spot a distance that is smaller than a diffraction limit.

[0094] Example 25: The system of any of Examples 18 through 35, wherein the light originates from the scene through fluorescence, scattering, reflection, self-emission, phosphorescence, transmission, phase-contrast, or a combination thereof.

[0095] Example 26: The system of any of Examples 18 through 35, wherein the moving comprises scanning in one direction, scanning in multiple directions, random vibrations, or a combination thereof. Example 27: The system of any of Examples 18 through 35, wherein the event-based imaging sensor comprises a linear array of pixels.

[0096] Example 28: The system of any of Examples 18 through 35, wherein the moving comprises scanning the event-based imaging sensor relative to the scene in a direction perpendicular to the linear pixel array.

[0097] Example 29: The system of any of Examples 18 through 35, further comprising a moving stage configured to support the scene and move the scene, wherein the event-based imaging sensor is configured to remain stationary.

[0098] Example 30: The system of any of Examples 18 through 35, wherein the event-based imaging sensor comprises a two-dimensional array of pixels.

[0099] Example 31 : The system of any of Examples 18 through 35, further comprising a diffractive filter array positioned to disperse the light onto the two-dimensional array of pixels.

[0100] Example 32: The system of any of Examples 18 through 35, wherein the diffractive filter array is configured to disperse the light in a spatially variant manner, a spectrally variant manner, or a combination thereof.

[0101] Example 33: The system of any of Examples 18 through 35, further comprising a slit configured to admit a linear portion of light from the scene, such that the first portion of the scene and the second portion of the scene are linear portions, and wherein the diffractive filter array disperses the light spectrally in a direction perpendicular to the slit.

[0102] Example 34: The system of any of Examples 18 through 35, further comprising a beam splitter and at least one additional sensor, wherein the beam splitter is positioned to split the light and direct the light to the event-based imaging sensor and the at least one additional sensor. Example 35: The system of any of Examples 18 through 35, wherein the at least one additional sensor comprises a polarization-resolved camera, a spectrum-resolved camera, or a combination thereof.

Claims

CLAIMSWhat is claimed is:

1. A super-resolution imaging method, comprising: using an event-based imaging sensor having one or more pixels, receiving light originating from a first portion of a scene within a focused spot; moving at least one of the event-based imaging sensor and the scene relative to one another such that a different second portion of the scene is in the focused spot; using the event-based imaging sensor, detecting a change in light intensity caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot, wherein the feature has a dimension smaller than the focused spot; estimating a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot; and generating a super-resolution image showing the feature.

2. The method of claim 1, wherein the characteristic is a position, a dimension, an orientation, or a combination thereof.

3. The method of claim 1, wherein the feature of the first portion or the feature of the second portion is a particle.

4. The method of claim 1, wherein the feature of the first portion or the feature of the second portion has a dimension smaller than a diffraction limit.

5. The method of claim 1, wherein the super-resolution image has a resolution finer than a diffraction-limited resolution.

6. The method of claim 1, wherein the moving comprises moving the focused spot a distance that is smaller than the focused spot.

7. The method of claim 6, wherein the moving comprises moving the focused spot a distance that is smaller than a diffraction limit.

8. The method of claim 1, wherein the light originates from the scene through fluorescence, scattering, reflection, self-emission, phosphorescence, transmission, phase-contrast, or a combination thereof.

9. The method of claim 1, wherein the moving comprises scanning in one direction, scanning in multiple directions, random vibrations, or a combination thereof.

10. The method of claim 1, wherein the event-based imaging sensor comprises a linear array of pixels.

11. The method of claim 10, wherein the moving comprises scanning the event-based imaging sensor relative to the scene in a direction perpendicular to the linear pixel array.

12. The method of claim 1, wherein the event-based imaging sensor comprises a two- dimensional array of pixels.

13. The method of claim 12, further comprising dispersing the light onto the two-dimensional array of pixels using a diffractive filter array.

14. The method of claim 13, wherein the diffractive filter array disperses the light in a spatially variant manner, a spectrally variant manner, or a combination thereof.

15. The method of claim 14, wherein the first portion of the scene and the second portion of the scene are linear portions, and wherein the diffractive filter array disperses the light spectrally in a direction perpendicular to the linear portions.

16. The method of claim 1, further comprising splitting the light using a beam splitter, wherein the beam splitter directs the light to the event-based imaging sensor and at least one additional sensor.

17. The method of claim 16, wherein the at least one additional sensor comprises a polarization-resolved camera, a spectrum-resolved camera, or a combination thereof.

18. A super-resolution imaging system, comprising: an event-based imaging sensor having one or more pixels configured to receive light originating from a first portion of a scene within a focused spot, wherein the event-based imaging sensor is configured to move relative to the scene such that a different second portion of the scene moves into the focused spot; and a processor configured to: detect a change in light intensity using the event-based imaging sensor, wherein the change in light intensity is caused by a feature of the first portion moving out of the focused spot or by a feature of the second portion moving into the focused spot, wherein the feature has a dimension smaller than the focused spot, estimate a characteristic of the feature based on the change in light intensity and motion of the scene relative to the focused spot, and generate a super-resolution image showing the feature.

19. The system of claim 18, the characteristic is a position, a dimension, an orientation, or a combination thereof.

20. The system of claim 18, wherein the feature of the first portion or the feature of the second portion is a particle.

21. The system of claim 18, wherein the feature of the first portion or the feature of the second portion has a dimension smaller than a diffraction limit.The system of claim 18, wherein the super-resolution image has a resolution finer than a diffraction-limited resolution.

23. The system of claim 18, wherein the moving comprises moving the focused spot a distance that is smaller than the focused spot.

24. The system of claim 23, wherein the moving comprises moving the focused spot a distance that is smaller than a diffraction limit.

25. The system of claim 18, wherein the light originates from the scene through fluorescence, scattering, reflection, self-emission, phosphorescence, transmission, phase-contrast, or a combination thereof.

26. The system of claim 18, wherein the moving comprises scanning in one direction, scanning in multiple directions, random vibrations, or a combination thereof.

27. The system of claim 18, wherein the event-based imaging sensor comprises a linear array of pixels.

28. The system of claim 27, wherein the moving comprises scanning the event -based imaging sensor relative to the scene in a direction perpendicular to the linear pixel array.

29. The system of claim 28, further comprising a moving stage configured to support the scene and move the scene, wherein the event-based imaging sensor is configured to remain stationary.

30. The system of claim 18, wherein the event-based imaging sensor comprises a two- dimensional array of pixels.

31. The system of claim 30, further comprising a diffractive filter array positioned to disperse the light onto the two-dimensional array of pixels.

32. The system of claim 31, wherein the diffractive filter array is configured to disperse the light in a spatially variant manner, a spectrally variant manner, or a combination thereof.

33. The system of claim 32, further comprising a slit configured to admit a linear portion of light from the scene, such that the first portion of the scene and the second portion of the scene are linear portions, and wherein the diffractive filter array disperses the light spectrally in a direction perpendicular to the slit.

34. The system of claim 18, further comprising a beam splitter and at least one additional sensor, wherein the beam splitter is positioned to split the light and direct the light to the event-based imaging sensor and the at least one additional sensor.

35. The system of claim 34, wherein the at least one additional sensor comprises a polarization-resolved camera, a spectrum-resolved camera, or a combination thereof.

Images