Temporal super-resolution

Abstract

A system and method for imaging a target, the method comprising: transmitting N electromagnetic wave pulses to illuminate the target, N being a plurality; receiving at an imager sensitive to the electromagnetic waves electromagnetic wave pulses reflected by the target from each of the transmitted pulses; integrating energy in a plurality of the received pulses during a same exposure period of the imager to provide a measure of the integrated energy; and processing the measure of the integrated energy to provide N images of the target.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 219,378, filed on July 8, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The subject matter disclosed herein relates generally to signal processing, and more particularly to temporal resolution of signals. Background Art

[0004] The resolution of a digital signal is generally related to its frequency content. A high-resolution (HR) signal is band-limited to a wider frequency range than a low-resolution (LR) signal. Resolution is generally limited by two factors: physical device limitations and sampling rate. For example, digital image resolution is generally limited by the optics of the imaging device (i.e., the diffraction limit) and the pixel density of the sensor (i.e., the sampling rate).

[0005] A technique used to increase resolution is temporal super-resolution (TSR). This technique is based on increasing the temporal sampling frequency beyond the Nyquist frequency (which is limited by the sampling rate). Different approaches can be used when applying TSR (also commonly referred to as "upsampling"), particularly for image signals. Some methods rely on hardware to increase the temporal frequency detection, others on software, others on deep learning models, and still others on some combination of the aforementioned. Summary of the Invention

[0006] In various embodiments, a method for imaging a target is provided, the method comprising: transmitting N (a plurality of) pulses of electromagnetic (EM) waves to illuminate the target; receiving, at an imager sensitive to EM waves, an EM wave pulse reflected by the target from each transmitted pulse; integrating energy in the plurality of received pulses during a same exposure period of the imager to provide a measure of integrated energy; and processing the measure of integrated energy to provide N images of the target.

[0007] In various embodiments, an imaging system operable to image a target is provided, the imaging system comprising: an EM wave source controllable to emit a plurality of EM waves to illuminate the target; a sensor sensitive to the EM waves and controllable to have an exposure period during which the sensor is enabled to receive and integrate energy in the EM waves reflected by the target from the emitted EM waves; and a controller configured to control the EM wave source and the sensor to process measurements of the integrated energy to provide N images of the target.

[0008] In some embodiments, the sensor integrates the energy of each of the different M characterizing features independently of integrating the energies of other distinguishing features to provide a measure of the integrated energy of each of the M characterizing features.

[0009] In some embodiments, the controller processes the measure of the integrated energy for each of the M features to provide N images of the target for each of the M features, for a total of N x M images of the target.

[0010] In some embodiments, processing the integrated energy by the controller to provide the N images includes minimizing a cost function.

[0011] In some embodiments, the transmitted EM energy pulse comprises an EM wave characterized by M different distinctive features.

[0012] In some embodiments, the M different distinguishing features comprise different wavelength bands of EM energy.

[0013] In some embodiments, the M different distinguishing features include different polarization directions.

[0014] In some embodiments, integrating the energy includes integrating the energy of each of the different M characterizing features independently of integrating the energies of other distinguishing features to provide a measure of the integrated energy of each of the M characterizing features.

[0015] In some embodiments, processing the integrated energy includes processing a measure of the integrated energy for each of the M features to provide N images of the object for each of the M features, totaling to provide N×M images of the object.

[0016] In some embodiments, processing the integrated energy to provide the N images comprises minimizing a cost function.

[0017] In some embodiments, the cost function comprises a time cost function.

[0018] In some embodiments, the cost function comprises a spatiotemporal cost function.

[0019] In some embodiments, the cost function comprises a Lagrangian cost function.

[0020] In some embodiments, the EM waves include visible light waves.

[0021] In some embodiments, the EM waves include infrared (IR) waves.

[0022] In some embodiments, the EM waves include ultraviolet (UV) waves. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Non-limiting examples of the embodiments disclosed herein are described below with reference to the accompanying drawings, which are listed after this paragraph. The drawings and description are intended to illustrate and clarify the embodiments disclosed herein and should not be considered limiting in any way. Identical elements in different figures may be represented by the same numerals. Elements in the drawings are not necessarily drawn to scale. In the drawings:

[0024] Figure 1 schematically illustrates an exemplary imaging system including an imager, an illuminator, and a controller that controls the illuminator and the imager and processes measures of integrated energy from the imager to provide N images of a target;

[0025] Figure 2A yes Figure 1 A flowchart of an exemplary method of operating an imaging system;

[0026] Figure 2B schematically shows an exemplary graph illustrating a temporal relationship between an exposure period of an imager to reflected light and an exposure period to emission pulses from an illuminator;

[0027] Figure 3 is a flow chart of a method for processing an image by a controller to generate an upsampled image of a target;

[0028] Figure 4A Two graphs are illustrated showing a time comparison and a spectral comparison between the three signals TS, x1 and x3 for N=3, respectively;

[0029] Figure 4B Two graphs are illustrated showing a time comparison and a spectral comparison between the three signals TS, x1 and x3 for N=4, respectively;

[0030] Figure 4C Two graphs are shown, which respectively show a time comparison and a spectrum comparison between the three signals TS, x1 and x3 for N=5;

[0031] Figure 4D Two graphs are shown, which respectively show a time comparison and a spectrum comparison between the three signals TS, x1 and x3 for N=6;

[0032] Figure 5 illustrates examples of images of a rotating fan obtained for N=3, 4, 5 and 6 for determining motion estimation;

[0033] Figure 6illustrates a graph showing measurements of the SNR of a combined RGB signal without and with pulsed light emission, as well as graphs showing each color individually;

[0034] Figure 7 illustrates a graph showing a measure of cosine similarity between a true signal and a reconstructed signal for different values ​​of α; and

[0035] Figure 8 Illustrated is a graph comparing the temporal error in motion estimation between an original video and an upsampled video reconstructed from the original video. DETAILED DESCRIPTION

[0036] Applicants have recognized that TSR supported by hardware (e.g., optics or sensors) has the potential to increase temporal sampling frequency to much higher rates and reliability than other approaches. Applicants have further recognized that one disadvantage is the complexity of known systems and the price associated with such systems.

[0037] As a result, the applicant has developed a new method for TSR that allows the use of a low-complexity imaging system and provides a high temporal sampling frequency with high reliability of spectral reconstruction. The method uses the optical reflectance properties of a "target" (an object or entity, or an area therein), such as its surface polar reflectance and / or its spectral reflectance (the color of the target). An imaging system (hereinafter also referred to as the "system") combines an imager, a high-frequency illumination source (illuminator) that emits electromagnetic pulses, and a controller that processes the optically encoded signals (electromagnetic waves reflected from the emitted electromagnetic pulses) received by the imager at a fixed sampling rate. Optionally, the imaging system includes a neural network.

[0038] One aspect of an embodiment of the present invention relates to a TSR method for upsampling the sampling rate of an imaging system, which optionally enhances the system's sensitivity to high-frequency features of an object whose image is captured by an imager. The method comprises operating the imager to perform an exposure cycle having a duration T and an exposure period repetition frequency f substantially equal to 1 / T. eAn image of a target is acquired during each exposure period in a sequence of exposure periods while illuminating the target with a temporally periodic illumination pattern of EM waves (emitted EM pulses). T can be in the range of 1 ms to 1 second, but can optionally be greater than 1 second, such as 1.3 seconds, 1.5 seconds, 1.8 seconds, 2 seconds, or even longer. The illumination pattern can have a time period equal to approximately T / N, where N is an integer greater than 1 and includes EM waves characterized by M different distinctive features that are processed by the system in M ​​different corresponding imaging channels. N can be in the range of 2 to 10, but can optionally be greater than 10, such as 12, 15, 20, 30, 45, 60, or even greater. M can be in the range of 1 to 10, but can optionally be greater than 10, such as 12, 15, 20, 30, 45, 60, or even greater.

[0039] An EM wave characterized by a characteristic feature m (1≤m≤M) may be referred to as an EM wave in channel m or imaging channel m. As examples, the different distinguishing features may be different wavelength bands or polarization directions. The different wavelength bands may be different wavelength bands of visible light, infrared (IR) light, or ultraviolet (UV) light. For an illumination pattern with an illumination cycle frequency f equal to approximately N / T, l During each exposure cycle that illuminates the target, the imager acquires one image of the target for each imaging channel, resulting in a total of M images of the target. Each of the M images acquired of the target during a single exposure cycle is generated by integrating the energy in the EM wave reflected by the target and collected by the system in the corresponding Mth imaging channel over all N cycles of the illumination pattern that illuminated the target during the exposure cycle.

[0040] In some embodiments, the data in the M images are processed to generate an image of the target for each of the N illumination cycles that occur during the exposure period. The result is a total of N×M images of the target. The generated images are generated at a frequency equal to the illumination cycle frequency f l = N / T, the corresponding sampling frequency of the EM waves reflected by the target and the effective image acquisition rate provide a sequence of N images of the target, where the illumination cycle frequency is N times the exposure cycle frequency. At the sampling frequency N / T, the image sequence encodes the temporal characteristics of the target up to an upper bound frequency, which is approximately equal to the Nyquist frequency f l / 2=N / 2T, which is the same as the exposure cycle repetition frequency f e The associated upper bound Nyquist frequency f e / 2=N times of 1 / 2T (N is the upsampling factor).

[0041] In some embodiments, for N=M, the data from the M images is sufficient to accurately determine the N images, and according to embodiments, the data from the M images can be processed to determine N "precise" images of the target. For N>M, the N images are underdetermined by the data in the M images, and the data from the M images are processed to satisfy the constraints based on the cost function to determine an approximation of the N images.

[0042] Figure 1 Schematically illustrated is an exemplary imaging system 100 including an imager 102, an illuminator 104, and a controller 106. Also shown is a target 112 imaged by the imaging system 100, which applies TSR, as described in the following method, to enhance high-frequency features in the target. Note that while the following description refers to the use of visible light as the pulsed EM wave, other types of EM waves may also be used, including, for example, IR and UV.

[0043] Imager 102 may include an RGB camera or other imaging device adapted to receive EM waves 116 (e.g., light) reflected from target 112 and acquire an image of the target while the target is temporally illuminated by pulsed EM waves 114 (e.g., pulsed light) from illuminator 104. For convenience, hereinafter, light received by the imager (i.e., light 116) may also be referred to as "received light" or "reflected light," and light emitted by the illuminator (i.e., light 114) may be referred to as "emitted light," "pulsed light," or "emitted pulsed light." Imager 102 may acquire images of target 112 during a sequence of exposure cycles having a duration T and an exposure cycle repetition frequency fe substantially equal to 1 / T. Imager 102 may additionally acquire a total of M images associated with M different distinctive features in pulsed light 114 that originate from illuminator 104 and are reflected back as light 116, and the M different distinctive features may be associated with polarization and / or color of the light (optionally RGB light). Illuminator 104 can emit pulsed light 114 with a time period equal to T / N. For exemplary purposes, pulsed light 114 can be RGB light, where M = 3, N = 6, and T = 33 ms. For polarized light, it can be M = 2, N = 5, and T = 10 ms. Note that, as previously mentioned, pulsed light 114 can be IR or UV light. Exemplary parameters for IR or UV light can be M = 1, N = 3, and T = 20 ms.

[0044] The controller 106 includes a processor 108 and a memory 110. Optionally, the controller 106 includes a neural network 111. The processor 108 controls the illuminator 104 to emit and illuminate the target with pulsed light 114 for each of M different characteristics of light according to a time period T / N. The processor 108 additionally controls the imager 102 to emit and illuminate the target with pulsed light 114 having a duration T and an exposure period repetition frequency f equal to approximately 1 / T. e 4 , the processor 108 processes the received imaging information to enhance high-frequency features in the received imaging information associated with the target 112. The processor 108 may additionally control all other functionality associated with the operation of the imaging system 100. It should be noted that the processor 108, while shown as a single unit within the controller 106, may include more than one processor within the controller and / or one or more processors external to the controller.

[0045] Memory 110 may store all executable instructions required for the operation of processor 108. These may include instructions associated with the execution of the TSR algorithm. Memory 108 may additionally store imaging information associated with M channels generated by imager 102 from reflected light 116 for each of the M distinguishing features in pulsed light 114, as well as the combined image after TSR is applied. Note that memory 110, while shown as a single unit within controller 106, may include more than one memory unit within the controller and / or one or more memory units external to the controller and / or one or more memory units within processor 108.

[0046] A neural network (NN) 111 (optionally included in the controller 106) may optionally be an unsupervised NN. An exemplary NN 111 architecture may be based on a Unet and may include a first stage that may function as an encoder and a second stage that may function as a decoder. In the encoding stage, the NN 111 may use downsampling, optionally nonlinear downsampling, such as downsampled max pooling, to extract the maximum value associated with each of the M characterizing features in the M imaging channels for all N periods. In the decoding stage, upsampling may be applied to transfer the mapping produced from the first stage to a larger pixel space. Optionally, nonlinear filtering using ReLU activation filters may be applied.

[0047] Figure 2A is a flow chart 200 of an exemplary method of operation of the imaging system 100. For exemplary purposes, Figure 2B Describing the flow chart 200, Figure 2BAn exemplary graph 210 having four timelines 212 , 214 , 216 and 218 illustrating a temporal relationship between an exposure period 220 of the imager 102 to the reflected light 116 and an exposure period 220 of the pulses 114 emitted from the illuminator 104 is schematically shown.

[0048] At block 202, the illuminator 104 emits a pulse train LP of emission light 114 m , the pulse train LP m Including pulse At the lighting frequency f l For exemplary purposes, the pulsed light 114 has M=3 EM energy pulse trains LPm (1≤m≤3) for each exposure period 220 of the imager 102, which includes an optional N=5 EM energy pulses. (1≤n≤N=5). Therefore, the pulse train LP m is configured to have a value equal to approximately Nf e The lighting cycle frequency f l . Pulse Train LP m Optionally pulses of R, G and B light, respectively Visible light pulse train. Pulse train LP 1 LP 2 LP 3 This is schematically shown along timelines 212 , 214 , and 216 , respectively.

[0049] At block 204, imager 102 receives N pulses of reflected light 116 associated with each pulse train LPm from target 112. During exposure periods 220, imager 102 receives and records reflected light 116. Exposure periods 220 are shown along timeline 218. During each exposure period 220, imager 102 collects pulse trains LPm from pulsed light 114, respectively, at pixels of a light sensor (not shown) included in the imager. 1 LP 2 LP 3 N light pulses in (1≤n≤N=5) of reflected light 116 from target 112 and images it.

[0050] At block 206, different corresponding imaging channels C of the pixels are m(1≤m≤3), during the exposure period 220, each pixel integrates the energy from the reflected light pulses imaged on the pixel in each pulse train to record the light. Typically, the imaging channel of a pixel for recording R, G, or B light includes a photosensitive area covered by an R, G, or B filter, respectively, and an electronic device for integrating the energy in the incident light passing through the filter and converting it into an electronic signal. Let C 1 、C 2 and C 3 The pixel responds to the emitted light pulse The pulse of reflected light 116 generates an electronic signal through the region 102 of the target imaged on the pixel during the exposure period 210. Signal C 1 、C 2 and C 3 can be thought of and optionally referred to as an image of the area imaged on the pixel. Figure 2B In the example, we use image C 1 、C 2 and C 3 To mark the exposure period 220, image C 1 、C 2 and C 3 It can be generated from the light in N light pulses collected and integrated by the pixel during the exposure period.

[0051] Image C 1 、C 2 and C 3 can be equal to f e The sampling frequency of the image is acquired, and the image pairs from the region on the target 112 are limited by a frequency equal to approximately the Nyquist frequency f l The image can be encoded with data characterized by time-frequency in a bandwidth with a cutoff frequency of 1 / 2. Thus, the image can be blind to high frequency features, such as transient features exhibited for very short periods of time (not shown).

[0052] At block 208, the controller 106 processes the image C to increase the temporal cutoff frequency of the image acquired by the imager 102. 1 、C 2 and C 3 For each pulse (which illuminates the target during each exposure period 220) generates an image of the target 112, and provides N images of the target for each exposure period. At N images per exposure period, the imager 102 illuminates the target with an image size equal to approximately Nf l / 2 effective time cutoff frequency operation.

[0053] The processing of the image C by the controller 106 is described with reference to the flowchart 300. 1 、C 2and C 3 Each pulse A method for generating an upsampled image of target 112 is also described. 1 、C 2 and C 3 Integration method. It should be noted that for C m The method is generally described with respect to imaging channels (ie, images).

[0054] At block 302, to determine C m , it can be assumed that the imager 102 is respectively m Each of the M channels, defined by sensitivity to light in different wavelength bands represented by (1≤m≤M), generates an image of the target 102 in response to the reflected light 116. Linear optics can also be assumed so that the reflected light does not undergo any changes as it passes through the channels. Let C m (T, t) represents the image generated by a pixel in the imager 102 during a particular exposure period having a duration T starting at a given time t. Let Q m (λ) represents the sensitivity of a pixel in the imager 102 as a function of wavelength λ to the intensity of incident light in the wavelength band λm, and let c m (λ,t) represents the wavelength band λ m The intensity of light in the illumination pattern emitted by illuminator 104 at time t as a function of the wavelength in λ. If R(λ,t) represents the reflectivity of an area in target 112 at time t as a function of wavelength λ, then a pixel generates an image C in response to incident light reflected by the area of ​​target 112 imaged at that pixel. m (T, t), the image can be expressed as follows:

[0055]

[0056] Assumption c m (λ,t) is separable and can be written as c m (λ,t)=c m (t)c m (λ), then Q m (λ) can be redefined to include c m The wavelength dependence of (λ,t) c m (λ), and equation (1) can be written as:

[0057]

[0058] where Q′ m =Q m (λ)c m (λ).

[0059] At block 304, the controller 106 may determine the discrete condition represented by the pulsed light 114 varying between two modes (off and on) over time according to equation (2). m A further assumption can be made, namely

[0060]

[0061] where γ m,k is a constant, and the pulsed light 114 emitted by the illuminator 104 comprises a pulse train having N substantially discrete light pulses during the exposure period T, so that equation (2) can be rewritten as

[0062]

[0063] in According to equation (4), it can be understood that C m has been determined, where N is the upsampling factor, and i n represents the average value of the image at sub-time step n.

[0064] At block 306, the controller 106 optionally applies a cost function. Note that in equation (4), extracting i n The value of is equivalent to upsampling in the time domain by a factor of N. This may cause problems for M channels, and the equation can only be solved for an upsampling factor of N = M. In practice, where the number of channels is relatively low and a high rate TSR is desired, a cost function can optionally be introduced.

[0065] The controller 106 may direct the nth light pulse in the mth channel of the imager 102 to the region of the target 112 imaged at a given pixel. The image is defined as (1≤n≤N, 1≤m≤M). The controller 106 may then be operable to determine the face region i for a given exposure period at time t and channel m by optionally selecting the temporal scene smoothness as the cost function. n And thus determine N images The cost function is optionally a Lagrangian, which can be given by

[0066]

[0067] in is the Lagrange multiplier. In matrix representation, the solution of equation (5) can be written as

[0068]

[0069] where the vector and And the matrices S and M are defined as follows:

[0070]

[0071]

[0072] in is the intensity vector of size N for each exposure time, is of size M and is the captured value in each channel for a single exposure time when channel m is pulsed on or off. has a binary value of 0 or 1, and M represents the Lagrange multiplier for each channel.

[0073] In the above description, the controller 106 determines i in response to the Lagrangian cost function defined by equation (5). n and thus determine the value of However, the cost function to be applied may not be limited to equation (5), which may be considered as a temporal cost function that provides, for each channel M, a temporal sequence C of images provided only by a given pixel, for a given pixel. m For example, an alternative cost function may provide, for a given pixel, N images that are functions of the images provided by pixels in a pixel neighborhood "P" of the given pixel.

[0074] Let the image provided by a given pixel at pixel coordinates x,y during an exposure period T starting at a given time t be represented as The controller 106 may process to determine the Image The Lagrangian cost function of can be a spatiotemporal cost function that is responsive not only to the temporal sequence of images provided by a given pixel, but also to the images provided by pixels in a pixel neighborhood of the given pixel. Optionally, the pixel neighborhood can be a 4-neighborhood. As an example, the optional spatiotemporal Lagrangian 4-neighborhood cost function can be given by the following expression:

[0075]

[0076] in, and is the weight.

[0077] The applicant has performed a number of tests to evaluate the efficacy of the disclosed method for TSR, which allows the use of a low complexity imaging system and allows a high temporal sampling frequency with high reliability of the spectral reconstruction. A description of the tests and the results obtained are given below.

[0078] A. Test Setup

[0079] The test setup consisted of a commercial CMOS camera with adjustable speed as the imager, a smartphone as the illuminator set at a refresh rate of 60 Hz, and a rotating household fan with blades covered with white paper as the target. The camera was set at different frame rates: 10 Hz, 20 Hz, and 80 Hz. The fan rotated at approximately 21.5 Hz. The same coding pattern was used for each N. The temporal illumination was RGB light with the following characteristics:

[0080] N=3:b → =(1,0,0),g → =(0,1,0),r → =(0,0,1)

[0081] N=4:b → =(1,0,0,1),g → =(1,0,1,0),r → =(0,1,0,1)

[0082] N=5:b → =(0,1,0,0,0),g → =(1,0,1,0,1),r → =(0,0,0,1,0)

[0083] N=6:b → =(1,0,1,0,1,0),g → =(0,1,0,1,0,1),r → =(1,1,1,1,1,1)

[0084] To avoid noise artifacts, white noise filtering was applied to all measured signals during testing.

[0085] Illumination correction is introduced as a comparison between the actual signal (which was captured in a high fps recording) and the same signal captured at a lower fps (and upsampled). A compensating gain is applied to the high fps signal to overcome illumination differences due to different exposure times. An additional correction is applied due to the object color (gamma factor), which represents the reflection of R, G, and B. To determine the gamma factor to balance the intensities of all colors, a reference measurement of a white target (center of the fan) is used to calibrate the intensity values ​​relative to it.

[0086] B. Test results

[0087] The experimental results are shown in Figures 4A-4DTS is the true signal emitted by the illuminator; x1 is the signal seen by the imager before upsampling; and x3 is the upsampled signal. The camera fps is 10Hz and the Nyquist frequency is 5Hz. Figures 4A-4D Description:

[0088] Figure 4A Illustrated are graphs 400 - 1 and 400 - 2 showing a temporal comparison and a spectral comparison, respectively, between three signals TS, x1 and x3 for N=3.

[0089] Figure 4B Two graphs 402 - 1 and 402 - 2 are illustrated, showing a temporal comparison and a spectral comparison, respectively, between the three signals TS, x1 and x3 for N=4.

[0090] Figure 4C Two graphs 404 - 1 and 404 - 2 are shown, which respectively illustrate a time comparison and a spectral comparison between the three signals TS, x1 and x3 for N=5.

[0091] Figure 4D Two graphs 406 - 1 and 406 - 2 are shown, which respectively illustrate a time comparison and a spectral comparison between the three signals TS, x1 and x3 for N=6.

[0092] exist Figures 4A-4D In , the signal axis is unitless and is used to provide a measure of comparison. From the results, it can be seen that the spectral components are successfully detected up to a frequency of 30 Hz.

[0093] C. Imaging results

[0094] For N=3, 4, 5, and 6, Figure 5 The imaging results are shown in FIG, as shown in rows 500, 502, 504, and 506. The first frame in each row 500-506, indicated by "TSR one frame," is the image seen by the imager before upsampling. The subsequent frames in each row are upsampled images generated sequentially based on N (N=3, 3 frames; N=4, 4 frames; N=5, 5 frames; N=6, 6 frames).

[0095] D. SNR and performance results

[0096] To evaluate the SNR for different α factors, a clean white paper was used, placed 40 cm in front of the camera and illuminator. Different ambient lighting using a white light projector was also used. The illumination values ​​were measured using a Lux-meter. The results are shown in Figure 6 and described as follows:

[0097] (a) Graph 600A illustrates a measure of system SNR versus α (light intensity) for combined RGB light when the illumination light is emitted without pulse modulation (as indicated by 602A) and when it is emitted with pulse modulation (as indicated by 602B).

[0098] (b) Graph 600B illustrates the system SNR versus α for each light color when each light color is emitted without and with pulse modulation. The SNR for blue light is shown by 604A and 604B for continuous blue light and pulsed blue light, respectively. The SNR for red light is shown by 606A and 606B for continuous red light and pulsed red light, respectively. The SNR for green light is shown by 608A and 608B for continuous green light and pulsed green light, respectively.

[0099] Understandably, the SNR improves with the use of an illuminator, as it increases the amount of light in the scene.

[0100] E. Signal reconstruction performance results

[0101] An additional experiment was to measure the signal reconstruction performance (angle error) versus the α factor. The cosine similarity of x1 and x3 was determined for different values ​​of α. For N=3, the results are shown in Figure 7 When decreasing the α factor, it is noticed that the performance of the disclosed TSR method decreases, which is related to the increase in the illumination of the environment relative to the illuminator.

[0102] F. Motion estimation improvements

[0103] A fundamental task in computer vision is motion estimation or optical flow estimation. Given the spatial and temporal derivatives of an image, the velocity of a pixel in the xy plane can be calculated. The estimation of the temporal derivative depends heavily on the camera frames per second rate. Since high temporal frequencies cannot be detected in a camera with low frames per second, the use of the disclosed TSR method and increasing the camera frames per second can improve the time. The blade velocity of a rotating fan (in the xy plane) is measured at each pixel and compared to the ground truth detected using a high frames per second camera. The results are shown in Figure 8 800 , which shows the angular error over time for a captured video of a rotating fan blade before upsampling (as shown at 802 ) and after upsampling (as shown at 804 ).

[0104] It can be appreciated that by using the disclosed upsampling method, there is a significant improvement in error.

[0105] Some stages (steps) of the above-described method may also be implemented in a computer program for running on a computer system, comprising at least a code portion for executing the steps of the relevant method when running on a programmable device such as a computer system, or for enabling the programmable device to perform the functions of the apparatus or system according to the present disclosure. Such a method may also be implemented in a computer program for running on a computer system, comprising at least a code portion for causing a computer to execute the steps of the method according to the present disclosure.

[0106] A computer program is a list of instructions for a specific application and / or operating system. A computer program may include, for example, one or more of the following: a subroutine, a function, a procedure, a method, an implementation, an executable application, an applet, a servlet, source code, code, a shared library / dynamically loaded library, and / or other sequence of instructions designed to be executed on a computer system.

[0107] The computer program may be stored internally on a non-transitory computer-readable medium. All or some of the computer program may be provided on a computer-readable medium that is permanently, removably, or remotely connected to an information processing system. The computer-readable medium may include, for example, but not limited to, any number of the following: magnetic storage media including magnetic disk and tape storage media; optical storage media such as optical disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; non-volatile memory storage media including semiconductor-based memory cells such as flash memory, EEPROM, EPROM, ROM; ferromagnetic digital memory; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.

[0108] A computer process typically consists of an executing (running) program or portion of a program, current program values ​​and state information, and the resources used by the operating system to manage the execution of the process. An operating system (OS) is software that manages the sharing of computer resources and provides programmers with an interface for accessing those resources. The OS processes system data and user input and responds to the system's users and programs by allocating and managing tasks and internal system resources as services.

[0109] A computer system may include, for example, at least one processing unit, associated memory, and a plurality of input / output (I / O) devices. When executing a computer program, the computer system processes information according to the computer program and generates result output information through the I / O devices.

[0110] Unless otherwise stated, use of the expression "and / or" between the last two members of a list of options for selection indicates that selection of one or more of the listed options is applicable and can be made.

[0111] It should be understood that when the claims or specification refer to an "a" or "an" element, such reference is not to be construed as being directed to only a single element but is meant to be a reference to one or more elements.

[0112] All references mentioned in this specification are hereby incorporated by reference in their entirety as if each individual reference was specifically and individually indicated to be incorporated by reference herein. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present disclosure.

[0113] While the present disclosure has been described in some embodiments and generally related aspects, changes and modifications will be obvious to those skilled in the art. The present disclosure is not to be limited to the specific embodiments described herein, but only by the scope of the appended claims.

Claims

1. A method for imaging a target, comprising: illuminating the target with a temporally periodic illumination pattern of electromagnetic (EM) waves, the illumination pattern having a temporal period of T / N, where N is the number of illumination periods in an exposure period T and N>1, and wherein the EM waves are characterized by M>1 substantially simultaneously emitted distinct, distinctive features selected from wavelength bands or polarization directions; operating an imager sensitive to the EM wave to acquire M images in M ​​imaging channels during each exposure cycle time period T, wherein each imaging channel corresponds to a respective distinguishing feature, wherein acquiring the M images comprises integrating energy in the corresponding mth imaging channel over all N illumination cycles within the exposure cycle; and The method upsamples the temporal sampling rate of the imager to provide N images of the target.

2. The method according to claim 1, wherein The imaging enhances sensitivity to high frequency characteristics of the target.

3. The method according to claim 1, wherein M=N.

4. The method according to claim 1, wherein The integrated energies are processed to generate N images of the target for each of the M features, for a total of N×M images.

5. The method according to claim 4, wherein Processing the integrated energy to generate the NxM images includes minimizing a cost function.

6. The method according to claim 5, wherein: The cost function includes a time cost function.

7. The method according to claim 5, wherein: The cost function includes a spatiotemporal cost function.

8. The method according to claim 5, wherein The cost function includes a Lagrangian cost function.

9. The method according to claim 1, wherein The EM waves include visible light waves.

10. The method according to claim 1, wherein The EM waves include infrared (IR) waves.

11. The method according to claim 1, wherein The EM waves include ultraviolet (UV) waves.

12. An imaging system operable to image a target, comprising: an electromagnetic (EM) wave source controllable to emit a temporally periodic illumination pattern of electromagnetic (EM) waves, the illumination pattern having a temporal period of T / N, where N is the number of illumination periods in an exposure period T and N>1, and wherein the EM waves are characterized by M>1 substantially simultaneously emitted distinct, distinctive features selected from wavelength bands or polarization directions; a sensor sensitive to the EM wave and operable to acquire M images in M ​​imaging channels during each exposure cycle time period T, wherein each imaging channel corresponds to a respective distinguishing feature, and wherein acquiring the M images comprises integrating energy in the corresponding mth imaging channel over all N illumination cycles within the exposure cycle; and A controller is configured to process the M acquired images so that the imaging system upsamples a temporal sampling rate of an imager to provide N images of the target.

13. The imaging system of claim 12, wherein: The imaging enhances sensitivity to high frequency characteristics of the target.

14. The imaging system of claim 12, wherein: M=N.

15. The imaging system of claim 12, wherein: The integrated energies are processed to generate N images of the target for each of the M features, for a total of N×M images.

16. The imaging system of claim 15, wherein: Processing the integrated energy to generate the NxM images includes minimizing a cost function.

17. The imaging system of claim 16, wherein: The cost function includes a time cost function.

18. The imaging system of claim 16, wherein: The cost function includes a spatiotemporal cost function.

19. The imaging system of claim 16, wherein: The cost function includes a Lagrangian cost function.

20. The imaging system of claim 12, wherein: The EM waves include visible light waves.

21. The imaging system of claim 12, wherein: The EM waves include infrared (IR) waves.

22. The imaging system of claim 12, wherein: The EM waves include ultraviolet (UV) waves.

Images