Method for processing imaging data from a set of pixels and pixel array architecture

Abstract

Methods and pixel array architectures for processing imaging data from a set of pixels are provided. A pixel for an image sensor includes a photon sensor, a memory, and an accumulator. The photon sensor element portion outputs a signal to the accumulator, which can then add together values stored in the memory and can transfer the result to the memory of another pixel. The pixels can be analog or digital and can be used in a pixel array for compressive sensing imaging. Pseudorandom masks can also be used during imaging. Data collected during imaging can be used for compressive sensing and can be sent to an off-line storage device for processing.

Description

Technical Field

[0001] The subject matter disclosed herein relates to compressed sensing with respect to image sensors. More specifically, the subject matter disclosed herein relates to methods for providing data for compressed sensing and corresponding pixel architectures. Background Technology

[0002] The background section below is intended only to provide the context necessary for understanding the inventive ideas and concepts disclosed herein. Therefore, this background section may contain patentable subject matter and should not be considered a disclosure of prior art.

[0003] In signal processing, there are many techniques for acquiring and reconstructing signals. One such technique is called compressed sensing. The idea behind compressed sensing is based on the principle that sparse signals can be reconstructed from fewer samples than required by the Nyquist-Shannon sampling theorem. Compressed sensing attempts to reconstruct a signal by finding a solution to an underdetermined system. An underdetermined system is one in which there are more variables than an equation. Digitally capturing an image and attempting to reconstruct it can be an example of an underdetermined system.

[0004] Duarte et al. (Marco Duarte, Mark Davenport, Dharmpal Takhar, Jason Laska, Ting Sun, Kevin Kelly, and Richard Baraniuk, Single-pixel imaging via compressive sampling. (IEEE Signal Processing Journal, 25(2), p. 83, March 2008)) presented a design for a compressive sensing optical system that involves a digital micro-mirror device that reflects a portion of an incident beam toward a single photodiode sensor. For each clock cycle, a portion of the image can be imaged by the photodiode, and the data is transmitted offline for processing using a compressive sensing algorithm. This method requires additional processing and does not take into account multiple pixels. Furthermore, the technique is limited to stationary objects and has low resolution. Therefore, there is a need for a compressive sensing hardware design that takes into account better sensing options with higher resolution. With better resolution, additional applications including high-speed imaging, time-of-flight sensing, and high dynamic range imaging can be obtained. Summary of the Invention

[0005] An example embodiment provides a pixel for an image sensor, which may include a photon sensor, a memory, and an accumulator. The photon sensor can be used to collect photon information. The accumulator can be used as a temporary storage device to hold intermediate data values ​​of the photon information and to sum the data values ​​of the photon information. The photon sensor may be connected to the accumulator, and the memory may be connected to the accumulator. In some embodiments, the memory and the accumulator may be a single device providing both the functions of storing and accumulating data or charge. During a photon detection event, the photon sensor may convert photons into electrical signals and may send data to the accumulator. Additionally, the memory may send data to the accumulator, and the accumulator may sum the data values. The pixel may be an analog configuration or a digital configuration.

[0006] The example embodiment provides a pixel row or column that can be used in a pixel array in a readout method for compression sensing. A pixel row may include digital or analog pixels connected in series. An accumulator for a first pixel may be connected to the memory of a second pixel, and so on for a pixel row. A data bus may be connected to the memory of the pixels and may provide a clock signal. An accumulator for one of the pixels may be connected to an offline storage device that can be used to collect and process compression sensing data collected from the pixel row.

[0007] The example embodiment provides a pixel array that can be used for compressed sensing. A mask may be applied to the pixel array during sensing.

[0008] The example embodiment provides a series of pixels, wherein data can be transferred between pixels. In a first clock cycle, a first pixel may collect a first photon count from a photonic information element and store the first photon information in an accumulator, add a second photon count from a memory associated with the first pixel to the accumulator to obtain third photon information, send the third photon information to a memory associated with a second pixel, and reset the values ​​in the sensor element and the accumulator to zero.

[0009] Pixel architecture can be used for high-speed imaging to merge a desired number of time slices at a specific speed setting; high dynamic range imaging to limit the output of each pixel in each time slice using a desired upper limit of photon counts, and then merge the time slices together; time-of-flight sensing to capture multiple compressed images and reconstruct the time slices from each run, find the peak of each pixel to determine the time of flight and thus determine the distance; overlap detection, and many other applications. Attached Figure Description

[0010] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings.

[0011] Figure 1AThese are example pixels that can be used in example embodiments of pixel arrays, which can be used in some example embodiments of readout methods for compressed sensing.

[0012] Figure 1B These are example pixels that can be used in example embodiments of pixel arrays, which can be used in some example embodiments of readout methods for compressed sensing.

[0013] Figure 2A These are example columns or rows of pixels that can be used by example embodiments of pixel arrays, which may be used in some example embodiments of readout methods for compressed sensing.

[0014] Figure 2B These are example columns or rows of pixels that can be used by example embodiments of pixel arrays, which may be used in some example embodiments of readout methods for compressed sensing.

[0015] Figure 3 This is an example pixel array that can be used in some example embodiments of readout methods for compressed sensing.

[0016] Figures 4A to 4B It is a data stream and timing diagram that can be used in an example embodiment for compressed sensing of a single column or a single row of pixels in a pixel array.

[0017] Figure 5 It is a data stream and timing diagram that can be used in an example embodiment for compressed sensing of a single column or a single row of pixels in a pixel array.

[0018] Figure 6 It is possible Figures 4A to 4B or Figure 5 The example processing flow used in the example embodiments of data flow and timing diagrams.

[0019] Figure 7 This is an example pseudo-random mask that can be used in example embodiments of compressed sensing.

[0020] Figure 8 This is a schematic diagram of another example embodiment of a pixel architecture with auxiliary memory for merging.

[0021] Figure 9 This is a schematic diagram of another example embodiment of a digital pixel architecture with a SPAD sensor.

[0022] Figure 10 This is a schematic diagram of another example embodiment of a time-resolved pixel architecture with a CCD, CIS, or QIS sensor.

[0023] Figure 11This is another example implementation of a differential pixel architecture that applies a pseudo-random mask. Detailed Implementation

[0024] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. However, those skilled in the art will understand that the aspects disclosed may be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure the subject matter disclosed herein.

[0025] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment," "in an embodiment," or "according to an embodiment" (or other phrases with similar meanings) appearing in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, in one or more embodiments, a particular feature, structure, or characteristic may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, illustration, or description." Any embodiment described herein as "exemplary" is not to be construed as necessarily preferred or superior to other embodiments. Additionally, in one or more embodiments, a particular feature, structure, or characteristic may be combined in any suitable manner. Furthermore, depending on the context discussed herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "pre-determined", "specific pixel", etc.) may occasionally be used interchangeably with their non-hyphenated versions (e.g., "two-dimensional", "pre-determined", "specific pixel", etc.), and uppercase entries (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may be used interchangeably with their non-uppercase versions (e.g., "counter clock", "row select", "pixout", etc.). Such occasional interchangeability should not be considered inconsistent with each other.

[0026] Furthermore, depending on the context of this discussion, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should also be noted that the various figures shown and discussed herein (including component diagrams) are for illustrative purposes only and are not drawn to scale. Similarly, various waveform diagrams and timing diagrams are shown for illustrative purposes only. For example, for clarity, the size of some elements may be exaggerated relative to other elements. Furthermore, reference numerals are repeated in the figures where appropriate to indicate corresponding and / or similar elements.

[0027] The terminology used herein is for the purpose of describing some exemplary embodiments only and is not intended to limit the subject matter of the claims. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “including” are used in this specification, they specify the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the terms “first,” “second,” etc., serve as labels for nouns that follow them and do not indicate any type of order (e.g., spatial, temporal, logical, etc.) unless so explicitly defined. Furthermore, the same reference numerals may be used between two or more figures to denote parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is merely for simplification and ease of discussion and does not imply that the construction or structural details of such components or units are identical across all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0028] It will be understood that when an element or layer is referred to as being "on," "connected to," or "bonded to" another element or layer, it may be directly on, directly connected to, or directly bonded to the other element or layer, or there may be intermediate elements or layers present. Conversely, when an element or layer is referred to as being "directly on," "directly connected to," or "directly bonded to" another element or layer, there are no intermediate elements or layers present. The same reference numerals always denote the same element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0029] As used herein, the terms “first,” “second,” etc., serve as labels for nouns that follow them and do not indicate any kind of order (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used between two or more figures to denote parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is merely for simplification and ease of discussion and does not imply that the construction or structural details of such components or units are identical across all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0030] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject pertains. It will also be understood that, unless clearly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized manner.

[0031] Using compressed sensing, the sparsity of the signal in some domains is assumed to be true, or can be introduced to account for solving the image reconstruction problem. Sparsity reduces the number of variables to be solved in an underdetermined system. For example, some pixels on a pixel array can be forced to zero values ​​to create a sparser signal that can then be used for image reconstruction. One way to force some pixels to zero is to generate a mask. The mask can be a pseudo-randomly generated array of positions that can be applied to the pixel array used to capture the image. It can apply binary values ​​(0 and 1) or other weighted values ​​to the pixels affected by the mask.

[0032] An example compressed sensing measurement can be described as y = Ax + e, where x is a signal (such as an image), y is a sample (such as a sample captured by a pixel array), A is a measurement matrix or sampling operator in an M x N matrix, and e is measurement noise. The variable x can be a vector N x 1, and y can be a vector M x 1, where M can be less than N (typically much smaller than N). In another embodiment, the variable x can be a vector N + b x 1, where b represents the number of shifts associated with the image capture from the image capturing device, which will... Figures 4A to 4B The discussion describes this. Each measurement can be an inner product or cross-relation between a signal x and a sensing function. For a vector x, N can be the value of the signal x sampled S times, which can be considered S-sparse. Using compressed sensing, the vector x is assumed to be S-sparse, or has a pseudo-random mask applied to it to allow sparsity. The compressed sensing algorithm can solve for matrix A.

[0033] If the signal x is sparse, it can have a representation in a basis Z or a redundant dictionary D, where y = Zx or y = Dx. Furthermore, if A satisfies the restricted isometric property, the coefficients of A can be reconstructed through an L1 optimization problem: y = AZx + e, where... Make If the signal x is not sparse, a pseudo-random mask can be applied to the signal to satisfy the optimization problem. Algorithms for solving the optimization problem may include, but are not limited to, basic purist, gradient projection for sparse reconstruction, L1 regularized least squares, fixed-point continuity, fast iterative shrinking thresholding algorithms, deep learning, and Bayesian methods.

[0034] Compressed sensing applications include, but are not limited to, optical systems (such as time-of-flight sensing, high dynamic range imaging, high-speed imaging, holography, and facial recognition), magnetic resonance imaging, and analog-to-digital conversion. Within optical systems, hardware design for pixel arrays and data acquisition architectures is required to implement compressed sensing.

[0035] Figure 1A Example pixel 100 is an example pixel array that can be used in some example embodiments of a readout method for compressed sensing. Pixel 100 includes a photon sensor 101, a memory 102, and an accumulator 103. The photon sensor 101 can be used to collect photon information values ​​and can be a digital or analog element (such as a photodiode, pinned photodiode, single-photon avalanche diode (SPAD), avalanche photodiode (APD), quantum image sensor (QIS), contact image sensor (CIS), charge-coupled device (CCD), radiative thermal imager, or other sensor type). Depending on the specific photon sensor 101 used, the photon information values ​​generated by the photon sensor 101 can be digital (such as a photon count) or analog (such as a charge). The memory element 102 (which may be referred to as memory 102) can be digital or analog memory (such as RAM, SRAM, DRAM) or a register, capacitor, or other memory source. The photon sensor 101 can be connected to the accumulator 103. In some example embodiments, accumulator 103 may be a register that can act as a temporary storage device to hold intermediate values. Accumulator 103 may also add values ​​together. For example, a value already in the accumulator may be added to a value received from photon sensor 101 (e.g., a photon information value). Memory 102 may be connected to accumulator 103. During a photon detection event, photon sensor 101 may convert photons into electrical signals and may send data to accumulator 103. Additionally, memory 102 may send data to accumulator 103, and accumulator 103 may add the data together. Clock signal (CLK) 104 may be used to transmit a clock signal on data bus 105. Data bus 105 may be connected to memory 102 and photon sensor 101 (not shown) and may transmit clock signal 104. Data bus 105 may be used to send clock signal 104 to pixel 100, read data from pixel 100, or both. In some embodiments, separate buses may be used to transmit data and clock signals (not shown). In some embodiments, the photon sensor 101 can be reset to zero after one clock cycle. Furthermore, as will be considered for... Figure 2A The accumulator 103 can also be reset after data is sent to the memory 102 of different pixels 100.

[0036] In some embodiments, memory 102 and accumulator 103 may be combined into a single logic unit that performs the same tasks of memory storage and accumulation, respectively. For example, a capacitor may be an accumulator and a memory storage device and may be used in analog pixel designs. The existing charge in the capacitor, along with additional charge added from the output of photon sensor 101, is accumulated and stored.

[0037] In another embodiment, mask memory 106 may be connected to photon sensor 101 and may provide logic to stop photon sensor 101 from sensing. In another embodiment, mask memory 106 may be connected to accumulator 103 and may provide logic to stop accumulator 103 from adding values ​​from photon sensor 101. Data bus 107 may be connected to mask memory 106 to provide data transfer capability. Mask memory 106 and data bus 107 may be elements used in alternative embodiments for compressed sensing applications utilizing masks, which will target... Figure 1B and Figure 7 Describe it.

[0038] Figure 1B Example pixel 110 is an example pixel that can be used in an example embodiment of a pixel array, which can be used in some example embodiments of a readout method for compressed sensing. Pixel 110 may have the features described above. Figure 1A The elements of pixel 100, in addition to data transmission from accumulator 103, can be connected to output selector 108, which may include memory for reading values ​​from accumulator 103. Data bus 109 may be connected to output selector 108 and may provide commands to read values ​​from accumulator 103.

[0039] Figure 2A This is an example pixel column (or row) 200 that can be used by example embodiments of a pixel array, which may be used in some example embodiments of a readout method for compressed sensing. Pixel column 200 depicts n example pixels 100a, 100b, and 100c connected in series, and the nth last pixel 100n (note that n can be any number greater than 1). For example, pixel 100a may be connected to pixel 100b, pixel 100b may be connected to pixel 100c, and so on. Each pixel may have a photon sensor (e.g., one of photon sensors 101a through 101n). Accumulator 103a may be connected to memory 102b, accumulator 103b may be connected to memory 102c, and so on (e.g., for accumulator 103c and beyond). Data bus 105 may be connected to memories 102a, 102b, 102c, all the way to memory 102n, and can transmit data from memory 102a, 102b, 102c, all the way to memory 102n, and can transmit data from memory 102b, 102b, 102c, 102n, and can transmit data from memory 102b, 102c, 102n, ... Figure 1A or Figure 1BThe clock signal 104 is provided to pixels 100a to 100n. The data bus 107 can be connected to mask memories 106a, 106b, 106c, and all the way to mask memory 106n, and can transmit data from… Figure 1A The mask data is provided to pixels 100a to 100n. Accumulator 103n can be connected to offline storage device 201, which can be used to collect and process compressed sensing data collected from pixel column 200.

[0040] In one embodiment, pixel column 200 may include a digital pixel architecture in which photons can be counted and transmitted as digital packets of information between pixels. In other embodiments, pixel column 200 may include an analog pixel architecture in which photons can be converted into electrical charges, and these charges can be transferred between pixels.

[0041] In other embodiments, data from pixel 100a may not be sent to the memory of pixel 100b, but may instead be sent to pixel 100c. In other embodiments, data from pixel 100a may be sent to subsequent pixels (not shown) in pixel column 200. In other embodiments, as will be considered for... Figure 3 Further described, additional pixel rows 200 may exist, and pixels from the first row may be connected to pixels from the second row. In other embodiments, a pseudo-random mask may be applied to one or more pixel columns 200 to force the values ​​of selected photon sensors 101a to 101n to return to zero or non-zero values, which will target Figure 7 This will be discussed in more detail. That is, an accumulator from a single pixel does not need to be fed into the memory of the immediately adjacent pixel. As will be apparent to those skilled in the art upon reading this disclosure, there are many accumulator-memory feed patterns.

[0042] Figure 2B This is an example pixel column (or row) 200b that can be used by an example embodiment of a pixel array, which may be used in some example embodiments of a readout method for compressed sensing. Pixel column 200b may have Figure 2A The elements of pixel column 200, except for pixel column 200b, can be used Figure 1B Replaced by 110 pixels Figure 1APixel 100. In one embodiment, pixel column 200b can transmit data from pixels 100a to 100n via a set of output selectors 108a to 108n. For example, for pixel 100a, when a data read request for pixel 100a is sent via data bus 109, output selector 108a can select a value from accumulator 103a, which can then send the value to output selector 108a and via data bus 109 to offline storage device 201 or another pixel 100a to 100n. This process can be repeated for any pixel 100a to 100n in pixel column 200b.

[0043] Figure 3 This is an example pixel array 300 that can be used in some example embodiments of a readout method for compressed sensing. The pixel array 300 may have a first column (or row) 301 comprising a plurality of pixels. In this example, eight pixels 100a to 100h are shown, and pixels 100a to 100h may be... Figure 2A Column 200 or Figure 2B Column 200b contains pixels 100a to 100n. A second column 302 may exist with pixels 302a to 302h, a third column 303 with pixels 303a to 303h, and so on, up to column 304 with pixels 304a to 304h, which can also be similar. Figure 2A Column 200 or Figure 2B Column 200b. In other embodiments, different row and column sizes may exist for the pixel array 300, ranging from 1x1 to any other size. As previously stated for Figure 2A As mentioned, pixels in the first column 301 can be connected to each other (e.g., the accumulator of one pixel can be connected to the memory of another pixel), for example, pixel 100a can be connected to pixel 100b, pixel 100b can be connected to pixel 100c, and so on. In other embodiments, pixel 100a from the first column 301 can be connected to pixel 302b of the second column 302, and then pixel 302b of the second column 302 can be connected to pixel 303c of the third column 303, and so on. In other embodiments, other pixel connection configurations spanning columns, rows, or combinations of both may exist.

[0044] Figures 4A to 4B It can be used for Figure 3 Example data streams and timing diagrams 400 used in example embodiments of compressed sensing of a single column or row of pixels in pixel array 300. The data streams described below can be applied to... Figure 3 The pixel array 300 contains all or some of its pixels. In one embodiment, the timing diagram 400 may be... Figure 2AAn example of pixel column 200 is shown, which involves the addition and linear shifting of data without using a mask. In another embodiment, timing diagram 400 may be an example of another series of connected pixels.

[0045] Figures 4A to 4B Depicting a series of connected pixels 100a to 100h over a process of example 7 clock cycles (t = 0 to t = 6), the series of connected pixels 100a to 100h have Figure 3 The first column 301, their respective memories 102a to 102h, and Figure 2A and Figure 2B Offline storage device 201. Pixels 100a to 100h may have accumulators 103a to 103h (not shown). There are three clock cycles for setting the shutter value to open or closed, and four clock cycles for setting the shutter value to close or closed. The shutter may be a physical shutter mechanism, or it may be a logic shutter. The logic shutter allows photon sensors 101a to 101h to be activated or deactivated.

[0046] The shutter value can represent a logical state of whether imaging is occurring or not. An open or closed shutter value indicates that the photon sensors 101a to 101h can sense photons. A closed or closed shutter value indicates that the photon sensors 101a to 101h cannot sense photons.

[0047] In this example, data can be transferred between adjacent pixels. For instance, data from accumulator 103a of pixel 100a can be transferred to memory 102b of pixel 100b, data from accumulator 103b of pixel 100b can be transferred to memory 102c of pixel 100c, and so on. For additional rows where no pixels exist, data can be considered to be moved to... Figure 2A One or more locations within the offline storage device 201 / in Figure 2A The device moves between one or more locations within the offline storage device 201 and is stored... Figure 2A Within one or more locations of the offline storage device 201. The logic used to describe the data flow in column 301 can be applied to... Figure 3 The other columns of the pixel array 300. Generally, the value of pixel 100a at time 0 will be represented as 100a,0, at time 1 it will be represented as 100a,1, and so on. Figures 4A to 4B The numbers in the document will use this format.

[0048] During the first clock cycle (t = 0), pixels 100a through 100h image the scene (not shown). Pixel 100a has 0 in its corresponding memory 102a. Pixel 100a can receive a first photon information value, which may be a count of photons or charges, and can store the first photon information value in accumulator 103a. The value stored in accumulator 103a is added to a second photon information value in memory 102a (which may be 0 at t = 0) to form a third photon information value, and the third photon information value is shifted to memory 102b of pixel 100b. In one embodiment, the shift is performed at the end of the same clock cycle. Then, the photon sensor 101a associated with pixel 100a and accumulator 103a can be set to zero before the next clock cycle, or set to zero at the beginning of the next clock cycle before imaging begins. In another embodiment, the shift can be performed at the beginning of the next clock cycle. In other words, a pixel captures a photon count, stores the photon count in an accumulator associated with the pixel, adds the memory value to the accumulator value, and shifts the new value to the next memory location (which can be an adjacent pixel, a pixel elsewhere in the pixel array, or the next memory location in the offline storage device 201). In one embodiment, a pixel captures a charge, stores the charge in an associated accumulator that also acts as memory, accumulates the charge in memory and from the pixel, and transfers the charge to an accumulator associated with a different pixel in the pixel array. The photon sensor (not shown) of pixel 100a can be reset to zero in preparation for sensing in the next clock cycle. After the data shift and before any new photon sensing, the photon sensor and accumulators 103a to 103h are reset.

[0049] In the second clock cycle (t=1), pixel 100b receives a photon count and stores the photon count in accumulator 103b. The value stored in accumulator 103b is added to the value in memory 102b and shifted to memory 102c of pixel 100c. The photon sensor 101b and accumulator 103b of pixel 100b (not shown) can be reset to zero in preparation for sensing in the next clock cycle.

[0050] Simultaneously, in the first clock cycle (t=0), pixel 100b has 0 in its corresponding memory 102b. Pixel 100b can receive photon counts and store the photon counts in accumulator 103b. The value stored in accumulator 103b is added to memory 102b and shifted to memory 102c. In one embodiment, the shift is performed at the end of the same clock cycle. In another embodiment, the shift can be performed at the beginning of the next clock cycle. The photon sensor (not shown) of pixel 100b can be reset to zero to prepare for sensing in the next clock cycle. In the second clock cycle (t=1), pixel 100c receives photon counts and can store the photon counts in accumulator 103c. The value stored in accumulator 103c is added to the value in memory 102c and shifted to memory 103d. The accumulator 103b of pixel 100c can be reset to zero to prepare for sensing in the next clock cycle.

[0051] The above operation can be repeated for each pixel of the pixel array 300. Therefore, for each clock cycle, each pixel 100a to 100h can add its photon count value to the value in its memory 102a to 102h, which can be the value provided to it from the previous pixel for all t>0. The added value can then be sent to the memory of the next attached pixel.

[0052] At time t=3, the clock shutter state changes from open to closed, and pixels 100a to 100h no longer detect or count photons. The values ​​in each cell memory will no longer be modified by the accumulator and will be shifted over time to offline storage device 201, which is the receiver of the data shift from the memory of the last pixel (102h in this example).

[0053] Data stored in offline storage device 201 can be used for compressed sensing signal reconstruction. Data can be shifted between memories to offline storage device 201. In one embodiment, at time t=1, data from pixel 100h at time t=0 can be shifted to offline storage device 201. Each data value can be shifted to another pixel memory location, to offline storage device 201, or can be kept in the same memory location within each clock cycle.

[0054] In other embodiments, data stored in memory 102a may be sent to memory 102c or to... Figure 3 Other memory in pixel column 301. In other embodiments, memory 102a can send data to... Figure 3The memory in other columns. In other embodiments, memory 102a can be sent back and forth between the memory of another pixel and itself. In other embodiments, it can be sent only to the region of interest (i.e., Figure 3 Data is transmitted from a subset of pixels within the pixel array 300. That is, as mentioned above, the accumulator of each pixel does not need to be connected to the memory of the adjacent pixel, and other interconnection modes are feasible.

[0055] In other embodiments, a pseudo-random mask may be applied. Figure 3 A pixel array of 300. Return to reference. Figures 4A to 4B For example, a pseudo-random mask can be applied to the first column 301, which will refer to Figure 5 This will be discussed in more detail. Other pixel types (such as pixels in pixel array 200b) can be used. In one embodiment, the pseudo-random mask can be the same for all clock cycles. In other embodiments where memory values ​​are not shifted and can remain stored in the same memory cell, a modified pseudo-random mask can be applied. This pseudo-random mask can change for each clock cycle. In one embodiment, the mask can be a series, and this series can be repeated. For example, if four mask patterns A, B, C, and D exist, a mask series ABCD can be used, and this mask series ABCD can be repeated over multiple clock cycles. The mask values ​​can be binary (such as 0 or 1), such that the output values ​​of photon sensors 101a to 101h under the influence of the mask can be forced to 0 or remain unchanged. The following is for... Figure 5 Further description of the mask's operations with respect to pixels.

[0056] The techniques described above can be used with any type of image sensor. For example, regarding... Figures 7 to 10 More specifically, the hardware configuration can vary depending on the type of image sensor.

[0057] In some embodiments, the compressed sensing measurement y = Ax + e can change the variable x from an N x 1 matrix to an N + b x 1 matrix, where b represents the number of “data shifts” that have occurred during imaging (i.e., shutter opening). For example, if there are three shifts within three time periods, the size of x can be N + 3 x 1 instead of N + 1.

[0058] Figure 5 It can be used for Figure 3 Example data stream and timing diagram 500 used in an example embodiment of compressed sensing of a single column or a single row of pixels 301 in pixel array 300. Figure 5 The effect of the mask on pixel array 300 and individual pixels 100 is also described. The data stream described below can be applied to... Figure 3A pixel array of 300 pixels, including all or some pixels. The mask and data stream usage described below can be applied to... Figures 4A to 4B The logic. In one embodiment, timing diagram 500 may be configured with accumulators 103a to 103h, output selectors 108a to 108h, and mask memories 106a to 106h. Figure 2B The example uses pixel column 200b. Pixel 100a can be connected to output selector 108a, pixel 100b can be connected to output selector 108b, and so on for pixel 100h connected to output selector 108h. Mask memory 106a can be part of pixel 100a, mask memory 106b can be part of pixel 100b, and so on for mask memory 106h connected to pixel 100h.

[0059] In an alternative embodiment, the following can be used: Figure 2A Pixel column 200a, or another series of pixels in the pixel array can be used. Mask memories 106a to 106h can Figures 4A to 4B Pixels 100a to 100h and any other arrangement of data stream operations or pixel connections are used.

[0060] Generally speaking, the values ​​of pixel 100a and mask memory 106a will be represented as 100a,0 and 106a,0 respectively at time 0, and as 100a,1 and 106a,1 respectively at time 1, and so on. Figure 5 Numbers in this format can be used.

[0061] Figure 5 Depicting a system with four clock cycles (t = 0 to t = 3) Figure 3 The first column 301 comprises a series of connected pixels 100a to 100h, and photon sensors 101a to 101h for pixels 100a to 100h, respective memories 102a to 102h for pixels 100a to 100h, and respective mask memories 106a to 106h for pixels 100a to 100h. There are three clock cycles in which the shutter value is set to open and one clock cycle in which the shutter value is set to close. In this example, data can be retained in each pixel after each cycle; for example, data from the accumulator of pixel 100a can be accumulated in memory 102a, data from the accumulator of pixel 100b can be accumulated in memory 102b, and so on. That is, in this embodiment, each pixel transfers the result of the cycle to its own memory, rather than the second pixel in the series.

[0062] During the first clock cycle (t=0), pixels 100a to 100h image the scene (not shown). Pixel 100a has a value of 0 in its corresponding memory 102a. The photon sensor 101a of pixel 100a can receive a first photon information value, which can be a count of photons or charges. This count can be multiplied by a binary value from mask memory 106a, and the result can be stored in accumulator 103a. If the binary value from mask memory 106a is equal to 1, the accumulated first photon information value can be stored. If the binary value from mask memory 106a is equal to 0, the accumulator can store a value of 0. This multiplication can be logical, or it can be the result of a physical operation (e.g., the mask can enable or disable photon sensor 101a, resulting in a normal read or a 0 read). The value stored in accumulator 103a is added to memory 102a. In one embodiment, a pixel captures a photon count, a mask transmits a binary value multiplied by the pixel's photon count, the result is stored in an accumulator associated with the pixel, the memory value is added to the accumulator value, and can be stored in memory again. The photon sensor 101a and accumulator 103a of pixel 100a can be reset to zero to prepare for sensing in the next clock cycle.

[0063] In the second clock cycle (t=1), the photon sensor 101a of pixel 100a receives a first photon information value, which is multiplied by a value stored in mask memory 106a, and the result is stored in accumulator 103a. The value stored in accumulator 103a is added to the second photon information value in memory 102a, and the result is stored again in memory as a third photon information value. The photon sensor 101a and accumulator 103a of pixel 100a can be reset to zero to prepare for sensing in the next clock cycle.

[0064] Simultaneously, in the first clock cycle (t=0), pixel 100b has 0 in its corresponding memory 102b. The photon sensor 101b of pixel 100b receives a photon count, which is multiplied by a binary value from mask memory 106b, and the result is stored in accumulator 103b. If the binary value from mask memory 106b is equal to 1, the photon count is stored. If the binary value from mask memory 106b is equal to 0, the accumulator stores a value of 0. The value stored in accumulator 103b is added to memory 102b, and the resulting value is stored in memory 102b. Photon sensor 101b and accumulator 103b can be reset to zero to prepare for sensing in the next clock cycle. In the second clock cycle (t=1), the photon sensor 101b of pixel 100b receives a photon count, which is multiplied by a value stored in mask memory 106b, and the resulting value is stored in accumulator 103b. The value stored in accumulator 103b is added to the value in memory 102b, and the result is stored in memory 102b. Photon sensor 101b and accumulator 103b can be reset to zero to prepare for sensing in the next clock cycle. In another embodiment, accumulator 103b and memory 102b can be a single unit for both accumulation and memory storage, and during reset, the accumulated value can be stored in memory 102b associated with accumulator 103b.

[0065] The above operation can be repeated for each pixel of the pixel array 300. Therefore, for each clock cycle, each pixel 100a to 100h can multiply its photon count value with its mask value and add the result to its memory 102a to 102h. When the shutter opens, the result can be accumulated for t>0. The accumulated value can then be sent to the memory of the next attached pixel.

[0066] At time t=3, the clock shutter state changes from open to closed, and pixels 100a to 100h no longer count photons. The values ​​in each unit memory will no longer be modified by the accumulator, and will be modified by, for example, Figure 1B The output selector element (not shown) of element 108 is read. This data can be sent to offline storage device 201. The data stored in offline storage device 201 can be used for compressed sensing signal reconstruction. The accumulator / memory of the pixel can be reset.

[0067] In some embodiments, the mask value may change for each clock cycle. In other embodiments, the mask value may remain constant. In various embodiments, the number of shutter opening and closing cycles may vary.

[0068] Figure 6 It is possible Figures 4A to 4B or Figure 5 Example processing 600 is used in the example embodiments of data flow and timing diagrams.

[0069] Processing 601 can be used Figure 3 Imaging begins with the pixel array 300. The shutter value can be a logical value that can be switched from closed to open.

[0070] Processing 602 may initiate data acquisition from pixels 100 of pixel array 300. In some embodiments, all pixels 100 of pixel array 300 may be used to acquire data. In other embodiments, a subset of pixels may be used to acquire data.

[0071] Process 603 is, in an alternative embodiment, the application of a pseudo-random mask to pixels 100 of pixel array 300. In some embodiments, the pseudo-random mask may be the same pattern throughout the imaging process. In other embodiments, the pseudo-random mask may vary periodically. In some embodiments, process 603 may be skipped if no mask is used. The mask may allow or prevent photon sensor 101 from sensing, or the mask may allow or prevent the value of photon sensor 101 from being accumulated. The mask may have a value multiplied by the value of photon sensor 101 (such as 0 or 1), or it may prevent charge from photon sensor 101 from being stored.

[0072] Processing 604 can add or sum the values ​​captured by the photon sensor 101 of pixel 100 in pixel array 300 and the values ​​stored in memory 102.

[0073] Processing 605 may store the data accumulated in processing 604. Processing 605 may also include resetting the photon sensor 101 back to zero.

[0074] Processing 606 is an operation in an alternative embodiment that can shift data stored from processing 605. In some embodiments, the data may be shifted to memory 102 associated with different pixels 100. In other embodiments, the data may remain in the same memory 102 and may never be shifted. In other embodiments, the data may be transferred back and forth between a first memory (such as that of the first pixel 100) and a second memory (such as that of the second pixel 100).

[0075] Processing 607 can be repeated from 602 to 606 until a closed shutter value is received.

[0076] Processing 608 can transfer data from pixel array 300 to offline storage device 201 for compressed sensing processing.

[0077] Figure 7This is an example pseudo-random mask 700 that can be used in an example embodiment of compressed sensing. The pseudo-random mask 700 may include a region of interest 701 and a mask region 702. The pseudo-random mask 700 can prevent some pixels in the pixel array 300 from imaging within the region of interest 701. Furthermore, the mask region 702 can prevent all pixels outside the region of interest 701 from imaging and can be used to reduce the imaging area on the pixel array 300. The pseudo-random mask 700 can be used as described above. Figures 4A to 4B / Figure 5 The pseudo-random mask 700 is used during processing. For each clock cycle, the pseudo-random mask 700 can remain constant, or it can be dynamic and changeable. The value of the pseudo-random mask 700 can be stored for later use during signal reconstruction.

[0078] For example, regarding Figure 11 Further described, the pseudo-random mask 700 can be used for lossless reconstruction by capturing differential images. The pseudo-random mask 700 can also be used when the data is too dense, and it can add sparsity to the image to find a solution for image reconstruction. The type of pseudo-random mask 700 used can vary depending on the type of imaging task. The pseudo-random mask 700 can be Bernoulli, Hadamard, Gaussian, Discrete Cosine Transform (DCT), or other mask types. The pseudo-random mask 700 can include binary values, grayscale values, color values, or other values.

[0079] In some embodiments, the mask 700 may be stored in an offline storage device 201 for use during image reconstruction. The mask 700 may be generated by a processor (not shown) and provided during imaging. Figure 3 The pixel array 300 contains pixels. The mask 700 may be provided by an offline storage device 201, or it may be generated by a processor (not shown) and sent to the pixels via a data bus 107. The mask 700 may be stored in mask memories 106a to 106n for use... Figure 2A and Figure 2B Pixels 100a to 100n are used. Mask 700 can be stored in offline storage device 201 for use during signal reconstruction.

[0080] Figure 8 This is a schematic diagram of another example embodiment of a pixel architecture 800 with auxiliary memory for merging. Merging can be the accumulation of photonic information within a single memory. Figure 8 Show Figure 2A Example pixel column 200 or Figure 2BColumn 200b. Pixel 801a may be a pixel in pixel column 200. Photon sensor 101a may be a photodiode. A second memory 802a, a second accumulator 803a, and a second data path 804 may also be present and connected to clock signal 104. During imaging, the second memory 802a may accumulate the value provided by the second accumulator 803a, which may collect data from photon sensor 101a. For each clock cycle when imaging occurs, the second memory 802a may store the value and may not shift the value to other memories. When imaging is complete, the second data path 804 may be used to transfer data from memory 802a to memories 802b, 802c, and / or offline storage device 201 (not shown). In alternative embodiments, (such as, in Figure 1B The second data path 804 can be connected to the output selector 108 (not shown), and data can be transferred to the offline storage device 201 by selecting instead of shifting the data to the offline storage device 201 through other pixels.

[0081] Figure 9 This is a schematic diagram of another example embodiment of a digital pixel architecture 900 with a SPAD sensor. Figure 9 Show Figure 2A Example pixel column 200 or Figure 2B Column 200b. Pixel 901a can be a pixel in pixel column 200. Photon sensor 101a can be a single-photon avalanche diode or other digital sensor. Counters 902a to 902c and a second data path 903, which can also be connected to clock signal 104, may exist. Counter 902a can count photons detected by photon sensor 101a, accumulate and store the data in each clock cycle, and transmit the data. During imaging, counter 902a can accumulate or combine the values ​​provided by photon sensor 101a. For each clock cycle when imaging occurs, counter 902a can store the value and may not shift the value to other counters. When imaging is complete, the second data path 903 can be used to transfer data from memory from counter 902a to counters 902b, 902c and / or to offline storage device 201 (not shown). In another embodiment, counter 902a may employ an algorithm to achieve high dynamic range imaging. If the photon sensor 101a becomes saturated, or if there is a photon signal stronger than the photon signal that the sensor can sense, the counter 902a can use an algorithm to avoid saturation.

[0082] Figure 10 This is a schematic diagram of an example embodiment of the time-resolved pixel architecture 1000. Figure 10 Show Figure 2A Example pixel column 200 or Figure 2BColumn 200b. Pixels 1001a to 1001c can be pixels in pixel column 200. Photon sensor 101a can be combined with memory accumulator unit 1002a to represent an analog output sensor (such as a CCD, CIS, or QIS pixel) that can transfer charge to adjacent pixels. Similarly, photon sensor 101b can be combined with memory accumulator 1002b, and photon sensor 101c can be combined with memory accumulator 1002c. Photon sensor 101a can transfer charge to the combined memory accumulator unit 1002a. The temporal resolution of photon sensor 101a can be based on the clock period of clock signal 104. For example, if the clock period is short, fewer photons can be detected (higher resolution). If the clock period is long, more photons can be detected (lower resolution). During a sensing event, for a single pixel 1001a, the detected charge can be transferred from photon sensor 101a to unit 1002a, which may be a combined memory and accumulator unit. Unit 1002a may have a memory module including a mode for storing memory and an accumulator module including a mode for accumulating charge. After a clock cycle, the charge can be transferred from unit 1002a to unit 1002b associated with pixel 1001b. After sensing is complete, the charge stored in unit 1002a can be transferred to unit 1002b, which can then transfer the charge to unit 1002c, which can send the charge or a digital representation of the charge to an offline storage device 201 (not shown). In another embodiment, such as in Figure 1B In this process, the charge stored in cell 1002a can be connected to output selector 108 (not shown), and data can be transferred to offline storage device 201 by selection instead of by pixel shifting.

[0083] Figure 11 This is an example embodiment of the differential pixel architecture 1100 that applies a pseudo-random mask 1104. Figure 11 Show Figure 2A Example pixel column 200 or Figure 2BColumn 200b. Pixels 1105a to 1105c can be pixels in pixel column 200. Photon sensors 101a to 101c can be analog output sensors (such as CCD, CIS, or QIS sensors). In some embodiments, photon sensors 101a to 101c can be digital output sensors. Photon sensor 101a can transfer charge (in analog cases) or digital values ​​(in digital cases) to combined memory accumulator units 1100a and 1100b. The resolution of photon sensor 101a can be the clock cycle of clock signal 104. During a sensing event, mask 1104 can be applied to pixels 1105a to 1105c. For a single pixel 1105a, the detected charge / value can be transferred from photon sensor 101a to units 1100a and 1100b, which can be combined memory and accumulator units. Units 1100a and 1100b may have a memory module and mode for storing memory and an accumulator module and mode for accumulating charge. Unit 1100b may receive a charge / value multiplied together with a mask value (Mask*) from a pixel regarding mask 1104. Unit 1100a may receive a charge / value multiplied together with an inverted or reversed mask value (!Mask*) from a pixel regarding mask 1104.

[0084] For example, if the value of a pixel in mask 1104 is 0, then the inverted or reversed value of the pixel can be 1. Lossless signal reconstruction can be achieved by providing the value of mask 1104 and the reversed value of mask 1104 to the pixels in pixel architecture 1100.

[0085] After a clock cycle, charge / value can be transferred from unit 1100a to unit 1101a, and from unit 1100b to unit 1101b associated with pixel 1105b. After sensing is complete, the charge / value stored in unit 1100a can be transferred to unit 1101a, and subsequently to unit 1102a, which can send the charge / value to offline storage device 201 (not shown). Similar charge / value transfers can occur for units 1100b to 1102b. This can be used for lossless image reconstruction. In another embodiment, such as, in Figure 1B In this process, the charge stored in cell 1100a can be connected to output selector 108 (not shown), and data can be transmitted to offline storage device 201 by selection instead of by pixel shift data.

[0086] The embodiments of the subject matter and operation described in this specification can be implemented as digital electronic circuit systems, or as computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or as a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a computer storage medium for execution by a data processing device or to control the operation of a data processing device. Optionally or additionally, the program instructions may be encoded on artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals generated to encode information suitable for transmission to a receiver device for execution by the data processing device). The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media may also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or may be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). Furthermore, the operations described herein can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0087] While this specification may contain numerous specific details of implementation, such details should not be construed as limiting the scope of any claimed subject matter, but rather as descriptions of features specific to particular embodiments. Specific features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a particular combination or even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0088] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all the shown operations to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0089] Therefore, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0090] As those skilled in the art will recognize, the innovative concepts described herein can be modified and varied across a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any particular example taught in the foregoing discussion, but is defined by the appended claims.

[0091] Embodiments of the inventive concept can be extended to the following statements without limitation:

[0092] Statement 1: A method for processing imaging data from a pixel set, comprising:

[0093] The clock signal is received at the pixel set, so the first pixel in the pixel set performs the following action:

[0094] Collect the first photon information value from the photon sensor and store the first photon information value in the accumulator;

[0095] The second photon information value from the memory associated with the first pixel is added to the accumulator to obtain the third photon information value;

[0096] Storing third photon information; and

[0097] Reset the photon sensor and accumulator to zero.

[0098] Statement 2: According to the method described in Statement 1, the third photon information is stored in a memory associated with the second pixel.

[0099] Statement 3: According to the method described in Statement 1, the first photon information value, the second photon information value, and the third photon information value are one of the photon count or charge.

[0100] Statement 4: According to the method described in Statement 1, the first pixel in the pixel set repeats the action upon receiving multiple subsequent clock signals.

[0101] Statement 5: The method according to Statement 4, wherein the shutter opens during at least one period of the clock cycle.

[0102] Statement 6: The method according to Statement 5, wherein the shutter is closed during at least one period of the clock cycle.

[0103] Statement 7. According to the method described in Statement 1, the third photon information is stored by shifting to an offline storage device for use in reconstructing the compressed sensing signal.

[0104] Statement 8: The method according to Statement 1, wherein a pseudo-random mask is applied to the pixel set.

[0105] Statement 9: The method according to Statement 8, wherein the pseudo-random mask remains constant during the collection period.

[0106] Statement 10: The method described in Statement 8, wherein the pseudo-random mask is dynamic during the collection period.

[0107] Statement 11: The method according to Statement 1, wherein the pixel set is selected as a region of interest as a subset of a larger pixel array.

[0108] Statement 12. According to the method described in Statement 1, the first pixel also stores photonic information from the photonic sensor in a second memory for merging photonic information in the second memory.

[0109] Statement 13: The method according to Statement 1, wherein the pixel set is digital or analog.

[0110] Statement 14. The method according to Statement 1, wherein the third photon information is stored in a memory associated with the first pixel.

[0111] Statement 15: A pixel array architecture for compressed sensing, comprising:

[0112] A pixel array with rows and columns;

[0113] The data bus is connected to the pixel array;

[0114] Each of at least two pixels in the pixel array further includes:

[0115] Photon sensor;

[0116] Memory;

[0117] Accumulator; and

[0118] The output path connects the accumulator to memory.

[0119] Statement 16: According to the pixel array architecture described in Statement 15, the output path of the first pixel is connected to the memory of the second pixel in the pixel array.

[0120] Statement 17. According to the pixel array architecture described in Statement 15, the photonic sensor includes one of the following: a photodiode, a single-photon avalanche diode, a quantum image sensor, a charge-coupled device, and a contact image sensor.

[0121] Statement 18. According to the pixel array architecture described in Statement 15, the photon sensor further includes a mask memory and a mask output path, wherein the mask memory is used to store mask data and the mask output path connects the mask memory to the photon sensor.

[0122] Statement 19. According to the pixel array architecture described in Statement 15, the pixel array further includes a data selector and a selector output path, the data selector being used to select data from an accumulator, and the selector output path connecting the data selector to an offline storage device.

[0123] Statement 20: According to the pixel array architecture described in Statement 15, wherein the photon sensor is a digital sensor, and each pixel further includes:

[0124] Second memory; and

[0125] The second accumulator is used to accumulate data collected from the photon sensor and the second memory, and to store the accumulated data in the second memory.

[0126] Statement 21: According to the pixel array architecture described in Statement 15, the photon sensor is a single-photon avalanche diode (SPAD), and each pixel also includes a counter for accumulating the output from the SPAD.

[0127] Statement 22: According to the pixel array architecture described in Statement 15, the photonic sensor is an analog sensor and is capable of transferring charge to a memory and an accumulator, and the accumulator is capable of transferring charge to a memory or accumulator associated with different pixels in the pixel array.

[0128] Statement 23: According to the pixel array architecture described in Statement 15, each pixel includes a second memory and a second accumulator, and the second memory is capable of storing the inverse value stored in the memory.

[0129] Statement 24: The pixel array architecture according to Statement 15, wherein the memory and accumulator comprise a single device for storing and accumulating data or charge.

[0130] Statement 25: A method for processing imaging data from a pixel set, comprising:

[0131] The clock signal is received at the pixel set, so the first pixel in the pixel set performs the following action:

[0132] Receive binary mask value;

[0133] Collect the first photon information value from the photon sensor;

[0134] Multiply the binary mask value by the first photon information value and store the result in the accumulator;

[0135] The second photon information value from the memory associated with the first pixel is added to the accumulator to obtain the third photon information value;

[0136] The third photon information value is stored in the memory associated with the first pixel; and

[0137] Reset the photon sensor to zero.

[0138] Statement 26: The method described in Statement 25, wherein the binary mask value is pseudo-randomly generated.

[0139] Statement 27. The method according to Statement 25, wherein the first pixel of the pixel set repeats the action upon receiving a plurality of subsequent clock signals.

[0140] Statement 28. The method according to Statement 25, wherein the shutter opens during at least one period of a clock cycle.

[0141] Statement 29. The method according to Statement 28, wherein the shutter is closed during at least one period of the clock cycle.

[0142] Statement 30, the method according to Statement 25, wherein the third photon information is selected by an output selector and sent to an offline storage device for reconstructing the compressed sensing signal.

[0143] Statement 31: The method according to Statement 25, wherein binary mask values ​​are applied to the pixel set.

[0144] Statement 32, the method according to Statement 26, wherein the pseudo-randomly generated binary mask value remains constant within each clock signal during the collection period.

[0145] Statement 33: The method described in Statement 26, wherein the pseudo-randomly generated binary mask value is dynamic during the collection period.

[0146] Statement 34, the method according to Statement 25, wherein the pixel set is selected as a region of interest as a subset of a larger pixel array.

[0147] Statement 35: The method described in accordance with Statement 25, wherein the pixel set is digital or analog.

Claims

1. A method for processing imaging data from a pixel set, comprising: Receive clock signals at the pixel set. Perform the following action using the first pixel in the pixel set: Collect the first photon information value from the photon sensor and store the first photon information value in the accumulator; The second photon information value from the memory associated with the first pixel is added to the accumulator to obtain the third photon information value; Storing third photon information values; and The photon sensor can be allowed or prevented from sensing by a mask stored in a mask memory connected to the photon sensor, or the value of the photon sensor can be allowed or prevented from being accumulated by a mask stored in a mask memory connected to the accumulator.

2. The method according to claim 1, wherein, The third photon information is stored in the memory associated with the second pixel.

3. The method according to claim 1, wherein, The first pixel in the pixel set repeats the action upon receiving multiple subsequent clock signals.

4. The method according to any one of claims 1 to 3, wherein, The first pixel also stores the first photon information value from the photon sensor in the second memory for merging photon information in the second memory.

5. The method according to any one of claims 1 to 3, wherein, The third photon information value is stored in the memory associated with the first pixel.

6. A pixel array architecture for compressed sensing, comprising: A pixel array with rows and columns; The data bus is connected to the pixel array; Each of at least two pixels in the pixel array includes: Photon sensor, memory, accumulator, The output path connects the accumulator to memory, and A mask memory, connected to a photon sensor and storing a mask for allowing or preventing the photon sensor from sensing, or connected to an accumulator and storing a mask for allowing or preventing the value of the photon sensor from being accumulated.

7. The pixel array architecture according to claim 6, wherein, The output path of the first pixel is connected to the memory of the second pixel in the pixel array.

8. The pixel array architecture according to claim 6, wherein, The photon sensor includes a mask output path that connects a mask memory to the photon sensor.

9. The pixel array architecture according to claim 6, wherein, The pixel array also includes a data selector and a selector output path. The data selector is used to select data from the accumulator, and the selector output path connects the data selector to an offline storage device.

10. The pixel array architecture according to claim 6, wherein, Photon sensors are digital sensors, and each pixel also includes: Second memory; and The second accumulator is used to accumulate data collected from the photon sensor and the second memory, and to store the accumulated data in the second memory.

11. The pixel array architecture according to claim 6, wherein, Each pixel includes a second memory and a second accumulator, and the second memory is capable of storing the inverse value corresponding to the value stored in the memory.

12. The pixel array architecture according to any one of claims 6 to 11, wherein, Memory and accumulator comprise a single device that performs two functions: storing and accumulating data or charge.

13. A method for processing imaging data from a set of pixels, comprising: A clock signal is received at the pixel set. Perform the following action using the first pixel in the pixel set: Receive binary mask value; Collect the first photon information value from the photon sensor; Multiply the binary mask value by the first photon information value and store the result in the accumulator; The second photon information value from the memory associated with the first pixel is added to the accumulator to obtain the third photon information value; The third photon information value is stored in the memory associated with the first pixel or the memory associated with the second pixel; as well as The photon sensor can be allowed or prevented from sensing by a mask stored in a mask memory connected to the photon sensor, or the value of the photon sensor can be allowed or prevented from being accumulated by a mask stored in a mask memory connected to the accumulator.

14. The method according to claim 13, wherein, The binary mask value is generated pseudo-randomly.

15. The method according to claim 13, wherein, The first pixel in the pixel set repeats the action upon receiving multiple subsequent clock signals.

16. The method according to claim 13, wherein, The third photon information value is selected using an output selector and sent to an offline storage device for reconstructing the compressed sensing signal.

17. The method according to claim 13, wherein, The binary mask value is applied to the pixel set.

18. The method according to claim 17, wherein, The pseudo-randomly generated binary mask value remains constant within each clock signal during the collection period.

19. The method of claim 17, wherein, The pseudo-randomly generated binary mask values ​​are dynamic during the collection period.

20. The method according to any one of claims 13 to 19, wherein, The pixel set is selected as the region of interest, which is a subset of the pixel array.

Images