An on-chip integrated multi-spectral coded TDI sensor and imaging method thereof

Abstract

The present invention proposes an on-chip integrated multispectral coded TDI sensor and imaging method thereof, comprising a TDI sensor, a multispectral coded mask, and an imaging model. The TDI sensor comprises a pixel array, a readout circuit, a buffer circuit, a reconstruction calculation circuit, and a timing control circuit, all of which are electrically connected. The multispectral coded mask covers the photosensitive surface of the pixel array, and the coded pixels of the multispectral coded mask are coated with multispectral filters, with some filters covered with an opaque medium. By integrating the multispectral mask with the TDI sensor on-chip, a multispectral coded measurement image obtained by scanning is reconstructed to produce a multispectral two-dimensional image. This enables the TDI sensor to implement the multispectral data acquisition functions of multiple spectral sensors within a single sensor, while simultaneously overcoming the high energy and space costs associated with the multispectral TDI sensor requiring multiple spectral sensors.

Description

Technical Field

[0001] The present invention relates to the field of multispectral TDI imaging, and in particular to an on-chip integrated multispectral coded TDI sensor and an imaging method thereof. Background Art

[0002] Each spectral band of a multispectral TDI sensor requires independent operating circuitry to collect distinct spectral data. In the aerospace remote sensing field, equipment equipped with multispectral TDI sensors requires limited space and low power supply. Multispectral TDI sensors suffer from excessive size and excessive energy consumption by the drive circuits. Furthermore, the space and energy costs of multispectral TDI sensors increase with the number of spectral bands. These issues often limit the widespread adoption of multispectral TDI sensors in practical applications. Summary of the Invention

[0003] To address the challenges of using a single TDI sensor to complete the multispectral data acquisition task of a multispectral TDI sensor and solve the high space and energy costs of multispectral TDI sensors, the present invention proposes an on-chip integrated multispectral coded TDI sensor and its imaging method to solve the problems raised in the above background technology. The present invention provides the following technical solutions:

[0004] An on-chip integrated multi-spectral encoding TDI sensor includes a TDI sensor, a multi-spectral encoding mask, and an imaging model; the TDI sensor includes a pixel array, a readout circuit, a buffer circuit, a reconstruction calculation circuit, and a timing control circuit, and all parts of the TDI sensor are electrically connected;

[0005] The multi-spectral coding mask is covered on the photosensitive surface of the pixel array; the multi-spectral coding mask is precisely aligned with the pixel array, and the coding pixel array scale of the multi-spectral coding mask is the same as the pixel array scale. The size of a single coded pixel of the multispectral coding mask is the same as the size of a single pixel of the pixel array;

[0006] The total number of multi-spectral coding mask levels is The first level contains all the coded pixels in a column perpendicular to the scanning direction of the TDI sensor. Level is a group, total number of groups , is a natural number greater than or equal to 2; each coded pixel at each level in the group is plated with a filter of the same multispectral band, and the corresponding multispectral band is The filters coated on the coding pixels of different levels are different, corresponding to the light passing through different spectral bands; each group is repeatedly coated with the same filters as the previous group;

[0007] According to the designed array, some filters on the coding pixels are covered with opaque medium, and the coverage of the opaque medium on each group of filters is the same as that of the previous group;

[0008] The imaging model is a coding matrix and its coding imaging model.

[0009] Preferably, the multi-spectral coding mask is integrated on-chip in close contact with the surface of the TDI image sensor, and the distance between the multi-spectral coding mask and the TDI image sensor is zero.

[0010] Preferably, the designed array is a size of only 0 and 1. An array with each element in the array The coding pixel array corresponds one to one. When the array element corresponding to the coding pixel of the coding pixel array is 1, the filter surface on the coding pixel is covered with an opaque medium. If it is 0, it is not covered. The ratio of the number of 1s in the array to the total number of elements in the array is between 0 and 1. The coverage of the opaque medium on each group of filters is the same as that of the previous group.

[0011] Preferably, the imaging model expression is:

[0012]

[0013] in , Indicates the number of rows of the coded pixel array of the multi-spectral coded mask of the TDI sensor; represents the encoding matrix, represents the spectral matrix; The vectorized form of the spectral intensity value corresponding to a set of coded pixels in the coded pixel array that you want to obtain. represents matrix dot product, represents the set of real numbers.

[0014] Preferably, the coding model is specifically expressed as:

[0015]

[0016] The elements Indicates the first pixel in a set of coded pixels Rank Level corresponding The opaque medium coding on the spectrum takes the value 0 when the filter surface on the coding pixel at that position is covered by the opaque medium, and takes the value 1 when it is not covered; the other elements in the matrix are 0;

[0017] Spectral Matrix Composed of 1s and 0s;

[0018] element Elements in Indicates the first pixel in a set of coded pixels Rank Level corresponding spectral intensity value of the spectrum; is the number of groups, which represents the number of times a group of identical coded pixels are repeated during the scanning process.

[0019] In a second aspect, an imaging method of an on-chip integrated multi-spectral coded TDI sensor comprises the following steps:

[0020] Step 1: Obtain one-dimensional coded measurement data through the on-chip integrated multi-spectral coded TDI sensor;

[0021] Step 2: caching the one-dimensional coded measurement value data to obtain a two-dimensional coded measurement value image;

[0022] Step 3: Reconstruct the two-dimensional coded measurement value image using the Snapshot-Compressive-Imaging open source method to obtain a multispectral two-dimensional image.

[0023] Preferably, step 1 specifically sets the x-axis direction to be parallel to the scanning direction of the TDI sensor, and the y-axis direction to be perpendicular to the scanning direction of the TDI sensor. The TDI sensor performs push-scanning along the x-axis direction. After the multi-spectral coding mask modulates the incident light, the coded incident light data is time-delayed integrated and then output through the readout circuit to output one-dimensional coded measurement value data. .

[0024] Preferably, step 2 is to output the one-dimensional coded measurement value data line by line to the cache circuit for caching. When the cached one-dimensional coded data reaches a preset number of lines, Afterwards, Row one-dimensional encoded data is combined into The two-dimensional coded measurement value image is output to the reconstruction calculation circuit.

[0025] Preferably, step 3 is to first map each pixel of the two-dimensional coded measurement value image into a of dimensional encoded image, and then The dimensional encoded image is used as the input of the reconstruction model, and the expression of the reconstruction algorithm is:

[0026]

[0027] in, Represents the reconstruction result, is a three-dimensional matrix representing the reconstructed input, is the two-dimensional code measurement value, represents the norm, and is a positive integer; the result after model reconstruction is of dimensional multispectral image, dimensional multispectral images are divided into corresponding Different spectral bands The two-dimensional multispectral image is the multispectral data ultimately required.

[0028] Compared with the existing technology, the beneficial effects achieved by the present invention are: by performing on-chip encoding on the TDI sensor and integrating a multispectral mask plate, the two-dimensional coded measurement value image obtained by scanning is reconstructed to obtain a multispectral two-dimensional image, so that the TDI sensor can realize the multispectral data acquisition function of multiple spectral sensors of multispectral TDI on a single sensor, and at the same time solve the shortcomings of high energy consumption and high space cost caused by the need for multiple spectral sensors of the multispectral TDI sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 This is a schematic diagram of the structure of the on-chip integrated multi-spectral coded TDI sensor provided by the present invention;

[0031] Figure 2 Schematic diagram of the on-chip integrated multi-spectral coded TDI sensor and coding mask provided by the present invention;

[0032] Figure 3 This is the comparison between the first set of algorithm reconstruction effect images and the real images provided by the present invention;

[0033] Figure 4 This is a comparison between the second set of algorithm reconstruction effect images and the real images provided by the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and effects of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1: An on-chip integrated multi-spectral coded TDI sensor, such as Figure 1 As shown, it includes a TDI sensor, a multi-spectral coding mask and an imaging model; the TDI sensor includes a pixel array, a readout circuit, a cache circuit, a reconstruction calculation circuit and a timing control circuit, and all parts of the TDI sensor are electrically connected.

[0037] The multi-spectral coding mask is covered on the photosensitive surface of the pixel array; the multi-spectral coding mask is precisely aligned with the pixel array, and the coding pixel array scale of the multi-spectral coding mask is the same as the pixel array scale. The size of a single coded pixel of the multispectral coded mask is the same as the size of a single pixel of the pixel array.

[0038] The multi-spectral coding mask is integrated on-chip closely to the surface of the TDI image sensor, and the distance between the multi-spectral coding mask and the TDI image sensor is zero.

[0039] The total number of multi-spectral coding mask levels is The first level contains all the coded pixels in a column perpendicular to the scanning direction of the TDI sensor. Level is a group, total number of groups , is a natural number greater than or equal to 2; each coded pixel at each level in the group is coated with a filter of the same multi-spectral band, which only allows light of the corresponding spectrum band to pass through. The corresponding multi-spectral band is The filters coated on the coding pixels of different levels are different, corresponding to the light passing through different spectral bands; each group is repeatedly coated with the same filters as the previous group.

[0040] According to the designed array, some filters on the coding pixels are covered with opaque media, and the coverage of the opaque media on each group of filters is the same as that of the previous group.

[0041] The designed array contains only 0 and 1 and its size is An array with each element in the array The coding pixel array corresponds one to one. When the array element corresponding to the coding pixel of the coding pixel array is 1, the filter surface on the coding pixel is covered with an opaque medium. If it is 0, it is not covered. The ratio of the number of 1s in the array to the total number of elements in the array is between 0 and 1. The coverage of the opaque medium on each group of filters is the same as that of the previous group.

[0042] Figure 2Provided is a schematic diagram of an on-chip integrated multi-spectral coded TDI sensor and a coding mask. As shown in the figure, a small square surrounded by a gray solid line in the figure represents a coding pixel, and three multi-spectral bands of red, green and blue are selected, wherein the red, green and blue squares represent filters coated with the three multi-spectral bands of red, green and blue, which can only pass light of the corresponding bands. The black square represents the filter surface of the coding pixel covered with an opaque medium. As shown in the figure, if the x-axis direction is pushed and scanned, a column of coding pixels perpendicular to the x-axis direction is a level, and the filters coated on the level belong to the same spectral band. In the figure, the first level is coated with a blue filter, the second level is coated with a green filter, and the third level is coated with a red filter. The black square in the figure is a filter surface covered with an opaque medium. Every three levels form a group, and each group repeats the same filter as before, and the number of groups is ,satisfy is a natural number greater than or equal to 2.

[0043] Set the x-axis direction to be parallel to the scanning direction of the TDI sensor and the y-axis direction to be perpendicular to the scanning direction of the TDI sensor. The multi-spectral coded TDI sensor performs push-broom scanning along the x-axis direction. After the multi-spectral coded mask modulates the incident light, it performs time-delay integration on the electrical signal obtained by scanning and outputs a set of measurement value data.

[0044] Arrange the data of a set of measurement values into a one-dimensional vector in order , the expression is:

[0045]

[0046] in , Indicates the number of rows of the encoding pixel array of the multi-spectral encoding mask of the TDI sensor, that is, the number of pixels in each column of the encoding pixel array in the y-axis direction.

[0047] The spectral imaging model represents the imaging model of the multi-spectral coding mask without an opaque medium, and its expression is:

[0048]

[0049] in, represents the total number of groups, which is a natural number greater than or equal to 2; represents the spectral matrix; The vectorized form of the spectral intensity value corresponding to a set of coded pixels in the coded pixel array that you want to obtain is expressed as follows:

[0050]

[0051] in, is a spectrum matrix consisting of 1s and 0s;

[0052] Elements in Indicates the first pixel in a set of coded pixels Rank Level corresponding spectral intensity value of the spectrum; is the number of groups, which represents the number of times a group of identical coded pixels are repeated during the scanning process.

[0053] The imaging model of the multispectral coding mask with an opaque medium is the coding matrix and its coding imaging model, which is expressed as follows:

[0054]

[0055] is the encoding matrix, Represents matrix dot multiplication, the specific expression is:

[0056]

[0057] The elements Indicates the first pixel in a set of coded pixels Rank Level corresponding The opaque medium coding on the spectrum takes the value 0 when the filter surface on the coding pixel at that position is covered by the opaque medium, and takes the value 1 when it is not covered; the other elements in the matrix are 0.

[0058] Example 2: An imaging method for an on-chip integrated multi-spectral coded TDI sensor. The on-chip integrated multi-spectral coded TDI sensor provided in Example 1 includes the following steps:

[0059] Step 1: Obtain one-dimensional coded measurement data through the on-chip integrated multi-spectral coded TDI sensor.

[0060] Specifically, the x-axis direction is set to be parallel to the scanning direction of the TDI sensor, and the y-axis direction is set to be perpendicular to the scanning direction of the TDI sensor. The TDI sensor performs push-sweep scanning along the x-axis direction. After the multi-spectral coding mask modulates the incident light, the coded incident light data is time-delayed and integrated, and then outputted through the readout circuit to obtain one-dimensional coded measurement data. .

[0061] Step 2: Cache the one-dimensional coded measurement value data to obtain a two-dimensional coded measurement value image.

[0062] Specifically, the one-dimensional coded measurement data is output row by row to the cache circuit for caching. When the cached one-dimensional coded data reaches the preset number of rows, Afterwards, Row one-dimensional encoded data is combined into The two-dimensional coded measurement value image is output to the reconstruction calculation circuit.

[0063] Step 3: Reconstruct the two-dimensional coded measurement value image using the Snapshot-Compressive-Imaging open source method to obtain a multispectral two-dimensional image.

[0064] Specifically, when reconstructing the two-dimensional coded measurement value image, each pixel of the two-dimensional coded measurement value image is first mapped into a of dimensional encoded image, and then The dimensional encoded image is used as the input of the reconstruction model, and the expression of the reconstruction algorithm is:

[0065]

[0066] in, Represents the reconstruction result, is a three-dimensional matrix representing the reconstructed input, is the two-dimensional code measurement value, represents the norm, and is a positive integer; the result after model reconstruction is of dimensional multispectral image, dimensional multispectral images are divided into corresponding Different spectral bands The two-dimensional multispectral image is the multispectral data ultimately required.

[0067] It should be further explained that the on-chip integrated multi-spectral coded mask TDI sensor imaging model designed according to the present invention is a standard Snapshot-Compressive-Imaging problem with a complete mathematical convergence proof, and the Snapshot-Compressive-Imaging open source algorithm can be used for model training.

[0068] This embodiment utilizes an on-chip integrated multispectral coding mask to modulate the two-dimensional image scanned by the TDI image sensor. This image is then expanded into a multidimensional coded image, which is then reconstructed using a coded spectral matrix to produce a multispectral image. This addresses the high cost, space, and energy consumption of multispectral TDI sensors under current technology. Furthermore, this embodiment eliminates the need for an additional optical modulation system in front of the sensor, integrating multiple spectral sensors into a single sensor. The on-chip integration of the multispectral coding mask into the TDI image sensor is more economical and simpler than designing and manufacturing a multispectral TDI sensor with multiple spectral sensors.

[0069] In order to verify the beneficial effects of the present invention, this embodiment conducts a simulation experiment on the reconstruction algorithm through economic benefit calculation and simulation experiments for scientific demonstration.

[0070] Reference Figure 3 and Figure 4 , which compares the original images and the reconstructed images using the algorithm using two sets of remote sensing spectral images from different regions. GT represents the original image, and SCI represents the image reconstructed using the reconstruction algorithm of the present invention. The image resolution is 244×244. (a1) to (a9), and (b1) to (b9) represent nine different spectral bands, representing, from left to right, 0.4554μm, 0.5881μm, 0.6900μm, 0.8329μm, 1.1208μm, 1.2631μm, 1.6896μm, 2.2103μm, and 2.4303μm. The two numbers below the image, from left to right, represent the PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) indicators between the true and reconstructed values of the multispectral data. Higher values indicate a higher degree of image restoration. From the two sets of comparison figures, it can be seen that the algorithm can restore and reconstruct the image very well, and the image details and structure restoration and reconstruction effects are very good. This method can restore different spectral bands very well, and the algorithm has a very good reconstruction effect on the spectral image data of each band.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An on-chip integrated multi-spectral coded TDI sensor, characterized in that: It includes a TDI sensor, a multi-spectral coding mask and an imaging model; the TDI sensor includes a pixel array, a readout circuit, a buffer circuit, a reconstruction calculation circuit and a timing control circuit, and all parts of the TDI sensor are electrically connected; The multi-spectral coding mask is covered on the photosensitive surface of the pixel array; the multi-spectral coding mask is precisely aligned with the pixel array, and the coding pixel array scale of the multi-spectral coding mask is the same as the pixel array scale. The size of a single coded pixel of the multispectral coding mask is the same as the size of a single pixel of the pixel array; The total number of multi-spectral coding mask levels is The first level contains all the coded pixels in a column perpendicular to the scanning direction of the TDI sensor. Level is a group, total number of groups , is a natural number greater than or equal to 2; each coded pixel at each level in the group is plated with a filter of the same multispectral band, and the corresponding multispectral band is The filters coated on the coding pixels of different levels are different, corresponding to the light passing through different spectral bands; each group is repeatedly coated with the same filters as the previous group; According to the designed array, some filters on the coding pixels are covered with opaque medium, and the coverage of the opaque medium on each group of filters is the same as that of the previous group; The imaging model is a coding matrix and its coding imaging model. The coding imaging model expression is: ； in , Indicates the number of rows of the coded pixel array of the multi-spectral coded mask of the TDI sensor; represents the encoding matrix, represents the spectral matrix; The vectorized form of the spectral intensity value corresponding to a set of coded pixels in the coded pixel array that you want to obtain. represents matrix dot product, Represents the set of real numbers, specifically expressed as: ； The elements Indicates the first pixel in a set of coded pixels Rank Level corresponding The opaque medium coding on the spectrum takes the value 0 when the filter surface on the coding pixel at that position is covered by the opaque medium, and takes the value 1 when it is not covered; the other elements in the matrix are 0; Spectral Matrix Composed of 1s and 0s; element Elements in Indicates the first pixel in a set of coded pixels Rank Level corresponding spectral intensity value of the spectrum; is the number of groups, which represents the number of times a group of identical coded pixels are repeated during the scanning process.

2. The on-chip integrated multi-spectral coded TDI sensor according to claim 1, characterized in that: The multi-spectral coding mask is integrated on-chip closely to the surface of the TDI image sensor, and the distance between the multi-spectral coding mask and the TDI image sensor is zero.

3. The on-chip integrated multi-spectral coded TDI sensor according to claim 1, characterized in that: The designed array contains only 0 and 1 and its size is An array with each element in the array The coding pixel array corresponds one to one. When the array element corresponding to the coding pixel of the coding pixel array is 1, the filter surface on the coding pixel is covered with an opaque medium. If it is 0, it is not covered. The ratio of the number of 1s in the array to the total number of elements in the array is between 0 and 1. The coverage of the opaque medium on each group of filters is the same as that of the previous group.

4. An imaging method of an on-chip integrated multi-spectral coded TDI sensor, using an on-chip integrated multi-spectral coded TDI sensor according to any one of claims 1 to 3, characterized in that: The following steps are involved: Step 1: Obtain one-dimensional coded measurement data through the on-chip integrated multi-spectral coded TDI sensor; Step 2: caching the one-dimensional coded measurement value data to obtain a two-dimensional coded measurement value image; Step 3: Reconstruct the two-dimensional coded measurement value image using the Snapshot-Compressive-Imaging open source method to obtain a multispectral two-dimensional image.

5. The imaging method of an on-chip integrated multi-spectral coded TDI sensor according to claim 4, characterized in that: Step 1 specifically sets the x-axis direction to be parallel to the scanning direction of the TDI sensor and the y-axis direction to be perpendicular to the scanning direction of the TDI sensor. The TDI sensor performs push-scanning along the x-axis direction. After the multi-spectral coding mask modulates the incident light, the coded incident light data is time-delayed integrated and then output through the readout circuit to output the one-dimensional coded measurement value data. .

6. The imaging method of an on-chip integrated multi-spectral coded TDI sensor according to claim 5, characterized in that: Step 2 is to output the one-dimensional coded measurement data line by line to the cache circuit for caching. When the cached one-dimensional coded data reaches the preset number of lines, Afterwards, Row one-dimensional encoded data is combined into The two-dimensional coded measurement value image is output to the reconstruction calculation circuit.

7. The imaging method of an on-chip integrated multi-spectral coded TDI sensor according to claim 6, characterized in that: Step 3 is to reconstruct the two-dimensional coded measurement value image by first mapping each pixel of the two-dimensional coded measurement value image into a of dimensional encoded image, and then The dimensional encoded image is used as the input of the reconstruction model, and the expression of the reconstruction algorithm is: ； in, Represents the reconstruction result, is a three-dimensional matrix representing the reconstructed input, is the two-dimensional code measurement value, represents the norm, and is a positive integer; The result after model reconstruction is of dimensional multispectral image, dimensional multispectral images are divided into corresponding Different spectral bands The two-dimensional multispectral image is the multispectral data ultimately required.

Images