On-chip integrated multispectral coding TDI sensor and imaging method thereof

Abstract

The invention provides an on-chip integrated multispectral coding TDI sensor and an imaging method thereof. The on-chip integrated multispectral coding TDI sensor comprises a TDI sensor, a multispectral coding mask plate and an imaging model, the TDI sensor comprises a pixel array, a reading circuit, a cache circuit, a reconstruction calculation circuit and a time sequence control circuit, and all parts of the TDI sensor are electrically connected; the multispectral coding mask plate covers the photosensitive surface of the pixel array, the coding pixels of the multispectral coding mask plate are plated with multispectral optical filters, and a part of the optical filters are covered with a light-proof medium. On-chip coding is carried out on the TDI sensor to integrate a multispectral mask plate, and a two-dimensional coding measurement value image obtained by scanning is reconstructed to obtain a multispectral two-dimensional image, so that the TDI sensor realizes a multispectral data acquisition function of a plurality of multispectral TDI spectral sensors on a single sensor; and meanwhile, the defects of high energy consumption cost and high space cost caused by the fact that a multispectral TDI sensor needs a plurality of spectral band sensors are overcome.

Description

Technical Field

[0001] The present invention relates to the field of multi-spectral TDI imaging, and specifically to an on-chip integrated multi-spectral encoded TDI sensor and an imaging method thereof. Background Art

[0002] For each spectral band of a multi-spectral TDI sensor, an independent working circuit needs to be mounted on the sensor to complete the acquisition of different spectral data. In the field of aerospace remote sensing, the equipment carrying the multi-spectral TDI sensor provides a small space and low energy supply. The multi-spectral TDI sensor has problems of too large equipment volume and too much energy consumption of the driving circuit. At the same time, the space cost and energy consumption cost of the multi-spectral TDI sensor increase with the increase in the number of multi-spectral bands. In practical applications, these problems often limit the large-scale use of the multi-spectral TDI sensor. Summary of the Invention

[0003] Aiming at how to complete the multi-spectral data acquisition task of the multi-spectral TDI sensor through a single TDI sensor and solve the problems of high space cost and energy consumption cost of the multi-spectral TDI sensor, the present invention proposes an on-chip integrated multi-spectral encoded TDI sensor and an imaging method thereof to solve the problems raised in the above background art. The technical solutions provided by the present invention are as follows:

[0004] An on-chip integrated multi-spectral encoded TDI sensor includes a TDI sensor, a multi-spectral encoding mask plate, and an imaging model; the TDI sensor includes a pixel array, a readout circuit, a buffer circuit, a reconstruction calculation circuit, and a timing control circuit, and all parts of the TDI sensor are electrically connected;

[0005] The multi-spectral encoding mask plate covers the photosensitive surface of the pixel array; the multi-spectral encoding mask plate is accurately aligned with the pixel array, the scale of the encoded pixel array of the multi-spectral encoding mask plate is the same as that of the pixel array, which is pixels, and the size of a single encoded pixel of the multi-spectral encoding mask plate is the same as that of a single pixel of the pixel array;

[0006] The total number of levels of the multi-spectral encoding mask plate is , one level includes all the encoded pixels in a column perpendicular to the scanning direction of the TDI sensor, every consecutive levels form a group, and the total number of groups , which is a natural number greater than or equal to 2; on each encoded pixel of each level within a group, a filter of the same multi-spectral band is plated, and the corresponding multi-spectral band is , the filters plated on the encoded pixels between different levels are different, corresponding to the light passing through different spectral bands; each group is repeatedly plated with the same filter as the previous group;

[0007] According to the designed array, an opaque medium is covered on the partial filter on the encoded pixel, and the coverage of the opaque medium on each group of filters is the same as that of the previous group;

[0008] The imaging model is an encoding matrix and its encoding imaging model.

[0009] Preferably, the multispectral encoding mask plate is integrated on the surface of the TDI image sensor in a on-chip integration manner, and the distance from the TDI image sensor is 0.

[0010] Preferably, the designed array is an array of size containing only 0 and 1, and each element in the array corresponds one-to-one with the encoding pixel array. When the element in the array corresponding to the encoded pixel of the encoding pixel array is 1, an opaque medium is covered on the filter surface of the encoded pixel, and when it is 0, it is not covered. The ratio of the number of 1s in the array to the total number of elements in the array is between 0 and 1, and the coverage of the opaque medium on each group of filters is the same as that of the previous group.

[0011] Preferably, the expression of the imaging model is:

[0012]

[0013] where , represents the number of rows of the encoding pixel array of the multispectral encoding mask plate of the TDI sensor; represents the encoding matrix, represents the spectral matrix; represents the vectorized form of the spectral intensity values corresponding to a group of encoded pixels at the corresponding positions in the encoding pixel array to be obtained, represents matrix dot product, represents the set of real numbers.

[0014] Preferably, the encoding model is specifically expressed as:

[0015]

[0016] where the element represents the opaque medium encoding on the th row and th level corresponding to a group of encoded pixels. When the filter surface of the encoded pixel at this position is covered with an opaque medium, the value is 0, and when it is not covered, the value is 1; other elements in the matrix are 0;

[0017] The spectral matrix is composed of 1 and 0;

[0018] The element​ The elements in represent the spectral intensity value of the spectrum corresponding to the th row and th level in a group of coded pixels; is the number of groups, representing the number of repetitions of a group of identical coded pixels during scanning.

[0019] In a second aspect, an imaging method of an on-chip integrated multi-spectral coded TDI sensor includes the following steps:

[0020] Step 1: Obtain one-dimensional coded measurement value data through the on-chip integrated multi-spectral coded TDI sensor;

[0021] Step 2: Cache the one-dimensional coded measurement value data to obtain a two-dimensional coded measurement value image;

[0022] Step 3: Use the Snapshot-Compressive-Imaging open-source method to reconstruct the two-dimensional coded measurement value image to obtain a multi-spectral two-dimensional image.

[0023] Preferably, in Step 1, specifically, the x-axis direction is set to be parallel to the scanning direction of the TDI sensor, the y-axis direction is set to be perpendicular to the scanning direction of the TDI sensor, the TDI sensor performs push-broom scanning along the x-axis direction, after the multi-spectral coded mask plate modulates the incident light, the coded incident light data is subjected to time delay integration and then output through the readout circuit as one-dimensional coded measurement value data .

[0024] Preferably, in Step 2, specifically, the one-dimensional coded measurement value data is output row by row to the cache circuit for caching. When the cached one-dimensional coded data reaches the preset number of rows after that, rows of one-dimensional coded data are combined into two-dimensional coded measurement value image and output to the reconstruction calculation circuit.

[0025] Preferably, in Step 3, specifically, when reconstructing the two-dimensional coded measurement value image, first map each pixel of the two-dimensional coded measurement value image into a -dimensional coded image, and then use the -dimensional coded image as the input of the reconstruction model. The expression of the reconstruction algorithm is:

[0026]

[0027] where represents the reconstruction result, is a three-dimensional matrix representing the input of the reconstruction, is the two-dimensional coded measurement value, Denote the norm and is a positive integer; the result after model reconstruction is of dimensional multispectral image, and the dimensional multispectral image is divided into corresponding for different spectral bands The two-dimensional multispectral image is the final multispectral data required.

[0028] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: By integrating a multi-spectral mask on the TDI sensor through on-chip coding, and reconstructing the two-dimensional coded measurement value image obtained by scanning to obtain a multi-spectral two-dimensional image, the TDI sensor realizes the multi-spectral data acquisition function of multiple spectral sensors of multi-spectral TDI on a single sensor, and at the same time solves the problems of high energy consumption cost and high space cost caused by the need for multiple spectral band sensors in multi-spectral TDI sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, but do not constitute a limitation to the present invention. In the drawings:

[0030] Figure 1 is a schematic structural diagram of an on-chip integrated multi-spectral coded TDI sensor provided by the present invention;

[0031] Figure 2 is a schematic diagram of an on-chip integrated multi-spectral coded TDI sensor and a coded mask provided by the present invention;

[0032] Figure 3 is a comparison between the reconstruction effect diagram and the real diagram of the first group of algorithms provided by the present invention;

[0033] Figure 4 is a comparison between the reconstruction effect diagram and the real diagram of the second group of algorithms provided by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0035] To make the above objects, features, and effects of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0036] Embodiment 1: An on-chip integrated multi-spectral encoded TDI sensor, as Figure 1 shown, comprising a TDI sensor, a multi-spectral encoding mask plate, and an imaging model; the TDI sensor includes a pixel array, a readout circuit, a buffer circuit, a reconstruction calculation circuit, and a timing control circuit, and all parts of the TDI sensor are electrically connected.

[0037] The multi-spectral encoding mask plate covers the photosensitive surface of the pixel array; the multi-spectral encoding mask plate is precisely aligned with the pixel array, and the scale of the encoded pixel array of the multi-spectral encoding mask plate is the same as that of the pixel array, which is pixels, and the size of a single encoded pixel of the multi-spectral encoding mask plate is the same as that of a single pixel of the pixel array.

[0038] The multi-spectral encoding mask plate is integrated on-chip closely attached to the surface of the TDI image sensor, and the distance between it and the TDI image sensor is 0.

[0039] The total number of levels of the multi-spectral encoding mask plate is , one level contains all the encoded pixels in a column perpendicular to the scanning direction of the TDI sensor, and every consecutive levels form a group, and the total number of groups , which is a natural number greater than or equal to 2; on each encoded pixel of each level within a group, a filter of the same multi-spectral spectral band is coated, and this filter only allows light of the corresponding spectral band to pass through, and the corresponding multi-spectral spectral band is , and the filters coated on the encoded pixels between different levels are different, corresponding to passing light of different spectral bands; each group is repeatedly coated with the same filters as the previous group.

[0040] According to the designed array, an opaque medium is covered on some of the filters on the encoded pixels, and the coverage of the opaque medium on each group of filters is the same as that of the previous group.

[0041] The designed array is an array of size containing only 0 and 1, and each element in the array corresponds one-to-one with the encoded pixel array. When the element in the array corresponding to the encoded pixel of the encoded pixel array is 1, an opaque medium is covered on the surface of the filter on this encoded pixel, and if it is 0, it is not covered. Among them, the ratio of the number of 1s in the array to the total number of elements in the array is between 0 and 1, and the coverage of the opaque medium on each group of filters is the same as that of the previous group.

[0042] Figure 2A schematic diagram of an on-chip integrated multi-spectral encoded TDI sensor and an encoded mask plate is provided. As shown in the figure, a small square surrounded by gray solid lines represents an encoded pixel. Three multi-spectral spectral bands of red, green, and blue are selected. Among them, the squares of red, green, and blue colors represent filter films coated with the three multi-spectral spectral bands of red, green, and blue, and they can only pass light of the corresponding spectral bands. The black square represents that the surface of the filter film of the encoded pixel is covered with an opaque medium. As shown in the figure, when pushing and scanning along the x-axis direction, a column of encoded pixels perpendicular to the x-axis direction is one stage, and the filter films plated on one stage belong to the same spectral band. In the figure, the first stage is plated with a blue filter film, the second stage is plated with a green filter film, and the third stage is plated with a red filter film. The black squares in the figure are where the surfaces of the filter films are covered with an opaque medium. Every three stages form a group, and each group repeats the same filter films as before, and the number of groups is , satisfying is a natural number greater than or equal to 2.

[0043] It is set that the x-axis direction is parallel to the scanning direction of the TDI sensor, and the y-axis direction is perpendicular to the scanning direction of the TDI sensor. Then, the multi-spectral encoded TDI sensor performs push-broom scanning along the x-axis direction. After the multi-spectral encoded mask plate modulates the incident light, time-delay integration is performed on the electrical signal obtained through scanning, and a set of measured value data is output.

[0044] Arrange a set of measured value data in order to form a one-dimensional vector , and the expression is:

[0045]

[0046] where , represents the number of rows of the encoded pixel array of the multi-spectral encoded mask plate of the TDI sensor, that is, the number of pixels in each column of the encoded pixel array in the y-axis direction.

[0047] The spectral imaging model represents the imaging model when there is no opaque medium on the multi-spectral encoded mask plate, and its expression is:

[0048]

[0049] where, represents the total number of groups, which is a natural number greater than or equal to 2; represents the spectral matrix; represents the vectorized form of the spectral intensity values corresponding to a group of encoded pixels in the encoded pixel array that wants to be obtained. The specific expression is:

[0050]

[0051] where, is the spectral matrix, which is composed of 1 and 0;

[0052] The elements in represent the spectral intensity value corresponding to the th row and th level in a group of coded pixels; The spectral intensity value of the spectrum; is the number of groups, representing the number of times a group of identical coded pixels repeats during scanning.

[0053] The imaging model in the case of a multispectral coded mask plate with an opaque medium is the coding matrix and its coding imaging model, and its expression is:

[0054]

[0055] is the coding matrix, represents matrix dot product, and the specific expression is:

[0056]

[0057] where the element represents the opaque medium coding on the spectrum corresponding to the th row and th level in a group of coded pixels. When the filter surface of the coded pixel at this position is covered with an opaque medium, the value is 0; when not covered, the value is 1. Other elements in the matrix are 0.

[0058] Embodiment 2: An imaging method of an on-chip integrated multispectral coded TDI sensor. An on-chip integrated multispectral coded TDI sensor provided in Embodiment 1 includes the following steps:

[0059] Step 1, obtaining one-dimensional coded measurement value data through the on-chip integrated multispectral coded TDI sensor.

[0060] Specifically, set the x-axis direction to be parallel to the scanning direction of the TDI sensor, and the y-axis direction to be perpendicular to the scanning direction of the TDI sensor. The TDI sensor performs push-broom scanning along the x-axis direction. After the multispectral coded mask plate modulates the incident light, the coded incident light data is subjected to time-delay integration and then output through the readout circuit to obtain one-dimensional coded measurement value data. .

[0061] Step 2, caching the one-dimensional coded measurement value data to obtain a two-dimensional coded measurement value image.

[0062] Specifically, the one-dimensional coded measurement value data is output row by row to the cache circuit for caching. When the cached one-dimensional coded data reaches the preset number of rows after that, ​Combine the one-dimensional encoded data rows into the two-dimensional encoded measurement value image and output it to the reconstruction calculation circuit.

[0063] Step 3: Use the Snapshot-Compressive-Imaging open-source method to reconstruct the two-dimensional encoded measurement value image to obtain a multi-spectral two-dimensional image.

[0064] Specifically, when reconstructing the two-dimensional encoded measurement value image, first map each pixel of the two-dimensional encoded measurement value image into a -dimensional encoded image, and then use the -dimensional encoded image as the input of the reconstruction model. The expression of the reconstruction algorithm is:

[0065]

[0066] wherein, represents the reconstruction result, is a three-dimensional matrix representing the input of the reconstruction, is the two-dimensional encoded measurement value, represents the norm, and are positive integers; the result after model reconstruction is -dimensional multi-spectral image. Divide the -dimensional multi-spectral image into corresponding two-dimensional multi-spectral images for different spectral bands, which are the final multi-spectral data required.

[0067] It should be further noted that according to the on-chip integrated multi-spectral encoding mask template TDI sensor imaging model designed by the present invention, this model is a standard Snapshot-Compressive-Imaging problem with a complete mathematical convergence proof, and the Snapshot-Compressive-Imaging open-source algorithm can be used for model training.

[0068] In this embodiment, the two-dimensional image scanned by the TDI image sensor is modulated by means of an on-chip integrated multi-spectral encoding mask plate, and the image is extended to a multi-dimensional encoded image and then reconstructed through an encoded spectral matrix, thereby obtaining a multi-spectral image, which solves the problems that under the existing process level, multi-spectral TDI sensors are expensive, with high space and energy consumption costs. And this implementation case does not require an additional optical modulation system in front of the sensor, and integrates multiple spectral sensors onto a single sensor. The multi-spectral encoding mask template is integrated on the TDI image sensor in an on-chip integrated manner, which is more economical and simple than a multi-spectral TDI sensor that designs and manufactures multiple spectral sensors. ​​​

[0069] To verify the beneficial effects of the present invention, scientific demonstration was carried out through economic benefit calculation and simulation experiments. In this embodiment, simulation experiments were conducted on the reconstruction algorithm.

[0070] Refer to Figure 3 and Figure 4 , which are comparison charts of the real original images and algorithm-reconstructed images of remote sensing spectral images in two different regions. GT represents the real original image, and SCI represents the image reconstructed using the reconstruction algorithm of the present invention. The image resolution is 244×244 for both. (a1) to (a9), (b1) to (b9) respectively represent 9 different spectral bands. The represented bands are 0.4554μm, 0.5881μm, 0.6900μm, 0.8329μm, 1.1208μm, 1.2631μm, 1.6896μm, 2.2103μm, 2.4303μm from left to right. The two numerical values below the pictures represent the PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) indexes between the real value and the reconstructed value of the multispectral data from left to right. The higher the values of both, the higher the degree of image restoration. It can be seen from the two groups of comparison charts that the algorithm can well restore the reconstructed image. The details and structure of the image are restored and reconstructed very well. For different spectral bands, this method can restore very well, and the reconstruction effect of the algorithm on the spectral image data of each band is very good.

[0071] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An on-chip integrated multi-spectral encoded TDI sensor, characterized in that, It includes a TDI sensor, a multi-spectral encoding mask plate, and an imaging model; the TDI sensor includes a pixel array, a readout circuit, a cache circuit, a reconstruction calculation circuit, and a timing control circuit, and all parts of the TDI sensor are electrically connected; The multi-spectral encoding mask plate covers the photosensitive surface of the pixel array; there is precise alignment between the multi-spectral encoding mask plate and the pixel array. The scale of the encoded pixel array of the multi-spectral encoding mask plate is the same as that of the pixel array, which is pixels, and the size of a single encoded pixel of the multi-spectral encoding mask plate is the same as the size of a single pixel of the pixel array; The total number of levels of the multi-spectral encoding mask is , where the first level contains all the encoded pixels in a column perpendicular to the scanning direction of the TDI sensor. Every consecutive levels form a group, and the total number of groups is , which is a natural number greater than or equal to 2. On each encoded pixel of each level within a group, a filter for the same multi-spectral band is coated. The corresponding multi-spectral band is . The filters coated on the encoded pixels between different levels are different, corresponding to light rays passing through different spectral bands. Each group is repeatedly coated with the same filters as the previous group; According to the designed array, an opaque medium is covered on some of the filters on the encoded pixels, and the coverage of the opaque medium on each group of filters is the same as that of the previous group; The imaging model is an encoding matrix and its encoding imaging model.

2. The on-chip integrated multi-spectral encoded TDI sensor according to claim 1, characterized in that, The multi-spectral encoding mask plate is integrated on-chip closely to the surface of the TDI image sensor, and the distance between it and the TDI image sensor is 0.

3. The on-chip integrated multi-spectral encoded TDI sensor according to claim 1, characterized in that, The designed array is an array of size that only contains 0 and 1. Each element in the array corresponds one-to-one with the encoded pixel array. When the array element corresponding to the encoded pixel of the encoded pixel array is 1, an opaque medium is covered on the surface of the filter on the encoded pixel; when it is 0, no covering is done. The ratio of the number of 1s in the array to the total number of elements in the array is between 0 and 1, and the covering situation of the opaque medium on each group of filters is the same as that of the previous group.

4. The on-chip integrated multi-spectral encoding TDI sensor according to claim 3, wherein, The expression of the imaging model is: ； Among them , represents the number of rows of the encoded pixel array of the multispectral encoding mask plate of the TDI sensor; represents the encoding matrix, represents the spectral matrix; represents the vectorized form of the spectral intensity values corresponding to a set of encoded pixels in the encoded pixel array to be obtained, represents the matrix dot product, represents the set of real numbers.

5. The on-chip integrated multi-spectral encoding TDI sensor according to claim 4, characterized in that The encoding model is specifically expressed as: ； The elements Represents the first Line Level corresponding The opaque medium coding on the spectrum takes the value 0 when the filter surface on the coding pixel at this position is covered by the opaque medium, and takes the value 1 when it is not covered; the other elements in the matrix are 0; Spectral matrix Composed of 1s and 0s; Element The element in represents the spectral intensity value corresponding to the th row and th level in a group of encoded pixels; is the number of groups, representing the number of times a group of identical encoded pixels is repeated during scanning.​ 6. An imaging method for an on-chip integrated multi-spectral encoded TDI sensor, which uses an on-chip integrated multi-spectral encoded TDI sensor according to any one of claims 1-5, characterized in that, It includes the following steps: Step 1, obtain one-dimensional encoded measurement value data through on-chip integration of the multi-spectral encoded TDI sensor; Step 2, cache the one-dimensional encoded measurement value data to obtain a two-dimensional encoded measurement value image; Step 3, use the Snapshot-Compressive-Imaging open-source method to reconstruct the two-dimensional encoded measurement value image to obtain a multi-spectral two-dimensional image.

7. An imaging method of an on-chip integrated multi-spectral encoded TDI sensor according to claim 6, characterized in that, Step 1 specifically sets the x-axis direction to be parallel to the scanning direction of the TDI sensor, the y-axis direction to be perpendicular to the scanning direction of the TDI sensor. The TDI sensor performs push-broom scanning along the x-axis direction. After the multi-spectral encoding mask plate modulates the incident light, the encoded incident light data is subjected to time-delay integration and then passed through the readout circuit to output one-dimensional encoded measurement value data .

8. An imaging method of an on-chip integrated multi-spectral encoded TDI sensor according to claim 7, characterized in that, Step 2 specifically outputs the one-dimensional encoded measurement value data line by line to the buffer circuit for buffering. When the buffered one-dimensional encoded data reaches the preset number of lines After that, The line one-dimensional encoded data is combined into The two-dimensional encoded measurement value image is output to the reconstruction calculation circuit.

9. The imaging method of an on-chip integrated multi-spectral encoded TDI sensor according to claim 8, characterized in that, Step 3 specifically refers to when reconstructing the two-dimensional coded measurement value image, first mapping each pixel of the two-dimensional coded measurement value image into a -dimensional coded image, and then using the -dimensional coded image as the input of the reconstruction model. The expression of the reconstruction algorithm is: -dimensional coded image as the input of the reconstruction model. The expression of the reconstruction algorithm is: ； Among them, represents the reconstruction result, is a three-dimensional matrix representing the input for reconstruction, is a two-dimensional encoded measurement value, represents the norm, and are positive integers; The result after model reconstruction is of multi-spectral image. Divide the multi-spectral image into corresponding two-dimensional multi-spectral images for each of the different spectral bands. The two-dimensional multi-spectral images are the final multi-spectral data required.

Images