A compact two-dimensional high-resolution coded-aperture imaging method and system

Abstract

The application discloses a compact two-dimensional high-resolution coded aperture imaging method, comprising the following steps: designing a coded mask plate according to the scale of a focal plane array and the compression ratio of an expected high-resolution image; constructing a coded aperture imaging model; light emitted by a light source is focused by an objective lens to the coded mask plate after passing through a scene, reflection or a lens; a focal plane array collects an image modulated by the coded mask plate and transmits the image to a reconstruction calculation unit; a high-resolution image is obtained after the coded image collected is reconstructed by the reconstruction calculation unit; the coded mask plate is integrated on the surface of an image sensor device, a mapping model of a two-dimensional high-resolution image to a three-dimensional low-resolution image is constructed, based on the model, a high-quality two-dimensional high-resolution image is obtained by reconstructing and decoding a low-resolution aliasing image; the method has the characteristics of compact structure, high reconstruction quality and good generalization, and can be used in infrared imaging, low-light night vision imaging and remote sensing imaging.

Description

Technical Field

[0001] The technical field of this invention is image processing and computational imaging technology, and in particular, it relates to a compact two-dimensional high-resolution coded aperture imaging method and system. Background Technology

[0002] Spatial resolution is one of the core indicators of optoelectronic devices. Thanks to advancements in semiconductor technology, the pixel size of commercial cameras can now be as small as 2–4 μm, achieving a spatial resolution of 8K (7680×4320). However, for some industrial, scientific, and defense applications, the pixel size of mid- and far-infrared sensors is difficult to further reduce due to the longer wavelengths of mid- and far-infrared light, resulting in relatively low spatial resolution. In low-light night vision devices, to improve pixel sensitivity, the pixel size is generally designed to be larger, also leading to relatively low spatial resolution. Furthermore, high-resolution image sensors place a heavy burden on electronic systems such as readout circuits, resulting in higher power consumption and limiting their application in fields such as the Internet of Things and remote sensing.

[0003] In recent years, deep learning-based image super-resolution technology has been effective in improving spatial resolution. However, it lacks rigorous mathematical logic and accurate estimation of degradation kernels, and is currently limited to scenarios such as the restoration of old photos / videos, making it difficult to apply in fields such as remote sensing and scientific imaging. High-resolution computational imaging methods based on two-dimensional compressed sensing are also effective means of improving spatial resolution, but they require additional Fourier imaging lenses or scanning mechanisms, making them difficult to apply in practice. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problems solved by this invention are: the lack of rigorous mathematical convergence proof in existing high-resolution imaging technologies, the need for additional Fourier lenses or scanning strategies, the difficulty in further reducing the pixel size of high-resolution sensors, and the high load on readout circuits.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a compact two-dimensional high-resolution coded aperture imaging method, comprising:

[0009] Design a coding mask based on the focal plane array size and the desired compression ratio of the high-resolution image;

[0010] Based on the light field transmission equation, a coded aperture imaging model is constructed;

[0011] The light emitted by the light source passes through the scene, and after being reflected or focused by a lens, it is focused onto the coded mask by the objective lens. The focal plane array acquires the image modulated by the coded mask and transmits it to the reconstruction computing unit. The reconstruction computing unit reconstructs the acquired coded image to obtain a high-resolution image.

[0012] As a preferred option for a compact two-dimensional high-resolution coded aperture imaging method, wherein:

[0013] The design coding mask template includes:

[0014] The size of the coding mask is determined based on the focal plane array size and the desired compression ratio of the high-resolution image. Specifically, when the focal plane array contains M1×M2 pixels, each pixel has a size of c×d, and the desired compression ratio of the high-resolution image is k1 and k2 in the x-axis and y-axis directions respectively, then the coding mask contains N1×N2 pixels, each pixel has a size of a×b, and we have c=k1a, d=k2b, N1=k1M1, N2=k2M2.

[0015] As a preferred option for a compact two-dimensional high-resolution coded aperture imaging method, wherein:

[0016] The design of the encoding mask also includes: k1 and k2 are the compression ratios of the encoding mask in the x-axis and y-axis directions, respectively, and the values ​​are natural numbers greater than or equal to 2; each pixel in the focal plane array corresponds one-to-one with k1×k2 pixels in the encoding mask.

[0017] As a preferred option for a compact two-dimensional high-resolution coded aperture imaging method, wherein:

[0018] The design coding mask template also includes:

[0019] The encoding mask is an encoding aperture template containing closed and unclosed elements. The N1×N2 pixels in the encoding mask contain only closed and unclosed elements, which are denoted as "0" and "1" respectively. The distance between the encoding mask and the focal plane array is 0.

[0020] As a preferred option for a compact two-dimensional high-resolution coded aperture imaging method, wherein:

[0021] The construction of the coded aperture imaging model includes:

[0022] Each pixel in the focal plane array corresponds one-to-one with k1×k2 pixels in the coding mask. The measured value I(x,y) of the (x,y)th pixel in the focal plane array can be expressed as:

[0023]

[0024] in, It is the first encoding mask template. The element value of each pixel, which can be either "0" or "1". It is the desired high-resolution image.

[0025] As a preferred option for a compact two-dimensional high-resolution coded aperture imaging method, wherein:

[0026] The construction of the coded aperture imaging model further includes: in the construction of the coded aperture imaging model of the reconstruction computing unit, the k1×k2 pixel corresponding to each pixel in the focal plane array is transformed from two-dimensional space to three-dimensional space, denoted as a three-dimensional vector 1×1×k1k2; the desired high-resolution image It can be represented as

[0027] As a preferred option for a compact two-dimensional high-resolution coded aperture imaging method, wherein:

[0028] The construction of the coded aperture imaging model also includes: representing the coded aperture imaging model as:

[0029] y = Cx

[0030] in, It is a focal plane array The vectorized representation of, It is the desired high-resolution image. The vectorized representation of, It is the encoding-aware matrix, specifically represented as:

[0031]

[0032] Secondly, embodiments of the present invention provide a compact two-dimensional high-resolution coded aperture imaging system, characterized in that it includes:

[0033] Light source, objective lens, coding mask, focal plane array, and reconstruction computing unit;

[0034] In this process, the light emitted by the light source passes through the scene, is reflected or focused by the objective lens onto the coded mask, and the focal plane array acquires the image modulated by the coded mask and transmits it to the reconstruction computing unit. The reconstruction computing unit reconstructs the acquired coded image to obtain a high-resolution image. The distance between the coded mask and the focal plane array is 0.

[0035] Thirdly, embodiments of the present invention provide a computing device, including:

[0036] Memory and processor;

[0037] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the one or more programs are executed by the one or more processors, the one or more processors implement the compact two-dimensional high-resolution coded aperture imaging method as described in any embodiment of the present invention.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the compact two-dimensional high-resolution coded aperture imaging method.

[0039] The beneficial effects of this invention are as follows: This invention integrates an coded mask on the surface of an image sensor device to construct a mapping model from a two-dimensional high-resolution image to a three-dimensional low-resolution image. Based on this model, a high-quality two-dimensional high-resolution image is obtained by reconstructing and decoding the low-resolution aliased image. It has the characteristics of compact structure, high reconstruction quality, and good generalization, and can be used in infrared imaging, low-light night vision imaging, and remote sensing imaging. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0041] Figure 1 This is an overall flowchart of the compact two-dimensional high-resolution coded aperture imaging method described in the first embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the coded mask and image sensor in the compact two-dimensional high-resolution coded aperture imaging method described in the first embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the reconstruction result of a natural image in a simulation example of the compact two-dimensional high-resolution coded aperture imaging method described in the second embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the reconstruction result of an infrared image in a simulation example of the compact two-dimensional high-resolution coded aperture imaging method described in the second embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the reconstruction result of a remote sensing image in a simulation example of the compact two-dimensional high-resolution coded aperture imaging method described in the second embodiment of the present invention. Detailed Implementation

[0046] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0048] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0049] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0050] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0051] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0052] Example 1

[0053] Reference Figures 1-2 This is the first embodiment of the present invention, which provides a compact two-dimensional high-resolution coded aperture imaging method, comprising:

[0054] S1: Design the coding mask based on the focal plane array size and the desired compression ratio of the high-resolution image;

[0055] Specifically, the design coding mask template includes:

[0056] The size of the coding mask is determined based on the focal plane array size and the desired compression ratio of the high-resolution image. Specifically, when the focal plane array contains M1×M2 pixels, each pixel has a size of c×d, and the desired compression ratio of the high-resolution image is k1 and k2 in the x-axis and y-axis directions respectively, then the coding mask contains N1×N2 pixels, each pixel has a size of a×b, and we have c=k1a, d=k2b, N1=k1M1, N2=k2M2.

[0057] k1 and k2 are the compression ratios in the x-axis and y-axis directions of the coding mask, respectively, and their values ​​are natural numbers greater than or equal to 2; each pixel in the focal plane array corresponds one-to-one with k1×k2 pixels in the coding mask.

[0058] The encoding mask is an encoding aperture template containing closed and unclosed elements. The N1×N2 pixels in the encoding mask contain only closed and unclosed elements, which are denoted as "0" and "1" respectively. The distance between the encoding mask and the focal plane array is 0.

[0059] Furthermore, such as Figure 2 As shown, the coding mask of the present invention uses a coding aperture template that includes closed and non-closed elements.

[0060] The distance between the encoding mask and the focal plane array is 0.

[0061] The encoding mask contains N1×N2 pixels, each with a size of a×b; the focal plane array contains M1×M2 pixels, each with a size of c×d; where c = k1a, d = k2b, N1 = k1M1, N2 = k2M2. k1 and k2 are the compression ratios along the x-axis and y-axis of the encoding mask, respectively, and are natural numbers ≥ 2. Each pixel in the focal plane array corresponds one-to-one with the k1×k2 pixels in the encoding mask.

[0062] Figure 2 The red box in the middle represents a pixel in the focal plane array. There are 256×256 pixels in total, that is, M1=M2=256. The size of each pixel is 10μm, that is, c=d=10μm. Figure 2 The yellow box in the middle represents one pixel of the coding mask. There are a total of 1024×1024 pixels, that is, N1=N2=1024. The size of each pixel is 2.5μm, that is, a=b=2.5μm.

[0063] S2: Construct a coded aperture imaging model based on the light field transmission equation;

[0064] Specifically, the construction of the coded aperture imaging model includes:

[0065] Each pixel in the focal plane array corresponds one-to-one with k1×k2 pixels in the coding mask. The measured value I(x,y) of the (x,y)th pixel in the focal plane array can be expressed as:

[0066]

[0067] in, It is the first encoding mask template. The element value of each pixel, which can be either "0" or "1". It is the desired high-resolution image.

[0068] In the construction of the coded aperture imaging model of the reconstruction computational unit, the k1×k2 pixel corresponding to each pixel in the focal plane array is transformed from two-dimensional space to three-dimensional space, denoted as a three-dimensional vector 1×1×k1k2; the desired high-resolution image It can be represented as

[0069] The coded aperture imaging model is represented as:

[0070] y = Cx

[0071] in, It is a focal plane array The vectorized representation of, It is the desired high-resolution image. The vectorized representation of, It is the encoding-aware matrix, specifically represented as:

[0072]

[0073] S3: The light emitted by the light source passes through the scene, and after being reflected or focused by the objective lens onto the coded mask, the focal plane array acquires the image modulated by the coded mask and transmits it to the reconstruction computing unit. The reconstruction computing unit reconstructs the acquired coded image to obtain a high-resolution image.

[0074] It should be noted that, based on the representation of the coded aperture imaging model, its model is a standard Snapshot-Compressive-Imaging problem with a complete mathematical proof of convergence. High-resolution images can be obtained by reconstructing the acquired coded images using the open-source (OpenAccess) toolkit through the reconstruction computing unit.

[0075] Example 2

[0076] Reference Figures 3-5 As an embodiment of the present invention, a compact two-dimensional high-resolution coded aperture imaging method is provided. To verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0077] S1: Design the coding mask template

[0078] S1.1: Determine the size of the coding mask based on the focal plane array size and the desired compression ratio of the high-resolution image. That is, when the focal plane array contains M1×M2 pixels, each pixel has a size of c×d, and the desired compression ratio of the high-resolution image is k1 and k2 in the x-axis and y-axis directions respectively, then the coding mask contains N1×N2 pixels, each pixel has a size of a×b, and we have c=k1a, d=k2b, N1=k1M1, N2=k2M2.

[0079] S1.2: The N1×N2 pixels in the encoding mask contain only closed and non-closed elements, which are denoted as "0" and "1" respectively.

[0080] S2: Construction of Encoded Aperture Imaging Model

[0081] S2.1: Each pixel in the focal plane array corresponds one-to-one with k1×k2 pixels in the coding mask. The measured value I(x,y) of the (x,y)th pixel in the focal plane array can be expressed as...

[0082]

[0083] in, It is the first encoding mask template. The element value of each pixel, which can be either "0" or "1". It is the desired high-resolution image.

[0084] S2.2: The k1×k2 pixels corresponding to each pixel in the focal plane array are transformed from two-dimensional space to three-dimensional space, denoted as a three-dimensional vector 1×1×k1k2. Therefore, the desired high-resolution image... It can be represented as

[0085] S2.3: Based on S2.1 and S2.2, the coded aperture imaging model can be expressed as follows:

[0086] y = Cx

[0087] in, It is a focal plane array The vectorized representation of, It is the desired high-resolution image. The vectorized representation of, It is the encoding-aware matrix, specifically represented as

[0088]

[0089] S3: Based on the representation of the coded aperture imaging model, its model is a standard Snapshot-Compressive-Imaging problem with a complete mathematical convergence proof. It can be obtained by using open-source (Open Access) toolkits to reconstruct the acquired coded images through reconstruction computing units to obtain high-resolution images.

[0090] Reference Figure 3 , Figure 4 and Figure 5 These are the reconstruction results of natural images, infrared images, and remote sensing images, respectively. Figure 3 , Figure 4 and Figure 5 The parameters in the results are M1=M2=256, N1=N2=1024, k1=k2=4, c=d=10μm, a=b=2.5μm. In the figure, superscripts (a1), (a2), and (a3) ​​represent the reconstruction results of the real image, the direct imaging method using a focal plane array, and the compact two-dimensional high-resolution coded aperture imaging method proposed in this invention, respectively; (b1), (b2), and (b3) represent the magnified results within the boxes in (a1), (a2), and (a3), respectively. From the reconstruction results of the natural image, the infrared image, and the remote sensing image, it can be seen that the compact two-dimensional high-resolution coded aperture imaging method proposed in this invention can effectively improve spatial resolution.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A compact two-dimensional high-resolution coded aperture imaging method, characterized in that, include: Design a coding mask based on the focal plane array size and the desired compression ratio of the high-resolution image; Based on the light field transmission equation, a coded aperture imaging model is constructed; The light emitted by the light source passes through the scene, and after being reflected or focused by a lens, it is focused onto the coded mask by the objective lens. The focal plane array acquires the image modulated by the coded mask and transmits it to the reconstruction computing unit. The reconstruction computing unit reconstructs the acquired coded image to obtain a high-resolution image. The design coding mask template includes: The size of the coding mask is determined based on the focal plane array size and the desired compression ratio of the high-resolution image. Specifically, when the focal plane array contains... 1 pixel, each pixel having a size of 1 The desired compression ratios for the high-resolution image are as follows: x-axis and y-axis are respectively... and The encoding mask template contains 1 pixel, each pixel having a size of 1 ,have , , , ; The design coding mask template also includes: and These are the compression ratios along the x-axis and y-axis of the encoding mask, respectively, and are natural numbers greater than or equal to 2; each pixel in the focal plane array and the pixel in the encoding mask... Each pixel corresponds one-to-one; The design coding mask template also includes: The encoding mask is a coding aperture template containing closed and non-closed elements. Each pixel contains only closed and open elements, denoted as "0" and "1" respectively; the distance between the encoding mask and the focal plane array is 0. The construction of the coded aperture imaging model includes: Each pixel in the focal plane array and the encoding mask Each pixel corresponds one-to-one with the pixel in the focal plane array. Measurement value of each pixel Represented as: ； in, It is the first encoding mask template. The element value of each pixel, which can be either "0" or "1". It is the desired high-resolution image; The construction of the coded aperture imaging model also includes: in the construction of the coded aperture imaging model by the reconstruction computing unit, each pixel in the focal plane array corresponds to... A pixel is transformed from a two-dimensional space to a three-dimensional space, denoted as a three-dimensional vector. ; Desired high-resolution image Represented as ; Constructing a coded aperture imaging model also includes: representing the coded aperture imaging model as: ； in, It is a focal plane array The vectorized representation of, It is the desired high-resolution image. The vectorized representation of, It is the encoding-aware matrix, specifically represented as: 。 2. A compact two-dimensional high-resolution coded aperture imaging system for implementing the compact two-dimensional high-resolution coded aperture imaging method of claim 1, characterized in that, include: Light source, objective lens, encoding mask, focal plane array, and reconstruction computing unit; In this process, the light emitted by the light source passes through the scene, is reflected or focused by the objective lens onto the coded mask, and the focal plane array acquires the image modulated by the coded mask and transmits it to the reconstruction computing unit. The reconstruction computing unit reconstructs the acquired coded image to obtain a high-resolution image. The distance between the coded mask and the focal plane array is 0.

3. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the compact two-dimensional high-resolution coded aperture imaging method of claim 1.

4. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the compact two-dimensional high-resolution coded aperture imaging method of claim 1.

Images