Coding filter array sensor for low-illumination imaging and calculation imaging method thereof

Abstract

The invention discloses a coding filter array sensor for low-illumination imaging and a calculation imaging method of the coding filter array sensor. The coding method comprises the following steps: step 1, determining a scale and a size; step 2, designing a coating film; step 3, constructing an imaging model; 4, reconstructing a two-dimensional image; and 5, obtaining a two-dimensional image. The method has the advantages that on the premise that the hardware structure of the image sensor is not changed, the imaging capacity of the image sensor under the low-illumination condition is improved by designing a specific filter array and a coding mode, and therefore a clean two-dimensional image is obtained.

Description

Technical Field

[0001] The present invention relates to a coding method, in particular to a coding filter array sensor for low-illuminance imaging and a computational imaging method thereof, belonging to the technical field of image processing and computational imaging. Background Art

[0002] In low-light environments, image sensors are susceptible to noise interference, resulting in reduced image quality. Due to design limitations and sensor performance limitations, the traditional one-to-one correspondence between individual image sensor pixels and filter array pixels often results in color distortion or reduced saturation. This affects the visual effect and subsequent processing of the image.

[0003] Although image quality can be optimized through software algorithms at this stage, hardware limitations are still a bottleneck for improving low-light imaging capabilities. Therefore, this application proposes a coded filter array sensor for low-light imaging and a computational imaging method thereof, which measures the aliased output of multiple filter array pixels through a single image sensor pixel, and combines image reconstruction algorithms and fusion algorithms to improve low-light environment perception capabilities. Summary of the invention

[0004] The purpose of the present invention is to provide a coded filter array sensor for low-light imaging and a computational imaging method thereof in order to solve the above-mentioned problems. Without changing the hardware structure of the image sensor, the imaging capability in low-light conditions is improved by designing a specific filter array and encoding method, thereby obtaining a clean two-dimensional image.

[0005] The present invention achieves the above-mentioned object through the following technical scheme, a computational imaging method of a coded filter array sensor for low-light imaging, the coding method comprising the following steps:

[0006] Step 1: Determine the scale and size. Determine the scale and size of the filter array according to the scale of the pixel array of the sensor. Determine the scale of the coding pixels in the coding pixel array of the coding mask and the size of the single coding pixel according to the scale of the single pixel of the filter array.

[0007] Step 2: Coating design: design coating for pixels in the filter array, encode the coded pixels in the coded pixel array, and generate a coded mask;

[0008] Step 3: Construct an imaging model based on the light field transmission principle;

[0009] Step 4: Reconstructing the two-dimensional image, reconstructing the two-dimensional measurement values ​​of the coded filter array sensor for low-light imaging by solving the mathematical convergence problem to obtain the image corresponding to each channel of the R channel, G channel, W channel, and N channel of the two-dimensional image;

[0010] Step 5: Obtain a two-dimensional image by inferring the two-dimensional measurement value of the B channel from the R channel, G channel, and W channel of the reconstructed two-dimensional image and performing color fusion on the images corresponding to the R channel, G channel, B channel, W channel, and N channel to obtain a two-dimensional image.

[0011] Preferably, the step of determining the scale and size is specifically as follows:

[0012] Step 1: Set the image sensor array size to N×N pixels, the single sensor pixel size to n×n, the filter array pixel size to M×M, the single pixel size to m×m, and the encoding mask pixel size and single pixel size to be the same as the filter array size;

[0013] Step 2, the coding mask contains M×M pixels, and the pixel size of a single coding mask is m×m. Then the image sensor and the coding mask array scale satisfy N=2M, the pixel size of a single sensor and the pixel size of a single coding mask satisfy n=2m, each pixel in the image sensor corresponds one-to-one to 2×2 pixels in the filter array, and each pixel in the filter array corresponds one-to-one to each pixel in the coding mask.

[0014] Preferably, the coating design steps are specifically as follows:

[0015] Step 1: First, the 2×2 filter array pixels corresponding to each image sensor pixel are coated in the order of R channel, G channel, W channel, and N channel, where the R channel is the red light-transmitting part, the G channel is the green light-transmitting part, the W channel is the white light-transmitting part, and the N channel is the infrared-transmitting part;

[0016] Step 2: The filter array is closely attached to the surface of the image sensor in an on-chip integrated manner;

[0017] Step 3: randomly encode the coding pixels in the coding mask plate with "0" or "1", where "0" is the part that visible light cannot penetrate and "1" is the part that visible light can penetrate, to obtain a coding mask plate;

[0018] Step 4: The coding mask is closely attached to the filter array surface in an on-chip integrated manner, with a distance of 0 from the filter array. The coding mask imaging model determines the structure of the coding matrix.

[0019] Preferably, the step of constructing the imaging model is specifically as follows:

[0020] Step 1: The light emitted by the light source passes through the scene and is focused by the objective lens onto the coding mask after reflection or lens. The filter array uses different types of coatings to collect data from the corresponding channels. Finally, the measurement value L(x,y) of the (x,y)th pixel in a low-resolution image with a resolution of N×N is obtained. The expression is:

[0021]

[0022] Among them, Z(x i ,y j ) is the first (x i ,y i ) pixel position value, which can be "0" or "1", S1(x i ,y j ) is a coded filter array sensor for low-light imaging to obtain the desired high-resolution image pixel value;

[0023] Step 2: Map the coded pixels in the two-dimensional space corresponding to each pixel in the two-dimensional coded image to the three-dimensional space, recorded as a three-dimensional vector. At this time, the coded pixels are R channel, G channel, W channel, and N channel from top to bottom along the Z-axis direction.

[0024] Preferably, the mathematical convergence problem expression is:

[0025]

[0026] in, represents the reconstruction result, y is a three-dimensional matrix, x represents the reconstruction input, ||·|| represents the norm, S1 and S2 are positive integers;

[0027] The result after reconstruction is The reconstruction result is layered and extracted along the Z-axis direction to extract two-dimensional images of four channels: R channel, G channel, W channel, and N channel.

[0028] Preferably, the step of obtaining a two-dimensional image is specifically as follows:

[0029] Step 1: Perform color fusion on the images corresponding to the R channel, the G channel, the B channel, the W channel, and the N channel to obtain a two-dimensional image;

[0030] Step 2, processing the reconstructed two-dimensional images of the four channels, namely, the R channel, the G channel, the W channel, and the N channel; calculating the measurement value of the B channel through the measurement values ​​of the R channel, the G channel, and the W channel, and obtaining a two-dimensional image by color fusion of the R channel, the G channel, the B channel, the W channel, and the N channel;

[0031] The color fusion formula is:

[0032] I fused =α×I visible +(1-α)×I infrared

[0033] Among them I fused represents the fused image, I visible represents the visible light image, I infrared Represents infrared image, α is the weight coefficient, and its value is between 0 and 1.

[0034] A coded filter array sensor for low-light imaging, comprising a filter array generation module, a coded mask generation module, an integrated design module, a data acquisition system and an image reconstruction module, wherein the filter array generation module, the coded mask generation module, the integrated design module, the data acquisition system and the image reconstruction module are used to implement the steps of a computational imaging method of a coded filter array sensor for low-light imaging;

[0035] The filter array generation module is used to design and optimize the structure of the filter array and determine the scale and layout of the filter array according to the characteristics of the image sensor;

[0036] The coding mask generation module is to design and optimize the coding pixel array of the coding mask, design the coding mask according to the scale and layout of the filter array, and perform random "0" and "1" coding;

[0037] The integrated design module physically integrates the filter array, the coding mask and the image sensor so that the filter array, the coding mask and the sensor are precisely aligned;

[0038] The data acquisition system controls the imaging device to perform an image acquisition process and acquires the encoded two-dimensional image;

[0039] The image reconstruction module reconstructs the encoded two-dimensional image into a three-dimensional image by solving a mathematical formula convergence.

[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of a computational imaging method of a coded filter array sensor for low-light imaging.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a computational imaging method for a coded filter array sensor for low-light imaging.

[0042] The beneficial effect of the present invention is that, without changing the hardware structure of the image sensor, by designing a specific filter array and encoding method, the imaging capability under low illumination conditions is improved, thereby obtaining a clean two-dimensional image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flow chart of the present invention;

[0044] Figure 2 It is a structural block diagram, filter array and coding mask diagram of the present invention;

[0045] Figure 3 It is a schematic diagram of the process of converting a two-dimensional coded image into a three-dimensional image according to the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The embodiment of the present invention discloses a computational imaging method of a coded filter array sensor for low-illuminance imaging.

[0048] According to the attached Figure 1-3 As shown, the encoding method includes the following steps:

[0049] Step 1: Determine the scale and size. Determine the scale and size of the filter array according to the scale of the pixel array of the sensor. Determine the scale of the coding pixels in the coding pixel array of the coding mask and the size of the single coding pixel according to the scale of the single pixel of the filter array.

[0050] Step 2: Coating design: design coating for pixels in the filter array, encode the coded pixels in the coded pixel array, and generate a coded mask;

[0051] Step 3: Construct an imaging model based on the light field transmission principle;

[0052] Step 4: Reconstructing the two-dimensional image, reconstructing the two-dimensional measurement values ​​of the coded filter array sensor for low-light imaging by solving the mathematical convergence problem to obtain the image corresponding to each channel of the R channel, G channel, W channel, and N channel of the two-dimensional image;

[0053] Step 5: Obtain a two-dimensional image by inferring the two-dimensional measurement value of the B channel from the R channel, G channel, and W channel of the reconstructed two-dimensional image and performing color fusion on the images corresponding to the R channel, G channel, B channel, W channel, and N channel to obtain a two-dimensional image.

[0054] Without changing the hardware structure of the image sensor, by designing a specific filter array and encoding method, its imaging capability in low-light conditions is improved, thereby obtaining a clean two-dimensional image.

[0055] Specifically disclosed, the steps for determining scale and size are:

[0056] Step 1: Set the image sensor array size to N×N pixels, the single sensor pixel size to n×n, the filter array pixel size to M×M, the single pixel size to m×m, and the encoding mask pixel size and single pixel size to be the same as the filter array size;

[0057] Step 2, the coding mask contains M×M pixels, and the pixel size of a single coding mask is m×m. Then the image sensor and the coding mask array scale satisfy N=2M, the pixel size of a single sensor and the pixel size of a single coding mask satisfy n=2m, each pixel in the image sensor corresponds one-to-one to 2×2 pixels in the filter array, and each pixel in the filter array corresponds one-to-one to each pixel in the coding mask.

[0058] Specifically disclosed, the coating design steps are as follows:

[0059] Step 1: First, the 2×2 filter array pixels corresponding to each image sensor pixel are coated in the order of R channel, G channel, W channel, and N channel, where the R channel is the red light-transmitting part, the G channel is the green light-transmitting part, the W channel is the white light-transmitting part, and the N channel is the infrared-transmitting part;

[0060] Step 2: The filter array is closely attached to the surface of the image sensor in an on-chip integrated manner;

[0061] Step 3: randomly encode the coding pixels in the coding mask plate with "0" or "1", where "0" is the part that visible light cannot penetrate and "1" is the part that visible light can penetrate, to obtain a coding mask plate;

[0062] Step 4: The coding mask is closely attached to the filter array surface in an on-chip integrated manner, with a distance of 0 from the filter array. The coding mask imaging model determines the structure of the coding matrix.

[0063] Specifically disclosed, the steps of constructing the imaging model are as follows:

[0064] Step 1: The light emitted by the light source passes through the scene and is focused by the objective lens onto the coding mask after reflection or lens. The filter array uses different types of coatings to collect data from the corresponding channels. Finally, the measurement value L(x,y) of the (x,y)th pixel in a low-resolution image with a resolution of N×N is obtained. The expression is:

[0065]

[0066] Among them, Z(x i ,y j ) is the first (x i ,y i ) pixel position value, which can be "0" or "1", S1(x i ,y j ) is a coded filter array sensor for low-light imaging to obtain the desired high-resolution image pixel value;

[0067] Step 2: Map the coded pixels in the two-dimensional space corresponding to each pixel in the two-dimensional coded image to the three-dimensional space, recorded as a three-dimensional vector. At this time, the coded pixels are R channel, G channel, W channel, and N channel from top to bottom along the Z-axis direction.

[0068] In particular, the mathematical convergence problem expression is:

[0069]

[0070] in, represents the reconstruction result, y is a three-dimensional matrix, x represents the reconstruction input, ||·|| represents the norm, S1 and S2 are positive integers;

[0071] The result after reconstruction is The reconstruction result is layered and extracted along the Z-axis direction to extract two-dimensional images of four channels: R channel, G channel, W channel, and N channel.

[0072] Particularly disclosed, the steps of obtaining a two-dimensional image are specifically as follows:

[0073] Step 1: Perform color fusion on the images corresponding to the R channel, the G channel, the B channel, the W channel, and the N channel to obtain a two-dimensional image;

[0074] Step 2, processing the reconstructed two-dimensional images of the four channels, namely, the R channel, the G channel, the W channel, and the N channel; calculating the measurement value of the B channel through the measurement values ​​of the R channel, the G channel, and the W channel, and obtaining a two-dimensional image by color fusion of the R channel, the G channel, the B channel, the W channel, and the N channel;

[0075] The color fusion formula is:

[0076] I fused =α×I visible +(1-α)×I infrared

[0077] Among them I fused represents the fused image, I visible represents the visible light image, I infrared Represents infrared image, α is the weight coefficient, and its value is between 0 and 1.

[0078] A coded filter array sensor for low-light imaging, comprising a filter array generation module, a coded mask generation module, an integrated design module, a data acquisition system and an image reconstruction module, wherein the filter array generation module, the coded mask generation module, the integrated design module, the data acquisition system and the image reconstruction module are used to implement a computational imaging method of a coded filter array sensor for low-light imaging;

[0079] The filter array generation module is used to design and optimize the structure of the filter array and determine the size and layout of the filter array according to the characteristics of the image sensor;

[0080] The coding mask generation module is to design and optimize the coding pixel array of the coding mask, design the coding mask according to the scale and layout of the filter array, and perform random "0" and "1" coding;

[0081] The integrated design module physically integrates the filter array, the coding mask and the image sensor to ensure precise alignment between the filter array, the coding mask and the sensor.

[0082] The data acquisition system controls the imaging device to perform the image acquisition process and acquire the encoded two-dimensional image;

[0083] The image reconstruction module reconstructs the encoded two-dimensional image into a three-dimensional image by solving a mathematical formula convergence.

[0084] A computer device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the processor implements the steps of a computational imaging method of a coded filter array sensor for low-light imaging.

[0085] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a computational imaging method for a coded filter array sensor for low-light imaging.

[0086] Determine the scale and size of the filter array according to the scale of the pixel array of the sensor, and determine the scale of the coding pixels in the coding pixel array of the coding mask and the size of the single coding pixel according to the scale of the single pixel of the filter array;

[0087] Designing coatings for pixels in the filter array, encoding coding pixels in the coding pixel array, and generating coding masks;

[0088] According to the principle of light field transmission, an imaging model is constructed;

[0089] The two-dimensional measurement values ​​of the coded filter array sensor for low-light imaging are reconstructed by solving the mathematical convergence problem to obtain the image corresponding to each channel of the R channel, G channel, W channel and N channel of the two-dimensional image;

[0090] A two-dimensional image is obtained by inferring the two-dimensional measurement value of the B channel from the R channel, G channel, and W channel of the reconstructed two-dimensional image and performing color fusion on the images corresponding to the R channel, G channel, B channel, W channel, and N channel.

[0091] Without changing the hardware structure of the image sensor, by designing a specific filter array and encoding method, its imaging capability in low-light conditions is improved, thereby obtaining a clean two-dimensional image.

[0092] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A computational imaging method for a coded filter array sensor for low-light imaging, characterized in that: The encoding method comprises the following steps: Step 1: Determine the scale and size. Determine the scale and size of the filter array according to the scale of the pixel array of the sensor. Determine the scale of the coding pixels in the coding pixel array of the coding mask and the size of the single coding pixel according to the scale of the single pixel of the filter array. Step 2: Coating design: design coating for pixels in the filter array, encode the coded pixels in the coded pixel array, and generate a coded mask; Step 3: Construct an imaging model based on the light field transmission principle; Step 4: Reconstructing the two-dimensional image, reconstructing the two-dimensional measurement values ​​of the coded filter array sensor for low-light imaging by solving the mathematical convergence problem to obtain the image corresponding to each channel of the R channel, G channel, W channel, and N channel of the two-dimensional image; Step 5: Obtain a two-dimensional image by inferring the two-dimensional measurement value of the B channel from the R channel, G channel, and W channel of the reconstructed two-dimensional image and performing color fusion on the images corresponding to the R channel, G channel, B channel, W channel, and N channel to obtain a two-dimensional image.

2. The computational imaging method of a coded filter array sensor for low-light imaging according to claim 1, characterized in that: The steps of determining the scale and size are specifically as follows: Step 1: Set the image sensor array size to N×N pixels, the single sensor pixel size to n×n, the filter array pixel size to M×M, the single pixel size to m×m, and the encoding mask pixel size and single pixel size to be the same as the filter array size; Step 2, the coding mask contains M×M pixels, and the pixel size of a single coding mask is m×m. Then the image sensor and the coding mask array scale satisfy N=2M, the pixel size of a single sensor and the pixel size of a single coding mask satisfy n=2m, each pixel in the image sensor corresponds one-to-one to 2×2 pixels in the filter array, and each pixel in the filter array corresponds one-to-one to each pixel in the coding mask.

3. The computational imaging method of a coded filter array sensor for low-light imaging according to claim 1, characterized in that: The coating design steps are specifically as follows: Step 1: First, the 2×2 filter array pixels corresponding to each image sensor pixel are coated in the order of R channel, G channel, W channel, and N channel, where the R channel is the red light-transmitting part, the G channel is the green light-transmitting part, the W channel is the white light-transmitting part, and the N channel is the infrared-transmitting part; Step 2: The filter array is closely attached to the surface of the image sensor in an on-chip integrated manner; Step 3: randomly encode the coding pixels in the coding mask plate with "0" and "1", where "0" is the part that visible light cannot penetrate, and "1" is the part that visible light can penetrate, to obtain a coding mask plate; Step 4: The coding mask plate is closely attached to the surface of the filter array in an on-chip integrated manner, with a distance between the coding mask plate and the filter array being 0.

4. The computational imaging method of a coded filter array sensor for low-light imaging according to claim 1, characterized in that: The steps of constructing the imaging model are specifically as follows: Step 1: The light emitted by the light source passes through the scene and is focused by the objective lens onto the coding mask after reflection or lens. The filter array uses different types of coatings to collect data from the corresponding channels. Finally, the measurement value L(x,y) of the (x,y)th pixel in a low-resolution image with a resolution of N×N is obtained. The expression is: Among them, Z(x i ,y j ) is the first (x i ,y i ) pixel position value, which can be "0" or "1", S1(x i ,y j ) is a coded filter array sensor for low-light imaging to obtain the desired high-resolution image pixel value; Step 2: Map the coded pixels in the two-dimensional space corresponding to each pixel in the two-dimensional coded image to the three-dimensional space, recorded as a three-dimensional vector. At this time, the coded pixels are R channel, G channel, W channel, and N channel from top to bottom along the Z-axis direction.

5. The computational imaging method of a coded filter array sensor for low-light imaging according to claim 1, characterized in that: The mathematical convergence problem expression is: in, represents the reconstruction result, y is a three-dimensional matrix, x represents the reconstruction input, ||·|| represents the norm, S1 and S2 are positive integers; The result after reconstruction is The reconstruction result is layered and extracted along the Z-axis direction to extract two-dimensional images of four channels: R channel, G channel, W channel, and N channel.

6. The computational imaging method of a coded filter array sensor for low-light imaging according to claim 1, characterized in that: The step of obtaining a two-dimensional image is specifically as follows: Step 1: Perform color fusion on the images corresponding to the R channel, the G channel, the B channel, the W channel, and the N channel to obtain a two-dimensional image; Step 2, processing the reconstructed two-dimensional images of the four channels, namely, the R channel, the G channel, the W channel, and the N channel; The measurement value of the B channel is calculated through the measurement values ​​of the R channel, the G channel, and the W channel, and a two-dimensional image is obtained by color fusion of the R channel, the G channel, the B channel, the W channel, and the N channel; The color fusion formula is: I fused =α×I visible +(1-a)×I infrared Among them I fused represents the fused image, I visible represents the visible light image, I infrared Represents infrared image, α is the weight coefficient, and its value is between 0 and 1.

7. A coded filter array sensor for low-light imaging, comprising a filter array generation module, a coded mask generation module, an integrated design module, a data acquisition system and an image reconstruction module, characterized in that: The filter array generation module, the coding mask generation module, the integrated design module, the data acquisition system and the image reconstruction module are used to implement the steps of the computational imaging method of the coding filter array sensor for low-light imaging as described in any one of claims 1 to 6; The filter array generation module is used to design and optimize the structure of the filter array and determine the scale and layout of the filter array according to the characteristics of the image sensor; The coding mask generation module is to design and optimize the coding pixel array of the coding mask, design the coding mask according to the scale and layout of the filter array, and perform random "0" and "1" coding; The integrated design module physically integrates the filter array, the coding mask and the image sensor so that the filter array, the coding mask and the sensor are precisely aligned; The data acquisition system controls the imaging device to perform an image acquisition process and acquires the encoded two-dimensional image; The image reconstruction module reconstructs the encoded two-dimensional image into a three-dimensional image by solving a mathematical formula convergence.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a computational imaging method of a coded filter array sensor for low-light imaging as described in any one of items 1-6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a computational imaging method of a coded filter array sensor for low-light imaging as described in any one of items 1-6 are implemented.

Images