A method of extending dynamic range of an image sensor

Abstract

The application discloses a method for expanding dynamic range of an image sensor, and belongs to the technical field of image sensors, and specifically comprises the following steps: obtaining image sensor device information, adding a liquid crystal cover layer in front of the image sensor according to the obtained image sensor device information; the liquid crystal cover layer controls the light transmittance of each point through a display interface, and performs on-demand attenuation on light irradiated on the image sensor; and when a master controller receives the attenuated image, performs overall equalization operation on the image to obtain an image with expanded dynamic range; the device is simple and reliable, can upgrade an existing camera system, and improves the performance of the camera system; the existing image sensor on the market can be transformed at low cost, so that the maximum light intensity input by the image sensor can be greatly improved, and the dynamic range of the image sensor is improved; the application can output an HDR image in a single frame, does not need to perform multi-frame synthesis, does not need to reduce the frame rate, and does not cause phenomena such as blur and trailing.

Description

Technical Field

[0001] This invention belongs to the field of image sensor technology, specifically a method for extending the dynamic range of an image sensor. Background Technology

[0002] With advancements in industries such as industry, aerospace, measurement and control, and monitoring, the demand for image acquisition equipment is growing stronger. Traditional image sensors, due to their fixed pixel count and limited dynamic range, often suffer from overexposure in high-contrast, backlit, and strong-light environments.

[0003] The traditional solution is to combine multiple frames of images with different exposure times into a single image with a wide dynamic range. However, this method significantly reduces the camera's output frame rate, and image blurring and trailing phenomena can occur between multiple frames due to the movement of objects.

[0004] Another approach is to use a novel high dynamic range pixel structure CMOS image sensor, such as the patent with publication number CN102647567B, which discloses a CMOS image sensor and its pixel structure. However, this method requires redesign, fabrication, and verification due to the use of new technologies. If this method is used for every image sensor, the cycle and cost will be extremely high. Moreover, the complexity of a single CMOS image sensor will increase significantly, and the terminal cost will inevitably rise as well.

[0005] Patent CN103945143B discloses a signal receiving device for increasing the dynamic range of an image sensor. The document also proposes a method for increasing the dynamic range, but this method involves a complex optical path, which reduces image quality and places high demands on the mechanical structure. Furthermore, in this patent, two images are imaged onto the same image sensor, directly reducing the effective pixel count (i.e., resolution) of the image sensor by half.

[0006] Therefore, based on the above problems, the present invention provides a method for extending the dynamic range of an image sensor. Summary of the Invention

[0007] To address the problems of the above solutions, this invention provides a method for extending the dynamic range of an image sensor.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A method for extending the dynamic range of an image sensor, specifically including:

[0010] The system acquires image sensor device information and adds a liquid crystal cover layer in front of the image sensor based on this information. The liquid crystal cover layer controls the transmittance of each point individually through the display interface, and attenuates the light shining on the image sensor as needed. When the main controller receives the attenuated image, it performs an overall deequalization operation on the image to obtain an image with expanded dynamic range.

[0011] Furthermore, the interface of the liquid crystal cover layer is a universal display screen interface.

[0012] Furthermore, the display interface of the liquid crystal cover layer is connected to the video output interface of the main controller, and the image sensor is connected to the video input interface of the main controller. The main controller receives the image transmitted from the image sensor in real time, generates an attenuation coefficient matrix based on the image, and then generates an attenuation pre-equalization image. The pre-equalization image is then transmitted to the liquid crystal cover layer through the video output interface.

[0013] Furthermore, methods for generating attenuation pre-equalization images include:

[0014] The process identifies non-overexposed and overexposed areas in the original image. Non-overexposed areas have a zero attenuation coefficient, while overexposed areas are fitted using the brightness gradient at their edges to identify their brightness values ​​and generate corresponding attenuation coefficients. The attenuated image is then sent to the main controller. The main controller receives the attenuated image and determines if any overexposed areas remain. If overexposed areas are found, the process is repeated until they disappear. If no overexposed areas are found, the corresponding attenuation coefficient matrix is ​​obtained. The number of rows and columns in the attenuation coefficient matrix is ​​the same as the pixel resolution of the image sensor. The resolution of the liquid crystal capping layer is also obtained. Based on the obtained liquid crystal capping layer resolution, the attenuation coefficient matrix is ​​scaled and transformed to match the liquid crystal capping layer resolution. The light attenuation rate of the liquid crystal capping layer is then obtained. Finally, based on the relationship between the obtained light attenuation rate of the liquid crystal capping layer and the input digital value, it is converted into an attenuation pre-equalization image.

[0015] Furthermore, the methods by which the master controller performs overall image de-equalization include:

[0016] By setting a deequalization matrix using a pre-equalized image and a mapping correction matrix, the equalized image is de-equalized using the set deequalization matrix to obtain an image with extended dynamic range.

[0017] Furthermore, methods for setting the deequalization matrix using the pre-equalized image and the mapping correction matrix include:

[0018] Obtain the image sensor resolution. Based on the obtained image sensor resolution, set a deequalization matrix with the same number of rows and columns as the image sensor resolution and all values ​​of zero. Perform pixel-by-pixel operation on the pre-equalized image. Multiply the pixel to be operated on by the mapping correction matrix corresponding to the pixel, and add the result to the item corresponding to the pixel in the deequalization matrix. After the pixel operation of all pre-equalized images is completed, the final deequalization matrix is ​​obtained.

[0019] Furthermore, the mapping correction matrix needs to be pre-calibrated. The calibration process is as follows: the corresponding device is placed facing a uniform planar light source, and then the main controller controls the liquid crystal cover layer to turn on the liquid crystal pixels one by one. Then, the output image of the image sensor is received, and the proportional coefficient of the liquid crystal pixel affecting the image sensor pixel is statistically recorded to form the mapping correction matrix of the liquid crystal pixel.

[0020] After calibration, the main controller stores the mapping correction matrix of all liquid crystal pixels in a non-volatile memory connected to the main controller, which will be read and used by the main controller in subsequent operations.

[0021] Furthermore, the liquid crystal capping layer is tightly bonded to the image sensor chip, and the effective pixel area of ​​the liquid crystal capping layer completely covers the effective pixel photosensitive area of ​​the image sensor.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] 1. The device of the present invention is simple and reliable, and can upgrade existing camera systems to improve their performance;

[0024] 2. This invention can significantly increase the maximum light intensity input to existing image sensors and improve their dynamic range by making low-cost modifications to existing image sensors.

[0025] 3. This invention can output HDR images in a single frame without the need for multi-frame synthesis, without reducing the frame rate, and without producing blurring or ghosting.

[0026] 4. This invention does not change the resolution of the original image sensor and has minimal impact on image quality. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a block diagram illustrating the principle of the present invention;

[0029] Figure 2 This is a top view of the liquid crystal cover layer of the present invention;

[0030] Figure 3 This is a side view of the liquid crystal cover layer of the present invention;

[0031] Figure 4 This is a schematic diagram illustrating the generation of the de-balancing matrix in this invention;

[0032] Figure 5 for Figure 4 Enlarged view of point A in the middle.

[0033] In the diagram: 101, image sensor; 102, liquid crystal cover layer; 103, display interface cable; 201, liquid crystal cover layer pixel; 202, image sensor pixel; 301, pre-equalized image; 302, mapping correction matrix item; 303, de-equalization matrix. Detailed Implementation

[0034] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0035] like Figures 1 to 5 As shown, a method for extending the dynamic range of an image sensor includes the following specific steps:

[0036] The system acquires image sensor device information, such as model and size. Based on this information, a liquid crystal cover layer is added in front of the image sensor. The liquid crystal cover layer is tightly bonded to the image sensor chip, and its effective pixel area completely covers the effective pixel photosensitive area of ​​the image sensor. The liquid crystal cover layer controls the transmittance of each pixel individually through a display interface, attenuating the light shining on the image sensor as needed to prevent pixel saturation and overexposure. When the main controller receives the attenuated image, it performs an overall de-equalization operation to obtain an image with expanded dynamic range.

[0037] The liquid crystal capping layer includes at least:

[0038] Photosensitive structure, the photosensitive structure comprising:

[0039] Photosensitive element:

[0040] A first polarizer is attached to the surface of the photosensitive element and has a first polarization direction;

[0041] The liquid crystal light guide structure includes a first transparent conductive layer bonded to the first polarizer, a liquid crystal layer bonded to the first transparent conductive layer, and a second transparent conductive layer bonded to the liquid crystal layer. The first transparent conductive layer and the second transparent conductive layer control the alignment direction of the liquid crystal layer through an external voltage signal to control the polarization direction of light.

[0042] The corresponding control voltage is output according to the external voltage signal. The control voltage is applied to the first transparent conductive layer and the second transparent conductive layer of the pixel structure to control the light conductivity of the pixel structure.

[0043] The second polarizer, combined with the liquid crystal light guide structure, has a second polarization direction;

[0044] Each pixel structure integrates a first polarizer, a liquid crystal light guide structure, and a second polarizer corresponding to a single pixel;

[0045] A pixel readout circuit, connected to the photosensitive element, is used to read out the electrical signals generated by the photosensitive structure.

[0046] The phase difference between the first polarization direction and the second polarization direction is 0 to 90 degrees.

[0047] Preferably, the phase difference between the first polarization direction and the second polarization direction is 90 degrees.

[0048] The first polarizer, the liquid crystal layer, and the second polarizer together control the light transmittance of the pixel structure.

[0049] In this invention, the liquid crystal cover layer is closely bonded to the image sensor to minimize light crosstalk.

[0050] The interface of the liquid crystal cover layer is a general display interface, including but not limited to LVDS interface, MIPI DSI interface and RGB interface. In this embodiment, the MIPI DSI interface commonly used in displays is used.

[0051] The main controller in this invention includes, but is not limited to, FPGA, ARM SoC, and general-purpose PC. In this embodiment, HiSilicon's ARM SoC Hi3559AV100 is used, which has a high-speed serial bus signal LVDS and a MIPI video input interface for connecting to an image sensor, and also has MIPI DSI and RGB video output interfaces for connecting to a liquid crystal cover layer.

[0052] In this invention, the display interface of the liquid crystal cover layer is connected to the video output interface of the main controller, and the image sensor is connected to the video input interface of the main controller. The main controller receives the image transmitted from the image sensor in real time, generates an attenuation coefficient matrix based on the image, generates an attenuation pre-equalization image, and then transmits the pre-equalization image to the liquid crystal cover layer through the video output interface.

[0053] In one embodiment, a method for generating a pre-equalized attenuation image includes:

[0054] The process involves identifying non-overexposed and overexposed areas in the original image. Non-overexposed areas have a zero attenuation coefficient, while overexposed areas are fitted using the brightness gradient at their edges to identify their brightness values ​​and generate corresponding attenuation coefficients. The attenuated image is then sent to the main controller. The main controller receives the attenuated image and determines if any overexposed areas remain. If overexposed areas are found, the process is repeated until they disappear. If no overexposed areas are found, the corresponding attenuation coefficient matrix is ​​obtained. The number of rows and columns in the attenuation coefficient matrix is ​​the same as the pixel resolution of the image sensor; that is, the attenuation coefficient matrix represents the attenuation coefficient of each pixel in the CMOS image. The resolution of the liquid crystal capping layer is then obtained. Based on this resolution, the attenuation coefficient matrix is ​​scaled and transformed to match the liquid crystal capping layer resolution to obtain its light attenuation rate. Finally, based on the relationship between the obtained light attenuation rate and the input digital value, an attenuation pre-equalization image is generated. The input digital value refers to the digital value input to the liquid crystal, which controls the light attenuation rate. Specific details not disclosed are common knowledge to those skilled in the art and are therefore not described in detail.

[0055] The brightness values ​​of overexposed areas are identified, and corresponding attenuation coefficients are generated. A corresponding attenuation coefficient matching table can be set according to the brightness values ​​of possible overexposed areas. The corresponding attenuation coefficients are generated by matching the identified brightness values ​​with the attenuation coefficient matching table. The brightness values ​​of overexposed areas can be obtained through existing technologies. The specific undisclosed parts are common knowledge to those skilled in the art and can be implemented and replaced using existing technologies.

[0056] In other embodiments, generating the attenuation pre-equalization image can also use methods such as the fixed attenuation coefficient method or the binary approximation method, but the effect is not as good as the previous embodiment.

[0057] The main controller sets the deequalization matrix through the pre-equalized image and the mapping correction matrix, and performs deequalization processing on the equalized image using the set deequalization matrix to obtain an image with extended dynamic range.

[0058] Methods for setting the deequalization matrix using a pre-equalized image and a mapping correction matrix include:

[0059] Obtain the image sensor resolution. Based on the obtained image sensor resolution, set a deequalization matrix with the same number of rows and columns as the image sensor resolution and all values ​​of zero. Perform pixel-by-pixel operation on the pre-equalized image. Multiply the pixel to be operated on by the mapping correction matrix corresponding to the pixel, and add the result to the item corresponding to the pixel in the deequalization matrix. After the pixel operation of all pre-equalized images is completed, the final deequalization matrix is ​​obtained.

[0060] The mapping correction matrix needs to be pre-calibrated. The calibration process is as follows: the corresponding device is placed facing a uniform planar light source, and then the main controller controls the liquid crystal cover layer to turn on the liquid crystal pixels one by one. Then the output image of the image sensor is received, and the proportional coefficient of the liquid crystal pixel affecting the image sensor pixel is statistically recorded to form the mapping correction matrix of the liquid crystal pixel.

[0061] After calibration, the main controller stores the mapping correction matrix of all liquid crystal pixels in a non-volatile memory connected to the main controller, which is then read and used by the main controller during subsequent operations. The undisclosed parts are common knowledge in the field and therefore will not be described in detail.

[0062] The working principle of this invention is as follows: Image sensor device information is acquired, and a liquid crystal overlay layer is added in front of the image sensor based on this information. The display interface of the liquid crystal overlay layer is connected to the video output interface of the main controller, and the image sensor is connected to the video input interface of the main controller. The main controller receives the image transmitted from the image sensor in real time and generates an attenuation coefficient matrix based on the image, thereby generating an attenuated pre-equalized image. The pre-equalized image is then transmitted to the liquid crystal overlay layer through the video output interface. The main controller sets a de-equalization matrix using the pre-equalized image and the mapping correction matrix, and performs de-equalization processing on the equalized image using the set de-equalization matrix to obtain an image with extended dynamic range.

[0063] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for extending the dynamic range of an image sensor, characterized in that, Specific methods include: The system acquires image sensor device information and adds a liquid crystal cover layer in front of the image sensor based on the acquired information. The liquid crystal cover layer controls the transmittance of each point individually through the display interface, and attenuates the light shining on the image sensor as needed. When the main controller receives the attenuated image, it performs an overall deequalization operation on the image to obtain an image with expanded dynamic range. The main controller receives images transmitted from the image sensor in real time, generates an attenuation coefficient matrix based on the images, generates an attenuation pre-equalization image, and then transmits the pre-equalization image to the liquid crystal cover layer through the video output interface. Methods for generating attenuation pre-equalization images include: The process involves identifying non-overexposed and overexposed areas in the original image. Non-overexposed areas have a zero attenuation coefficient, while overexposed areas are fitted using the brightness gradient at their edges to identify their brightness values ​​and generate corresponding attenuation coefficients. The attenuated image is then sent to the main controller. The main controller receives the attenuated image and determines if any overexposed areas remain. If overexposed areas are found, the process is repeated until they disappear. If no overexposed areas are found, the corresponding attenuation coefficient matrix is ​​obtained. The number of rows and columns in the attenuation coefficient matrix is ​​the same as the pixel resolution of the image sensor. The resolution of the liquid crystal capping layer is also obtained. Based on the obtained liquid crystal capping layer resolution, the attenuation coefficient matrix is ​​scaled and transformed to match the liquid crystal capping layer resolution. The light attenuation rate of the liquid crystal capping layer is then obtained. Finally, based on the relationship between the obtained light attenuation rate of the liquid crystal capping layer and the input digital value, it is converted into an attenuation pre-equalization image. The methods used by the master controller to perform overall image deequalization include: By setting a deequalization matrix using a pre-equalized image and a mapping correction matrix, and then using the set deequalization matrix to deequalize the equalized image, an image with extended dynamic range is obtained. The mapping correction matrix needs to be pre-calibrated. The calibration process is as follows: the corresponding device is placed facing a uniform planar light source, and then the main controller controls the liquid crystal cover layer to turn on the liquid crystal pixels one by one. Then the output image of the image sensor is received, and the proportional coefficient of the liquid crystal pixel affecting the image sensor pixel is statistically recorded to form the mapping correction matrix of the liquid crystal pixel. After calibration, the main controller stores the mapping correction matrix of all liquid crystal pixels in a non-volatile memory connected to the main controller, which will then read and use it during subsequent operations.

2. The method for extending the dynamic range of an image sensor according to claim 1, characterized in that, The interface of the liquid crystal cover layer is a universal display screen interface.

3. The method for extending the dynamic range of an image sensor according to claim 2, characterized in that, The display interface of the liquid crystal cover layer is connected to the video output interface of the main controller, and the image sensor is connected to the video input interface of the main controller.

4. The method for extending the dynamic range of an image sensor according to claim 1, characterized in that, Methods for setting the deequalization matrix using a pre-equalized image and a mapping correction matrix include: Obtain the image sensor resolution. Based on the obtained image sensor resolution, set a deequalization matrix with the same number of rows and columns as the image sensor resolution and all values ​​of zero. Perform pixel-by-pixel operation on the pre-equalized image. Multiply the pixel to be operated on by the mapping correction matrix corresponding to the pixel, and add the result to the item corresponding to the pixel in the deequalization matrix. After the pixel operation of all pre-equalized images is completed, the final deequalization matrix is ​​obtained.

5. The method for extending the dynamic range of an image sensor according to claim 1, characterized in that, The liquid crystal cover layer is tightly bonded to the image sensor chip, and the effective pixel area of ​​the liquid crystal cover layer completely covers the effective pixel photosensitive area of ​​the image sensor.

Images