A spatially independent exposure high dynamic range enhancement method for array cameras

Abstract

This invention belongs to the field of array camera imaging technology, specifically a method for enhancing high dynamic range (HDR) using spatially independent exposures for array cameras. The array cameras are configured as an N×N array, with each camera comprising a camera head, an image sensor, and an image processing chip. The cameras have four different exposures. The method includes independently controlling the exposure of each image processing chip; performing digital gain compensation on the spatially independent exposures to restore the image to a linear space consistent with the natural environment; and performing spatially independent stretching and unified fusion, where the different exposures output by the same image sensor are fused to produce a single result. This method overcomes the limitations of the unified exposure of traditional camera image sensors by fully utilizing each image sensor to protect its own field of view, ensuring that each region achieves the most appropriate exposure, minimizes noise, and maintains the optimal dynamic range for each region.

Description

Technical Field

[0001] The present invention belongs to the field of array camera imaging technology, and in particular relates to a spatially independent exposure high dynamic range enhancement method applicable to array cameras. Background Art

[0002] Large-scene cameras handle harsher scenes, offer a wider dynamic range, and provide more image content and detail within the scene. However, single-channel image sensors are limited by chip design frequency, frame rate limitations, readout speed limitations, and other factors, resulting in a limited number of exposures. Currently, the most widely used image sensors on the market use DOL 4-exposure technology, which typically only supports a maximum of 22 bits and a 132dB dynamic range. Furthermore, this dynamic range cannot capture all brightness ranges, and many details in between cannot be guaranteed.

[0003] This invention proposes a spatially independent exposure high dynamic range technology for array cameras. This technology leverages the array camera's numerous independent image sensors and image processors, allowing each image sensor to independently expose its own image, achieving optimal exposure for its specific area. This allows for multiple independent exposure values ​​across the entire array camera, covering a wider dynamic range. Each exposure area is appropriately exposed, and details in intermediate areas are well preserved. This enables array cameras to achieve a dynamic range far exceeding that of traditional cameras. Summary of the Invention

[0004] The purpose of the present invention is to propose a spatially independent exposure high dynamic range enhancement method suitable for array cameras facing more severe application scenarios, wider dynamic range, more scene content and details.

[0005] The present invention provides a method for enhancing the high dynamic range of an array camera space independent exposure, wherein the array camera is set as an N×N array, with a total of N 2 cameras; each camera has a camera head, an image sensor, and an image processing chip; each camera has four different exposures, of which the three typical ones are: long exposure (L_exposure), medium exposure (M_exposure), and short exposure (S_exposure); the other can be an exposure value between two adjacent exposure lengths; corresponding to the four different exposures, there are four exposure channels; an image processing chip includes four image processors, one corresponding to each of the four exposure channels; an N×N array camera has 4N 2 For example, if N=8, there are 64 cameras. Figure 1This is an illustration of an 8×8 array. The left array represents the real-world brightness seen by each camera, covering a wide dynamic range. The array area encompasses up to 256 exposures. The method of this invention ensures that each spatial region has the most optimal dynamic range. Specifically, an image is divided into multiple blocks, each with a different exposure value, ensuring that each region is exposed within the optimal range. This results in ultra-high-resolution video with a wider dynamic range, more optimal spatial exposure, clearer details, and reduced noise.

[0006] The present invention provides a method for enhancing the high dynamic range of an array camera by spatially independent exposure, and the specific steps are as follows.

[0007] (1) For each image processing chip, each independently controls the image sensor for exposure

[0008] Each image processing chip has 4 image processors. The image processors of each chip are spatially independent and statistically analyzed. The current chip makes a spatially independent automatic exposure recommendation value based on the histogram distribution statistics of the current area (the distribution statistics are realized by the camera's internal function) (the specific exposure method is not limited in this invention), and all the image processor exposure recommendation values ​​are transmitted between chips (the specific transmission method is not limited in this invention), and uniformly sent to the main core MCU (generally set to the camera in the upper left corner of the camera array) for overall decision-making. Then, the exposure values ​​of the 64 image processors are given to each chip through the broadcast mechanism. Each chip independently controls the image sensor for exposure. At this time, the image received by each image processor is spatially independent, which is completely inconsistent with the natural image seen by the human eye, and cannot be directly sent to the display, such as Figure 1 shown.

[0009] (2) Digital gain compensation for spatially independent exposure

[0010] The spatially independent exposure image is blocky and does not conform to the actual environment, such as up to 4N 2 Different exposures increase the overall dynamic range, but this cannot be displayed. The present invention solves this problem by performing spatially independent digital gain compensation on the exposure input of each image sensor, restoring the independently exposed images to the state of natural dynamic range distribution. The specific method is as follows:

[0011] First, calculate the digital gain of each channel image: first, 4N 2 (For example, 256) different exposure values ​​are sorted and the maximum exposure value Exposure.max is obtained. The digital gain Digtal_Gain of the area corresponding to this maximum exposure value is 1, and the other 4N 2 The digital gain of the -1 (255) block area is the maximum exposure value divided by the exposure value of the current area, and the formula is as follows:

[0012] Digital_Gain.Current=Exp.max / Exp.current, (1)

[0013] Get 4N 2 The digital gain of each of the (e.g. 256) channels, such as Figure 2 As shown;

[0014] Then, the digital gain of each channel is sent to the image processor of the corresponding exposure channel for gain processing (the gain processor is a digital multiplication calculation module on the pipeline in the camera), so 4N is obtained. 2 (For example, 256) types of exposure are independently compensated by digital gain, so that the image is restored to a linear space consistent with the natural environment.

[0015] (3) Independent spatial stretching and unified integration

[0016] At this point, all 256 different exposures have been restored to a linear space consistent with the natural environment through digital gain compensation, such as Figure 3 As shown, all exposures are stretched to a straight line. The specific calculation formula is explained above. Next, a unified fusion is performed. Here, we use a typical camera with three exposures: long exposure (L_exposure), medium exposure (M_exposure), and short exposure (S_exposure). These three exposures from the same image sensor are fused into a single result. The specific rules are as follows: Areas with no overexposure from the long exposure are prioritized using the stretched values ​​from the long exposure. Areas with overexposure from the long exposure but no overexposure from the medium exposure are limited to using the stretched values ​​from the medium exposure. Areas with overexposure from the medium exposure but no overexposure from the short exposure are using the stretched values ​​from the short exposure. This results in a single fused result for the three exposures output by the same image sensor. (There may be nonlinearity in areas where each exposure approaches overexposure, and interpolation can be used for transitions, but this is not a restriction here.) Furthermore, since the different image sensors are spatially independent, after fusion, no fusion is performed between them. The output is the spatially independently stretched and unified fused result. The fused image has an ultra-high dynamic range, consistent with what the human eye sees naturally, and boasts the best signal-to-noise ratio in each area.

[0017] The specific benefits of the present invention are: a spatially independent exposure high dynamic range enhancement method suitable for array cameras is proposed. Combined with a multi-chip communication mechanism, a wider dynamic range can be achieved in the entire system. Each spatial area obtains the most appropriate exposure, and the entire dynamic range maintains clearer details and lower noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1All image processors are spatially independent of exposure.

[0019] Figure 2 Spatially independent digital gain for all image processors.

[0020] Figure 3 All image processors are spatially independent and stretched and fused. DETAILED DESCRIPTION

[0021] The present invention proposes a spatial independent exposure high dynamic range enhancement method suitable for array cameras, such as Figure 1 As shown, the number of array cameras is 8x8, with a total of 64 cameras. Figure 1 The array on the left represents the real field brightness seen by each camera, covering a large dynamic range. The 64 cameras use spatially independent exposure, so that the area seen by each camera gets the most appropriate exposure. Figure 1 As shown in the array on the right, the exposure range of all image sensors may be very large, and each image sensor has its own exposure value. In this way, the target brightness is consistent, but this is completely incompatible with the real world that humans see, because the real world is continuous and contains a variety of rich brightness.

[0022] Next, set independent digital gain for each video stream through the corresponding image processor, such as Figure 2 As shown, the 64-channel image is restored to a continuous space, which requires the image processor to have a higher bit width, such as 28 bits, or even higher, so as to obtain a higher dynamic range of 168dB, where the exposure ratio of the maximum exposure to the minimum exposure is as high as 65536 times. Such a high dynamic range is impossible to achieve in traditional cameras (single image sensor generally has a maximum of 4 exposures). However, the array camera space independent exposure method proposed by the present invention can well cover each brightness range due to the 64-channel space independent exposure. Figure 3 The method shown here stretches the spatial range seen by each image processor to the longest exposure. Because each channel uses the most appropriate exposure, the stretching achieves the lowest digital gain for each channel, ultimately ensuring the best signal-to-noise ratio for objects across the entire dynamic range. In fact, each camera channel can also have four exposures, resulting in 4 x 64 = 256 different exposure values ​​within the entire image frame. This is also the physical advantage of array cameras.

Claims

1. A method for enhancing the high dynamic range of spatially independent exposures of an array camera, wherein: Each camera has a camera head, an image sensor, and an image processing chip; the array camera is set as an N×N array, with a total of N 2 cameras; each camera has 4 different exposures, of which the three typical ones are: long exposure, medium exposure, and short exposure; the other can be an exposure value between two adjacent exposure lengths; corresponding to the 4 different exposures, there are 4 exposure channels; an image processing chip includes 4 image processors, each corresponding to the 4 exposure channels; N×N array camera, there are 4 N 2 Type of exposure; characterized in that the specific steps are: (1) For each image processing chip, each independently controls the image sensor for exposure Each image processing chip has 4 image processors. The image processors of each chip are statistically independent in space. The current chip makes a spatially independent automatic exposure recommendation value based on the histogram distribution statistics of the current area. The exposure recommendation values ​​of all image processors are transmitted between chips and sent to the main core MCU in the upper left corner of the array for overall decision-making. Then, N 2 The exposure value of each image processing chip is broadcast to each chip in the array, and each chip independently controls the image sensor for exposure. At this time, the image received by each image processor is spatially independent and completely inconsistent with the natural image seen by the human eye. (2) Digital gain compensation for spatially independent exposure For each image sensor's exposure input, spatially independent digital gain compensation is performed to restore the independently exposed image to its natural dynamic range distribution. The specific approach is as follows: First, calculate the digital gain of each channel image, first 4N 2 Sort different exposure values ​​and get the maximum exposure value Exposure.max. The digital gain Digtal_Gain of the area corresponding to this maximum exposure value is 1, and the other 4N 2 The digital gain of the -1 block area is the maximum exposure value divided by the exposure value of the current area. The formula is as follows: Digital_Gain.Current=Exp.max / Exp.current, (1) Then, the digital gain of each channel is sent to the image processor of the corresponding exposure channel for gain processing to obtain 4N 2 Independent digital gain compensation for each exposure restores the image to a linear space consistent with the natural environment; (3) Independent spatial stretching and unified integration For cameras, the typical three exposures are long, medium, and short. The three exposures from the same image sensor are fused according to the following rules: areas without overexposure from the long exposure are given priority over the stretched value from the long exposure; areas with overexposure from the long exposure but not overexposed from the medium exposure are limited to using the stretched value from the medium exposure; and areas with overexposure from the medium exposure but not overexposed from the short exposure are given the stretched value from the short exposure. In this way, the three exposures output by the same image sensor are fused to produce one result.

Images