An ISP system for low-light image enhancement based on dark channel prior and bilateral filtering

Abstract

The application provides a low-illumination image enhancement ISP system based on dark channel prior and bilateral filtering, comprising a data preprocessing module, which performs format conversion and cross-clock domain processing on original image data and outputs preprocessed image data; a demosaicing module, which interpolates the preprocessed single-channel RAW format data into three-channel RGB image data; and a low-illumination enhancement module, which performs low-illumination enhancement processing on the RGB image data based on a dark channel prior algorithm and a bilateral filtering algorithm and outputs enhanced images. The application can improve image brightness and contrast in a low-illumination environment, while suppressing noise and halo artifacts, and realizes real-time and high-quality image enhancement processing.

Description

Technical Field

[0001] This invention relates to the field of image signal processing technology, and more specifically, to a low-light image enhancement ISP system based on dark channel prior and bilateral filtering. Background Technology

[0002] Low-light imaging has wide applications in security monitoring, autonomous driving, and industrial inspection. Existing image signal processing (ISP) systems typically employ enhancement methods based on histogram equalization or gamma correction, adjusting pixel grayscale distribution or mapping curves to improve image brightness. While simple to implement, these methods tend to amplify noise and lead to loss of detail. In recent years, multi-scale decomposition algorithms based on Retinex theory have achieved enhancement by separating illumination and reflection components, but their high computational complexity makes them unsuitable for real-time processing. Furthermore, deep learning-based low-light enhancement methods rely on large amounts of training data and high-performance computing platforms, resulting in high hardware deployment costs. Regarding transmittance estimation, while traditional dark channel prior algorithms can effectively recover scene depth information, their direct application to low-light images can lead to color distortion due to inaccurate atmospheric light value estimation, and the coarse estimation of the transmittance map can introduce halo artifacts and edge blurring. Bilateral filtering, as an edge-preserving denoising technique, can refine the transmittance map, but current hardware implementations often employ floating-point operations, resulting in high resource consumption and complex timing control, making them unsuitable for integration into programmable logic devices such as FPGAs. In the existing ISP pipeline architecture, the collaborative design of cross-clock domain data transmission, demosaicing interpolation, and enhancement modules lacks systematic optimization, resulting in low data throughput, frequent pipeline blockages, and difficulty in achieving high frame rate and low latency image enhancement processing.

[0003] In implementing the embodiments of the present invention, the prior art has at least the following problems or defects: existing low-light enhancement algorithms have high computational complexity and poor real-time performance when implemented in hardware; the direct application of dark channel priors to low-light scenes easily produces color distortion and halo artifacts; the floating-point operation implementation of bilateral filtering consumes a lot of resources and is difficult to integrate; and the insufficient cross-clock domain synchronization, demosaicing and enhancement module co-design in the ISP pipeline leads to low data throughput and pipeline blockage. Summary of the Invention

[0004] This invention provides a low-light image enhancement ISP system based on dark channel prior and bilateral filtering, comprising: The data preprocessing module is used to perform format conversion and cross-clock domain processing on the acquired raw image data to output preprocessed image data; The demosaic module is connected to the data preprocessing module and is used to interpolate the preprocessed image data from single-channel RAW format to three-channel RGB image data. The low-light enhancement module, connected to the de-mosaic module, is used to perform low-light enhancement processing on the RGB image data based on the dark channel prior algorithm and the bilateral filtering algorithm to output an enhanced image.

[0005] Furthermore, the data preprocessing module includes: A single-port SRAM control unit is used to perform differential read / write speed control on the input pixel data to resolve data read / write conflicts; The ping-pong operation unit, connected to the single-port SRAM control unit, is used to divide the even data storage space into two storage areas and alternately perform read and write operations. The cross-clock processing unit is connected to the single-port SRAM control unit and the ping-pong operation unit, respectively, and is used to synchronize the read data in the fast clock domain to the slow clock domain and generate the preprocessed image data that conforms to the ISP pipeline timing.

[0006] Furthermore, the cross-clock processing unit includes: The flag signal generation subunit is used to generate a flag signal with a pulse width greater than one slow clock cycle when the read data is valid in the fast clock domain. A two-stage synchronous flip-flop subunit is used to synchronize the flag signal in the fast clock domain to the slow clock domain to generate a slow clock domain flag signal. The data sampling subunit is used to sample the multi-bit readout data in the fast clock domain according to the slow clock domain flag signal in order to obtain stable preprocessed image data.

[0007] Furthermore, the demosaic module includes: A pixel matrix window building unit, used to construct a 5×5 pixel matrix window using four row-first-out buffers and multiple registers; The boundary completion unit is connected to the pixel matrix window construction unit and is used to fill in the missing elements of the matrix by copying adjacent pixels when the center pixel is located at the image boundary. The gradient calculation unit is used to calculate the Sobel gradient values ​​in the horizontal and vertical directions; A weight allocation unit, connected to the gradient calculation unit, is used to allocate horizontal and vertical weights based on the result of a multiple comparison of gradient values. A color difference and G-value calculation unit, connected to the weight allocation unit, is used to calculate the missing color component based on the horizontal weight and the vertical weight. The pipeline control unit is used to divide the calculation of the demosaic module into 8 pipeline stages to output the three-channel RGB image data.

[0008] Furthermore, the low-light enhancement module includes: The dark channel value calculation unit is used to perform an inversion operation on each pixel value of the RGB image data, and to obtain the dark channel value based on the minimum value of the inverted three-channel pixel values ​​and the minimum value of the 3×3 pixel matrix window.

[0009] Furthermore, the dark channel value calculation unit includes: The inverted subunit is used to perform an inversion operation on each pixel value of the RGB image data, so that the pixel value becomes the difference between it and the maximum gray value. The minimum value comparison subunit is used to compare the three-channel pixel values ​​after the inversion operation to obtain the minimum value of the three channels for each pixel. A matrix window construction sub-unit is used to construct a 3×3 pixel matrix window based on the first-in-first-out cache, and to take the minimum value of all three channels within the window again to obtain the dark channel value.

[0010] Furthermore, the low-light enhancement module also includes: An atmospheric light value calculation unit is used to calculate the atmospheric light value based on the dark channel value and the RGB image data using a frame-by-frame statistical method. The atmospheric light value calculation unit includes: The maximum value register subunit is used to compare the dark channel values ​​pixel by pixel and record the maximum value; The isolated point determination subunit is used to determine whether the current maximum value is an isolated white point based on the average value of the 8 points around the center pixel in the 3×3 matrix window corresponding to the dark channel value. An atmospheric light value determination subunit is used to take the average value of the three channels of the current pixel as the atmospheric light value in the case of a non-isolated point.

[0011] Furthermore, the low-light enhancement module also includes: The transmittance calculation unit is used to calculate an initial transmittance map based on the dark channel value, the atmospheric light value, and a preset sky region threshold. The transmittance calculation unit includes: The lookup table subunit is used to store the fixed-point value corresponding to the ratio of the transmittance coefficient to the atmospheric light value, which is calculated in advance based on the atmospheric light value, wherein the transmittance coefficient is a preset value; The multiplier subunit is used to multiply the dark channel value by the fixed-point value corresponding to the ratio output by the lookup table subunit to obtain the multiplication result; The sky region determination subunit is used to generate a threshold based on the product of the atmospheric light value and the preset alpha value, and compare the dark channel value with the threshold to determine whether the current pixel belongs to the sky region. The transmittance assignment subunit is used to determine, based on the judgment result, the transmittance of the non-sky region as the maximum value between a preset lower bound value and a subtraction of the multiplication result, or to use the fixed transmittance of the sky region as the initial transmittance and output the initial transmittance map.

[0012] Furthermore, the low-light enhancement module also includes: A bilateral filtering refinement unit is used to perform a bilateral filtering operation on the initial transmittance map to generate a refined transmittance map; The bilateral filter refinement unit includes: The spatial weight kernel lookup table sub-unit is used to store spatial weight coefficients based on a 3×3 window size and a preset spatial Gaussian standard deviation. The color weight kernel lookup table sub-unit is used to store color weight coefficients based on the pixel difference range and the preset color Gaussian standard deviation. A window construction subunit is used to construct a 3×3 pixel matrix of the initial transmittance map; The filter kernel calculation subunit is used to perform dot multiplication and summation of the spatial weight kernel and the color weight kernel to obtain the bilateral filter kernel and normalization parameters. The divider subunit is used to divide the dot product of the initial transmittance map and the bilateral filter kernel by the normalization parameter to obtain the refined transmittance map.

[0013] Furthermore, the low-light enhancement module also includes: The image enhancement unit is used to generate an enhanced image based on the atmospheric light value, the refined transmittance map, and the RGB image data, using an inverse operation of the atmospheric scattering model. The image enhancement unit includes: The difference calculation subunit is used to calculate the difference between the atmospheric light value and the RGB image data; The divider subunit is used to divide the difference by the maximum value of the refined transmittance map and the preset lower limit of transmittance to obtain the quotient. The enhanced image generation subunit is used to add the atmospheric light value to the quotient value, and then perform an inversion operation on the addition result to output the final enhanced image.

[0014] The embodiments of the present invention have at least the following beneficial effects: 1. This invention integrates the dark channel prior algorithm and the bilateral filtering algorithm into the ISP hardware pipeline. It uses a lookup table to store the fixed-point ratio of the pre-calculated transmittance coefficient to the atmospheric light value, and uses multipliers and dividers to perform fixed-point number operations. This avoids the high resource consumption and complex timing control problems caused by traditional floating-point operations, enabling the entire low-light enhancement module to be efficiently deployed in programmable logic devices such as FPGAs. This significantly reduces the hardware implementation cost and improves the computing efficiency, effectively solving the problems of difficult hardware implementation and high computational complexity of enhancement algorithms in the prior art.

[0015] 2. This invention adds an atmospheric light value calculation unit to the dark channel prior, which records the maximum value of the dark channel by frame-by-frame statistical method, and compares the mean of the neighborhood around the center pixel with the isolated point judgment subunit, effectively eliminating the interference of isolated white points and ensuring the accuracy of atmospheric light value estimation. At the same time, the transmittance calculation unit introduces a sky region judgment mechanism, which distinguishes between sky and non-sky regions based on the atmospheric light value and a preset threshold and adopts a differentiated transmittance assignment strategy. Combined with the bilateral filtering refinement unit, it uses spatial weight kernel and color weight kernel to perform edge-preserving filtering, which significantly suppresses color distortion, halo artifacts and edge blurring in low-light scenes, effectively solving the problem that the direct application of dark channel prior to low-light images in the prior art easily produces color distortion and halo artifacts.

[0016] 3. This invention achieves reliable data transmission from the fast clock domain to the slow clock domain through the collaborative design of the single-port SRAM control unit, ping-pong operation unit, and cross-clock processing unit in the data preprocessing module. It employs flag signal generation, two-stage synchronous trigger synchronization, and data sampling mechanisms. The de-mosaic module uses a 5×5 pixel matrix window construction, boundary padding, Sobel gradient calculation, weight allocation, and color difference interpolation, and is divided into 8-stage pipeline processing. Each sub-unit of the low-light enhancement module also adopts a pipeline architecture, enabling seamless data flow between modules of the entire ISP system, avoiding pipeline blockage, significantly improving data throughput and processing frame rate, and effectively solving the problems of low data throughput and pipeline blockage caused by insufficient cross-clock domain synchronization and lack of module collaborative design in the existing ISP pipeline. Attached Figure Description

[0017] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein: Figure 1 This is a schematic diagram of the overall hardware framework of a low-light enhancement ISP system according to an embodiment of the present invention; Figure 2 This is a hardware block diagram for data preprocessing provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the construction of a 5×5 pixel matrix window according to an embodiment of the present invention; Figure 4 This is a schematic diagram of matrix completion when the center pixel is at the boundary, according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the hardware structure for de-mosaicing provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a dark channel value calculation module provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an atmospheric light value calculation module provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of an initial transmittance calculation module provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a bilateral filter transmittance refinement module provided in an embodiment of the present invention. Detailed Implementation

[0018] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make the invention more thorough and complete, and to fully convey the scope of the invention to those skilled in the art.

[0019] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0020] It should be noted that the number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1As shown in this embodiment, a low-light image enhancement ISP system based on dark channel prior and bilateral filtering includes: a data preprocessing module, used to perform format conversion and cross-clock domain processing on the acquired raw image data to output preprocessed image data; a demosaic module, connected to the data preprocessing module, used to interpolate the preprocessed image data from single-channel RAW format to three-channel RGB image data; and a low-light enhancement module, connected to the demosaic module, used to perform low-light enhancement processing on the RGB image data based on the dark channel prior algorithm and the bilateral filtering algorithm to output an enhanced image.

[0022] The data preprocessing module includes: a single-port SRAM control unit, a ping-pong operation unit, and a cross-clock processing unit. The cross-clock processing unit further includes: a flag signal generation subunit, a two-stage synchronous flip-flop subunit, and a data sampling subunit.

[0023] Single-port SRAM control unit: This unit uses an asynchronous single-port SRAM with a data width of 16 bits and a depth 1.25 times the image width m (with reserved space for supplementation). Input signals: Clock CLKc (frequency 1.25 times the ISP clock CLKi), write data data_i[7:0] from the camera, write enable web (generated by the state machine), read enable (time-division multiplexed with write enable). Internally, there are write address register waddr[11:0] (bit width selected according to resolution) and read address register raddr[11:0]. Control state machine: Every 4 CLKc cycles, web is set low (write mode), and for 4 consecutive cycles, two 8-bit pixels are concatenated into 16 bits and written to the SRAM, while waddr increments; for the next 5 CLKc cycles, web is set high (read mode), and a read operation is performed every 5 cycles, reading 16 bits of data into the SRAM. raddr increases. This differential speed is used to achieve data caching.

[0024] Ping-Pong Operation Unit: Input Signal: Row Counter (From 0 to image height n-1), line effective low-level time (Configuration parameters), image width m. Internally, there are two base address registers: , (If the conflict condition is met, otherwise) +Normal space size). Address generation logic: When When the number of rows is even (even rows), write the address. ;when When it is an odd number (odd row), ,in After incrementing from 0 to m / 4-1, the address jumps to the supplementary space base address (base1+m / 4) and continues incrementing. The read address raddr uses similar logic, but the read speed is slower. The increment step of raddr is different from waddr, and it also jumps to the supplementary space when raddr reaches the upper limit of the normal space. By comparing the memory block flags of waddr and raddr, it is ensured that write and read operations are performed on different memory blocks at the same time.

[0025] Flag signal generation subunit: Input: Read completion pulse (One pulse is generated every 5 CLKc cycles). Output: Internally: A single D flip-flop is used to... Delay by one beat, and then ORing with the delayed signal yields a signal with a width of 2 CLKc cycles. .

[0026] Two-stage synchronous flip-flop subunit: Input: (Fast clock domain), slow clock CLKi. Output: Internally: Two cascaded D flip-flops, the first stage samples on the rising edge of CLKi. get Second-level sampling get . Synchronize with the slow clock domain and have a width of at least one CLKi cycle.

[0027] Data sampling subunit: such as Figure 2 As shown, input: [15:0] (SRAM read data), Slow clock CLKi. Output: Preprocessed image data. [7:0]. Internal: A 16-bit register. Its enable terminal is connected ,when Latch when =1 A modulo-4 counter (2 bits), incrementing each CLKi cycle. According to... The value, from The output will be sequentially as follows: Output when =0 [7:0] Output when =1 [15:8] When =2, output the next one. [7:0] (At this time) Valid again. (updated), and so on. Every 4 CLKi outputs 4 8-bit pixels. The demosaicing module includes: a pixel matrix window construction unit, a boundary padding unit, a gradient calculation unit, a weight assignment unit, a chrominance difference and G value calculation unit, and a pipeline control unit.

[0028] Pixel matrix window construction unit: Input: [7:0], pixel clock CLKi, row synchronization signal Hs. Output: 5×5 matrix elements m11~m55 (each 8 bits). Inside: Four FIFOs with a depth of m (each FIFO has a bit width of 8 bits, implemented using Block RAM or distributed RAM), and 25 8-bit registers. The write enable of the FIFO is the data valid signal, and the read enable is valid for each clock cycle (continuous read). Write to FIFO1 and the first stage of the register chain simultaneously. The output of FIFO1 is connected to the input of FIFO2, the output of FIFO2 is connected to the input of FIFO3, and the output of FIFO3 is connected to the input of FIFO4. The output of FIFO4, together with the outputs of FIFO3, FIFO2, FIFO1, and the current input, forms 5 rows of data. Each row of data is then shifted one column to the right every clock cycle through a 5-stage shift register (one 8-bit register per stage), resulting in a 5×5 matrix. This unit outputs a stable 5×5 window after a 5-clock cycle delay.

[0029] Boundary padding unit: Input: row counter value row (0~n-1), col (0~m-1), 5×5 matrix elements m11~m55. Output: Padded 5×5 matrix elements (all valid). Inside: A set of multiplexers that determine the position of the central pixel based on the values of row and col. When row = 0 and col = 0 (top left corner), copy the value of m33 to the missing positions such as m11, m22, m23, etc.; when row = 0 and col > 0 and col < m-1 (upper boundary non-corner), copy the value of the corresponding column in the second row to the first row; when row > 0 and col = 0 (left boundary), copy the value of the corresponding row in the second column to the first column; symmetrically handle other boundary cases. Implement using conditional statements and combinational logic, and the output of the selector is connected to the padded matrix bus. The combinational logic of this unit has a delay of approximately 2ns and can be completed within one pipeline stage.

[0030] Gradient calculation unit: Input: Padded 5×5 matrix. Output: Composite horizontal gradient [15:0], composite vertical gradient [15:0] (fixed-point number, with decimal places). Inside: First calculate the Sobel gradient of each 3×3 sub-region. For the sub-region centered at (i,j), the horizontal gradient , vertical gradient Similarly, iterating through i and j from 0 to 2 yields 9 results. and 9 Summation using an addition tree yields the result. and (Each 16 digits). Then calculate the average: = >>3 (right shift by 3 bits is equivalent to an approximation of dividing by 9). = >>3. Simultaneously calculate the horizontal and vertical gradient differences. =abs( - (16-bit). All adders are piped registers, and this unit occupies 2 stages of pipeline.

[0031] Weighting Unit: Input: [15:0] [15:0]. Output: Horizontal weights [3:0] (0~8), vertical weight [3:0]=8- Internally: First, and Shifting left by 1 bit yields double the value, shifting left by 2 bits yields quadrupled value, and adding them together yields tripled value. Comparison is performed using 6 comparators: ( <<2) and ;( <<1) and ;( )and( <<1), etc., generate flag signals compare1~compare6. For example: if ( <<2)< If the ratio is <0.25, then compare1=1; if ( <<1) < and( <<2)>= The ratio is 0.25~0.5, compare2=1; and so on. Based on the flag signal, a lookup table (6-to-1 multiplexer) is used to output the value. compare1-> =1, compare2-> =2, compare3-> =3, compare4-> =5, compare5-> 6, compare6-> =7, if and If the ratio is close to 1, then =4. =8- This unit is combinational logic and can be completed within a single pipeline.

[0032] As an optional hardware implementation, the Verilog hardware description language code for this unit is as follows: wire cond1 = (Gxmean << 2) < Gymean; wire cond2 = (Gxmean << 1) < Gymean && (Gxmean << 2) >= Gymean; wire cond3 = (Gxmean) < Gymean && (Gxmean << 1) >= Gymean; wire cond4 = (Gymean) < Gxmean && (Gymean << 1) >= Gxmean; wire cond5 = (Gymean << 1) < Gxmean && (Gymean << 2) >= Gxmean; wire cond6 = (Gymean << 2) < Gxmean; if (cond1) Wx = 4'd1; else if (cond2) Wx = 4'd2; else if (cond3) Wx = 4'd3; else if (cond4) Wx = 4'd5; else if (cond5) Wx = 4'd6; else if (cond6) Wx = 4'd7; else Wx = 4'd4; Color difference and G-value calculation unit: Input: 5×5 matrix, , The color type of the center pixel (determined by the lower two bits of the row and column counter). Output: Missing G, R, and B values ​​(8 bits each). Internally: Different interpolation formulas are selected based on the color type. Taking a B-pixel as the center as an example, the horizontal color difference operator... Vertical color difference operator The division is implemented using bit shifting: / 2 shifts the right by 1 bit, and / 4 shifts the right by 2 bits. Then the calculation is performed. (because + =8). Multiplier: It has 4 digits. It's 9 bits (because 8-bit addition might overflow), and the product is summed using a 13-bit adder. After obtaining G, then calculate... ,in and It is provided by a known G value (or an interpolated G). Similarly, a symmetric formula is used when the center is R or G pixels. All operations use integer arithmetic, intermediate results are temporarily stored in registers, and a 4-stage pipeline is allocated: Stage 1 calculation , The second stage performs a weighted summation to obtain G; the third stage calculates the color difference to recover another color; and the fourth stage outputs the final RGB value. Simultaneously, this unit also receives delayed horizontal and vertical sync signals, and outputs them after similar timing.

[0033] Production line control unit: such as Figure 5 As shown, this is an 8-stage pipeline controlling the entire de-mosaic module. Registers are inserted between each stage, generating corresponding enable signals. The input is the horizontal synchronization signal. The field synchronization signal Vs is obtained by passing it through an 8-stage shift register (with one clock cycle). , Align it with the output RGB data. The final output is 24-bit RGB data. ) and synchronization signals.

[0034] The dark channel value calculation unit includes: inverted sub-unit, minimum value comparison sub-unit, and matrix window construction sub-unit.

[0035] Inverted subunit: Input: Output: Internally: three 8-bit subtractors. Combinational logic output, no pipeline.

[0036] Minimum comparison subunit: Input: , , Output: [7:0]. Internal: First-stage comparator C1: Comparison and Output smaller value Second-stage comparator C2: Comparison and Output Each comparator is implemented using an 8-bit digital comparator and a multiplexer. This unit's combinational logic can be completed within a single pipeline.

[0037] like Figure 6 As shown, the matrix window constructs sub-cells: Input: [7:0], pixel clock, line synchronization signal Output: Dark channel value [7:0], and extended line synchronization signal Internally: It uses two FIFOs of depth m (each 8 bits wide) and nine 8-bit registers. Write to FIFO1, and connect the output of FIFO1 to the input of FIFO2. Currently... Together with the outputs of FIFO1 and FIFO2, they form three rows. Each row is shifted through three registers to obtain three columns, forming a 3×3 window. For the nine values ​​within the window, a three-stage comparator (similar to a minimum value comparison subunit) is used to find the minimum value, which is then output. Window operations require caching two lines, therefore Delayed by 2 lines from the input. For alignment, the line valid signal extension circuit uses a counter. right Pulse counting, when When the height is equal to the image height n, an additional [image height] is generated in the next line cycle. Pulse (width is 1 clock cycle), and with the original OR, generation Then By using a two-level register to time the data, we obtain the corresponding data. Synchronous This unit uses a 3-stage pipeline: stage 1 builds the window, stage 2 retrieves the minimum value of the window, and stage 3 outputs the value.

[0038] The atmospheric light value calculation unit includes: a maximum value register subunit, an isolated point judgment subunit, and an atmospheric light value determination subunit.

[0039] Maximum value register subunit: Input: [7:0], Field synchronization signal Clock. Output: Internally: an 8-bit register. The initial value is 0. Vs is reset on the rising edge at the start of each frame. =0. Each pixel clock will... and Compare; if it is greater than... Updated to Simultaneously output =1; otherwise =0. The comparator is an 8-bit digital comparator.

[0040] Isolated point judgment sub - unit: Input: , and its corresponding 3×3 window (obtained from the window construction sub - unit in the dark channel value calculation unit). Output: (valid flag). Inside: When = 1, read out the 8 dark channel values in the window except the center, sum them up with an adder tree, and then shift right by 3 bits to get the average value . The comparator judges < TH (TH is preset to 200), if less, it is regarded as an isolated point, and output = 0; otherwise = 1. The combinational logic of this unit can be completed within a single cycle.

[0041] As Figure 7 shown, the atmospheric light value determination sub - unit: Input: R, G, B, , , clock. Output: Atmospheric light value A[7:0]. Inside: When = 1 and = 1, calculate sum = R + G + B (three 8 - bit additions, and the sum is 9 - bit). Then calculate A=(sum341)>>10. Multiplier: 9 - bit × 9 - bit, the product is 18 - bit, and shift right by 10 bits to get an 8 - bit result. This A value is latched into a register and updated once per frame. This unit uses 1 - level pipeline (register the output after the multiplier).

[0042] The transmittance calculation unit includes: a look - up table sub - unit, a multiplier sub - unit, a sky region judgment sub - unit, and a transmittance assignment sub - unit. Preset parameters: , all converted to fixed - point numbers: . The fixed transmittance of the sky region = 9888 (corresponding to 0.30175×2 15 ).

[0043] Look - up table sub - unit: Implemented as a 256×16 - bit ROM, with address input A[7:0]. The ROM content is pre - calculated: Processed as Note: What actually needs to be stored is / A in fixed - point number, that is, ( ×2 15 ) / A. Since = ×2 15 , so LUT[a]= / a. But the division result requires 16 - bit fixed - point (1 - bit integer part, 15 - bit fractional part). When calculating, division can be used: LUT[a]=floor(( <<8) / a)>>8, etc. The specific generation is calculated by software. ROM output is 16-bit data. .

[0044] Multiplier subunit: Input: [7:0] [15:0]. Output: M[15:0]. Internal: 16-bit × 8-bit multiplier (e.g., using a DSP unit), product is 24 bits, take the high 16 bits as M, i.e., M = ( )>>8. Actually, M=( ( / A))>>8=( / A)>>8. However, to maintain the 16-bit fixed-point format, the final fractional part of M is 15 bits. The multiplier output is then registered.

[0045] Sky Region Judgment Subunit: Input: [7:0], A[7:0]. Output: Internally: Calculate threshold = ( A)>>15, that is The fixed-point number is ×A. Using a 16-bit × 8-bit multiplier, the product is right-shifted by 15 bits to obtain an 8-bit threshold. (Compare...) >threshold, if true =1, otherwise 0.

[0046] like Figure 8 As shown, the transmittance assignment subunit has the following input: M[15:0]. , =3277, =9888, =32768. Output: [15:0] (Initial transmittance). Internal: If =0, calculate temp= -M (16-bit subtraction), compares temp with Use a comparator and multiplexer to output the larger value; if =1, output This unit's combinational logic can be completed within a single cycle. The entire transmittance calculation unit uses a single-stage pipeline (multiplier followed by register, then other combinational logic outputs and registers).

[0047] The bilateral filter refinement unit includes: a spatial weight kernel lookup table subunit, a color weight kernel lookup table subunit, a window construction subunit, a filter kernel calculation subunit, and a divider subunit.

[0048] Spatial weight kernel lookup table sub-unit: stores nine 11-bit fixed-point numbers (10 decimal places). Pre-calculates the squared Euclidean distance based on a 3×3 window: (Corresponding position). .Pick =1.5, the calculation result is quantized to 11 bits. In hardware, these 9 values ​​are implemented as constant registers.

[0049] Color weight kernel lookup table subunit: 256×11-bit ROM, address is grayscale difference (0~255) ,Pick =0.2, quantized to 11 bits.

[0050] Window construction of sub-cells: Input: Initial transmittance [15:0], and the corresponding RGB image grayscale (RGB to grayscale). Output: Transmittance t11~t33 (16-bit) for a 3×3 window, and grayscale windows gray11~gray33 (8-bit). Internally: Uses two FIFOs of depth m (each 16-bit wide for transmittance, and two more FIFOs for grayscale, which can be reused), and nine 16-bit registers (transmittance) and nine 8-bit registers (grayscale). The construction method is the same as the window construction in the dark channel value calculation, requiring a two-row delay. This unit occupies a 2-stage pipeline.

[0051] Filter kernel calculation subunit: Input: 9 transmittance values ​​t[0:8] (one-dimensional array), 9 grayscale values ​​g[0:8], spatial weight coefficients s[0:8] (constant). Output: bilateral filter kernel elements. [0:8] (16-bit fixed-point, 15 decimal places), normalization parameter W (16-bit), summation (32-bit). Internal: First, calculate the absolute value of the difference between the center gray level g[4] and the surrounding 8 gray levels. i≠4. Each Use the address to look up the color weight ROM to obtain the color weight. For i≠4, (Because s and c are both 11-bit fixed points, their product is 22 bits, and shifting them 10 bits to the right restores the decimal to 15 bits). For the center point i=4, kernel[4]=s[4] (the color weight is 1, i.e. =1024, but to avoid multiplication, spatial weights are used directly. Then... (16-bit addition tree). It uses a 16-bit × 16-bit multiplier (reusable), multiplies by 32 bits, and sums the results to obtain a 32-bit result. This unit uses a 3-stage pipeline: the first stage is a table lookup. and calculation Level 2 calculation Summation of partial products; third-level summation yields W and .

[0052] Divider subunit: such as Figure 9 As shown, input: [31:0], W[15:0], row active signal Hsync. Output: Refined transmittance [15:0]. Internally: Uses the divider IP core provided by Quartus II, configured as a 32-bit divider by a 16-bit quotient, signed, with a pipeline delay of 2 cycles. The divider enable signal is generated from the delayed line sync signal: the line sync signal is passed through a 13-stage register (because approximately 13 pipeline stages are required from the initial transmittance input to the divider), resulting in... Delay by 2 levels to get ,Will and The phase OR is used as the input enable of the divider, ensuring that the divider starts during data validity. Divider output. This unit has a total of 7 pipeline stages (including 2 stages inside the divider IP).

[0053] The image enhancement unit includes: a difference calculation subunit, a divider subunit, and an enhanced image generation subunit.

[0054] Difference calculation subunit: Input: Atmospheric light value A[7:0], RGB data (delayed by two rows) [23:0]). Output: delta[15:0] (signed). Internally: Extends A to a 16-bit signed number (high 8 bits are 0, low 8 bits are A). Calculate each channel separately , , Implemented using a 16-bit subtractor.

[0055] Divider subunit: Input: delta[15:0], [15:0] =3277. Output: Q[15:0] (quotient). Internally: First calculate the divisor div=max( , The process involves using a 16-bit comparator and multiplexer. Then, another divider IP core (16-bit signed divider by 16-bit signed divider) outputs a 16-bit quotient. This divider has a 2-cycle delay. The enable signal is similarly generated by a delayed horizontal sync signal (the delay stage is adjusted based on the total number of pipeline stages, typically around 15). The divider outputs Q.

[0056] Enhanced Image Generation Subunit: Input: A[7:0], Q[15:0]. Output: Enhanced image L[7:0]. Internally: Calculate temp = A + Q (extend A to a 16-bit signed form, add it to Q; the result may exceed 0~255). Then perform saturation processing: if temp < 0, then temp = 0; if temp > 255, then temp = 255; otherwise, keep it unchanged. Finally, L = 255 - temp (8-bit subtraction). The same operation is performed on the R, G, and B channels respectively, and the final output is a 24-bit RGB enhanced image. This unit's combinational logic can be completed within a single pipeline stage.

[0057] With the detailed hardware description of each sub-unit, including signal bit width, operation type, pipeline stage number, memory configuration, parameter settings, etc., those skilled in the art can accurately implement the low-light image enhancement ISP system of this invention on an FPGA. All critical calculations avoid floating-point operations and complex division, using lookup tables, shifting, and fixed-point arithmetic, making it suitable for hardware implementation.

[0058] Furthermore, the storage medium in the embodiments of this application stores program instructions capable of implementing all the above methods. These program instructions can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0059] The above description is merely an explanation of some preferred embodiments of the present invention and the technical principles employed. Those skilled in the art should understand that the scope of the invention as described in the embodiments of the present invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A low-light image enhancement ISP system based on dark channel prior and bilateral filtering, characterized in that, include: The data preprocessing module is used to perform format conversion and cross-clock domain processing on the acquired raw image data to output preprocessed image data; The demosaic module is connected to the data preprocessing module and is used to interpolate the preprocessed image data from single-channel RAW format to three-channel RGB image data. The low-light enhancement module, connected to the de-mosaic module, is used to perform low-light enhancement processing on the RGB image data based on the dark channel prior algorithm and the bilateral filtering algorithm to output an enhanced image; The low-light enhancement module includes: The dark channel value calculation unit is used to perform an inversion operation on each pixel value of the RGB image data, and filter based on the minimum value of the inverted three-channel pixel values ​​and the minimum value of the 3×3 pixel matrix window to obtain the dark channel value; The low-light enhancement module also includes: An atmospheric light value calculation unit is used to calculate the atmospheric light value based on the dark channel value and the RGB image data using a frame-by-frame statistical method. The atmospheric light value calculation unit includes: The maximum value register subunit is used to compare the dark channel values ​​pixel by pixel and record the maximum value; The isolated point determination subunit is used to determine whether the current maximum value is an isolated white point based on the average value of the 8 points around the center pixel in the 3×3 matrix window corresponding to the dark channel value. An atmospheric light value determination subunit is used to take the average value of the three channels of the current pixel as the atmospheric light value in the case of a non-isolated point. The low-light enhancement module also includes: The transmittance calculation unit is used to calculate an initial transmittance map based on the dark channel value, the atmospheric light value, and a preset sky region threshold. The transmittance calculation unit includes: The lookup table subunit is used to store the fixed-point value corresponding to the ratio of the transmittance coefficient to the atmospheric light value, which is calculated in advance based on the atmospheric light value, wherein the transmittance coefficient is a preset value; The multiplier subunit is used to multiply the dark channel value by the fixed-point value corresponding to the ratio output by the lookup table subunit to obtain the multiplication result; The sky region determination subunit is used to generate a threshold based on the product of the atmospheric light value and the preset alpha value, and compare the dark channel value with the threshold to determine whether the current pixel belongs to the sky region. The transmittance assignment subunit is used to determine, based on the judgment result, the transmittance of the non-sky area as the maximum value between a preset lower bound value and a subtraction of the multiplication result, or to use the fixed transmittance of the sky area as the initial transmittance and output the initial transmittance map. The low-light enhancement module also includes: A bilateral filtering refinement unit is used to perform a bilateral filtering operation on the initial transmittance map to generate a refined transmittance map; The bilateral filter refinement unit includes: The spatial weight kernel lookup table sub-unit is used to store spatial weight coefficients based on a 3×3 window size and a preset spatial Gaussian standard deviation. The color weight kernel lookup table sub-unit is used to store color weight coefficients based on the pixel difference range and the preset color Gaussian standard deviation. A window construction subunit is used to construct a 3×3 pixel matrix of the initial transmittance map; The filter kernel calculation subunit is used to perform dot multiplication and summation of the spatial weight kernel and the color weight kernel to obtain the bilateral filter kernel and normalization parameters. The divider subunit is used to divide the dot product of the initial transmittance map and the bilateral filter kernel by the normalization parameter to obtain the refined transmittance map.

2. The low-light image enhancement ISP system based on dark channel prior and bilateral filtering according to claim 1, characterized in that, The data preprocessing module includes: A single-port SRAM control unit is used to perform differential read / write speed control on the input pixel data to resolve data read / write conflicts; The ping-pong operation unit, connected to the single-port SRAM control unit, is used to divide the even data storage space into two storage areas and alternately perform read and write operations. The cross-clock processing unit is connected to the single-port SRAM control unit and the ping-pong operation unit, respectively, and is used to synchronize the read data in the fast clock domain to the slow clock domain and generate the preprocessed image data that conforms to the ISP pipeline timing.

3. The low-light image enhancement ISP system based on dark channel prior and bilateral filtering according to claim 2, characterized in that, The cross-clock processing unit includes: The flag signal generation subunit is used to generate a flag signal with a pulse width greater than one slow clock cycle when the read data is valid in the fast clock domain. A two-stage synchronous flip-flop subunit is used to synchronize the flag signal in the fast clock domain to the slow clock domain to generate a slow clock domain flag signal. The data sampling subunit is used to sample the multi-bit readout data in the fast clock domain according to the slow clock domain flag signal in order to obtain stable preprocessed image data.

4. The low-light image enhancement ISP system based on dark channel prior and bilateral filtering according to claim 1, characterized in that, The demosaic module includes: A pixel matrix window building unit, used to construct a 5×5 pixel matrix window using four row-first-out buffers and multiple registers; The boundary completion unit is connected to the pixel matrix window construction unit and is used to fill in the missing elements of the matrix by copying adjacent pixels when the center pixel is located at the image boundary. The gradient calculation unit is used to calculate the Sobel gradient values ​​in the horizontal and vertical directions; A weight allocation unit, connected to the gradient calculation unit, is used to allocate horizontal and vertical weights based on the result of a multiple comparison of gradient values. A color difference and G-value calculation unit, connected to the weight allocation unit, is used to calculate the missing color component based on the horizontal weight and the vertical weight. The pipeline control unit is used to divide the calculation of the demosaic module into 8 pipeline stages to output the three-channel RGB image data.

5. The low-light image enhancement ISP system based on dark channel prior and bilateral filtering according to claim 1, characterized in that, The dark channel value calculation unit includes: The inverted subunit is used to perform an inversion operation on each pixel value of the RGB image data, so that the pixel value becomes the difference between it and the maximum gray value. The minimum value comparison subunit is used to compare the three-channel pixel values ​​after the inversion operation to obtain the minimum value of the three channels for each pixel. A matrix window construction sub-unit is used to construct a 3×3 pixel matrix window based on the first-in-first-out cache, and to take the minimum value of all three channels within the window again to obtain the dark channel value.

6. The low-light image enhancement ISP system based on dark channel prior and bilateral filtering according to claim 1, characterized in that, The low-light enhancement module also includes: The image enhancement unit is used to generate an enhanced image based on the atmospheric light value, the refined transmittance map, and the RGB image data, using an inverse operation of the atmospheric scattering model. The image enhancement unit includes: The difference calculation subunit is used to calculate the difference between the atmospheric light value and the RGB image data; The divider subunit is used to divide the difference by the maximum value of the refined transmittance map and the preset lower limit of transmittance to obtain the quotient. The enhanced image generation subunit is used to add the atmospheric light value to the quotient value, and then perform an inversion operation on the addition result to output the final enhanced image.

Images