A Real-Time HDR Image Processing System Based on FPGA for Multi-Exposure Images

Abstract

This invention discloses a real-time multi-exposure HDR image processing system based on FPGA, comprising: a DDR3 image storage control module, integrated within the FPGA, communicating with an external DDR3 chip via the AXI4 bus protocol, supporting high-speed storage and parallel reading of 8-channel multi-exposure images, and outputting synchronized multi-exposure image data and valid frame, line, and data signals; a multi-exposure HDR fusion module, connected to the output of the DDR3 image storage control module, employing a 6-stage hardware pipeline architecture to achieve pixel-level weighted fusion of 8-channel multi-exposure images, outputting a 24-bit RGB format HDR fusion image; and a Reinhard tone mapping module, connected to the output of the multi-exposure HDR fusion module, implementing dynamic range compression from HDR to LDR images based on an optimized Reinhard algorithm, outputting a 24-bit RGB format LDR image that meets the requirements of display devices. This invention achieves real-time processing of multi-exposure HDR images throughout the entire process through the integrated and synchronized optimization of these three modules.

Description

Technical Field

[0001] This invention belongs to the field of high-definition image processing technology, and specifically relates to a real-time multi-exposure HDR image processing system based on FPGA. Background Technology

[0002] High dynamic range (HDR) images can capture a wide range of brightness information from shadows to highlights in a real scene, with a dynamic range of up to 10. 5 1. Far exceeding traditional low dynamic range (LDR) images (dynamic range approximately 100:1). The traditional HDR processing workflow typically consists of four stages: "multi-exposure image acquisition - computer storage - software fusion - software tone mapping," which has the following technical drawbacks: 1. High storage latency: The data volume of multi-exposure images (8 channels) reaches 8×1280×720×3Byte=20.7MB / frame. The storage bandwidth under the traditional CPU / GPU architecture is insufficient, resulting in an image reading latency of more than 10ms, which cannot meet the requirements of real-time processing. For example, Chinese patent CN115830422A discloses an image processing method, apparatus, and electronic device. This invention sequentially and successively fuses multiple frames of images with an intermediate HDR image. This serial fusion architecture means that when processing the Nth frame, it is necessary to wait for the previous N-1 frames to complete the fusion, which cannot achieve parallel processing of multiple data streams and is difficult to meet the requirements of high frame rate real-time processing. 2. Low fusion accuracy: Software fusion algorithms mostly use simple averaging or fixed weights, without considering the differences in detail distribution of images with different exposures. Highlight areas are prone to overexposure, and dark areas are prone to loss of detail. The signal-to-noise ratio of the fused image is less than 30dB. 3. High synchronization difficulty: Multi-exposure image reading, HDR fusion, and tone mapping belong to different hardware units. The synchronization error of the control signal exceeds 5 clock cycles, resulting in line misalignment and frame tearing in the output image. 4. High resource consumption: Traditional architectures require separate configuration of storage chips, fusion processors, and mapping processors. The hardware is large and consumes a lot of power, which is not suitable for embedded scenarios (such as automotive and portable devices).

[0003] FPGAs offer advantages such as high hardware parallelism, predictable latency, and configurable resources, making them a preferred platform for solving real-time HDR processing. However, existing FPGA-based HDR solutions often focus on a single module (such as only implementing fusion or only implementing mapping), lacking an integrated "storage-fusion-mapping" design, and still suffer from problems such as difficulty in synchronizing between modules and fragmented processing flows. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time processing system for multi-exposure HDR images based on FPGA, so as to solve the technical problems of high storage latency, low fusion accuracy and difficulty in tone mapping synchronization in the traditional HDR processing flow. Through the integration and synchronization optimization of three modules, the system realizes real-time processing of multi-exposure HDR images throughout the entire process.

[0005] The technical solution of the present invention is as follows: On one hand, the present invention provides a real-time multi-exposure HDR image processing system based on FPGA, comprising: The DDR3 image storage control module is integrated inside the FPGA and communicates with the external DDR3 chip using the AXI4 bus protocol. It includes a write FIFO buffer submodule, a read FIFO buffer submodule, and a dual-state machine read / write control submodule. It supports high-speed storage and parallel reading of 8-channel multi-exposure images and outputs synchronized multi-exposure image data and frame, line, and data valid signals. The multi-exposure HDR fusion module is connected to the output of the DDR3 image storage control module. It adopts a 6-level hardware pipeline architecture, including an input parsing submodule, an RGB to grayscale submodule, a pixel mean and variance calculation submodule, a basic weight allocation submodule, an overexposure and underexposure weight adjustment submodule, and a weighted summation submodule. It realizes pixel-level weighted fusion of 8 multi-exposure images and outputs a 24-bit RGB format HDR fusion image. The Reinhard tone mapping module, connected to the output of the multi-exposure HDR fusion module, includes a synchronization signal processing submodule, a frame-level luminance statistics submodule, a 6-level delay chain mapping pipeline submodule, and an output synchronization submodule. Based on the optimized Reinhard algorithm, it realizes dynamic range compression from HDR image to LDR image and outputs a 24-bit RGB format LDR image that meets the requirements of the display device.

[0006] Preferably, the dual-state machine read / write control submodule of the DDR3 image storage control module includes a write state machine and a read state machine: The write state machine includes six states: IDLE, write configuration, write enable, FIFO data read, burst write count, and write complete. It supports allocating storage addresses for eight multi-exposure images, HDR fusion images, and LDR mapping images according to exposure. It adopts a fixed burst length transmission method, and the corresponding write FIFO cache submodule meets a storage depth of not less than a specified threshold to ensure that there is no packet loss in the eight multi-exposure images. The read state machine includes five states: IDLE, address switching, read enable, burst read count, and read complete. It supports parallel reading of eight multi-exposure images. The read FIFO buffer submodule is set with a "near full" threshold to trigger DDR3 read pause and prevent the read FIFO buffer submodule from overflowing. The output terminal is set with a channel flag to realize channel identification and synchronous output of eight multi-exposure image data.

[0007] Preferably, the functions of each level in the 6-level hardware pipeline architecture of the multi-exposure HDR fusion module are as follows: The first-level input parsing submodule is used to extract the 8-channel multi-exposure image RGB channel data output by the DDR3 module, buffer it into the pipeline register, and transmit frame, line, and data valid signals. The second-level RGB to grayscale submodule is used to optimize the multiplication operation of the grayscale conversion formula Y=a*R+b*G+c*4B based on the selected brightness conversion formula, where Y is the grayscale value, R, G, and B are the blue, red, and blue channel components, respectively, and a, b, and c are the weight coefficients corresponding to each component. After multiplying each weight coefficient by 256, the multiplication operation of the grayscale conversion formula is optimized by shifting and adding. Finally, the result is shifted 8 bits to the right to output an 8-bit grayscale value. The third-level pixel mean and variance calculation submodule is used to calculate the mean and standard deviation of the three RGB channels of each image, and uses Newton's iteration method to implement the square root operation. The fourth-level basic weight allocation submodule is used to prioritize the bright parts in the middle of the gray value. When the pixel gray value is <127, the weight = gray value + d, where d is a fixed constant. When the gray value is ≥127, the weight = 256 - gray value. The output is an 8-bit basic weight. The Level 5 overexposure and underexposure weight adjustment submodule is used to adjust the basic weights based on preset overexposure and underexposure thresholds, reduce the weight of overexposed images, reduce the weight of underexposed images, and increase the weight of intermediate exposure images, and calculate the total weight. The sixth-level weighted summation submodule is used to normalize the weights through the division IP core. The RGB channels of the eight multi-exposure images are multiplied by the normalized weights and then summed to output a 24-bit RGB format HDR fusion image.

[0008] Preferably, the frame-level luminance statistics submodule of the Reinhard tone mapping module adopts a 5-state machine design: IDLE state: Waiting for the rising edge of the HDR fused image frame start signal; FSTART state: Waiting for the rising edge of the row start signal, preparing to enter row statistics; LINE Status: When data is valid, accumulate the pixel grayscale value (Gray) and count the total number of pixels; CALC status: Calculate the frame average brightness mean_value and white point brightness Lwhite; READY state: Outputs a statistics completion flag, keeps mean_value and Lwhite stable, and waits for the next frame of statistics to be used for calculation; The 6-stage delay chain mapping pipeline submodule implements the optimized Reinhard algorithm based on fixed-point arithmetic, expressed by the formula: L_scaled=(0.3×L) / mean_value, where L is the original pixel brightness; L_out=(L_scaled×(1+L_scaled / Lwhite²)) / (1+L_scaled), where L_out is the brightness value after nonlinear mapping of the original brightness in the optimized Reinhard algorithm; Ratio=L_out / Gray, where Ratio is the proportion of brightness after mapping corresponding to each unit of original grayscale, and the three channels maintain color balance; R_out=In_pixel_r*Ratio, where In_pixel_r is the original red channel value of the pixel after HDR fusion, and R_out is the final displayed red component; the calculation of the R, G, and B channels is similar, with each stage delayed by 1 clock cycle to ensure alignment with the input synchronization signal.

[0009] Preferably, a channel synchronization submodule is set between the DDR3 image storage control module and the multi-exposure HDR fusion module, including an 8-channel flag register and a channel address mapping table. The 8-channel flag register corresponds to 8 multi-exposure images. When the DDR3 image storage control module reads the address in the storage range of the corresponding exposure image, the corresponding channel flag bit at the output of the read state machine is pulled high. The multi-exposure HDR fusion module identifies images with different exposure levels based on the channel flag and achieves pixel-level alignment and fusion.

[0010] Preferably, the weight adjustment submodule of the multi-exposure HDR fusion module adopts a smooth step function to optimize the transition: when the pixel grayscale is within the set ratio transition zone of the overexposure threshold or underexposure threshold, the weight is smoothly adjusted according to the smooth step function to avoid artifacts in the fused image caused by sudden weight changes.

[0011] Preferably, the output synchronization submodule of the Reinhard tone mapping module includes three sets of six-level delay chains, which delay the frame valid signal, line valid signal, and data valid signal respectively, so that the control signal of the output LDR image is strictly consistent with the delay of the mapped pixel data.

[0012] On the other hand, the present invention provides a real-time processing method for multi-exposure HDR images based on FPGA, comprising the following steps: S1: The DDR3 image storage control module caches 8 multi-exposure images through the write FIFO cache submodule, allocates storage addresses according to the exposure of the write state machine, and uses AXI4 bus burst write operation to write to the external DDR3 chip. S2: The read state machine is based on the channel flag. It reads 8 multi-exposure images in parallel through the read FIFO buffer submodule and outputs the synchronized image data and frame, line and data valid signals. S3: The multi-exposure HDR fusion module completes input analysis, RGB to grayscale conversion, pixel mean and variance calculation, basic weight allocation, overexposure and underexposure weight adjustment and weighted summation through a 6-stage pipeline, and outputs a 24-bit RGB format HDR fusion image. S4: The Reinhard tone mapping module first calculates the average brightness and white point brightness of the frame through frame-level brightness statistics, and then achieves dynamic range compression through a 6-level mapping pipeline to output a 24-bit RGB format LDR image that meets display requirements. S5: The output synchronization submodule performs 6-level delay alignment on the LDR image control signal to ensure that the data and control signal are transmitted synchronously.

[0013] Preferably, in step S3, the weight allocation adopts the strategy of "prioritizing intermediate exposure and optimizing detail preservation": the exposure difference of the 8 multi-exposure images is divided into three levels, namely low exposure level, intermediate exposure level and high exposure level. Each level corresponds to a preset upper limit threshold of weight. The upper limit threshold of weight for intermediate exposure level is higher than that for high exposure level and low exposure level, so as to ensure that the details in the highlight area come from the low exposure image and the details in the shadow area come from the high exposure image.

[0014] Preferably, in step S4, the division operation is implemented through the FPGA's built-in division IP core, and the multiplication operation is optimized through shifting and adding, thus avoiding the precision loss and delay caused by floating-point operations.

[0015] Compared with the prior art, the present invention has the following technical effects: 1. Strong real-time performance: The three modules in the system described in this invention are based on an FPGA hardware parallel architecture, which reduces the number of clock cycles for overall processing latency under a unified clock, supports real-time processing of 1280×720 resolution images (frame rate ≥60fps), and meets the requirements for real-time display.

[0016] 2. High fusion accuracy: This invention adopts the strategy of "mean / variance weight allocation + overexposure / underexposure adjustment", with a detail retention rate of ≥92% in the highlight area, a signal-to-noise ratio of ≥35dB in the dark area, and a peak signal-to-noise ratio (PSNR) of ≥40dB in the fused image.

[0017] 3. Good synchronization stability: In the system described in this invention, the DDR3 image storage control module and the multi-exposure HDR fusion module are aligned through the CH_flag channel flag, and the multi-exposure HDR fusion module and the Reinhard tone mapping module are synchronized through a 6-level delay chain. The control signal synchronization error is ≤1 clock cycle, with no frame tearing or line misalignment.

[0018] 4. Low resource consumption: The three modules in the system described in this invention are integrated into a single FPGA chip, with a logic resource utilization rate of ≤67%, a BlockRAM utilization rate of ≤70%, and a power consumption of ≤5W, making it suitable for embedded scenarios.

[0019] 5. Wide compatibility: This invention supports 8-channel multi-exposure image input (expandable to 16 channels), outputs LDR images to adapt to mainstream display devices (such as HDMI, LVDS), and can flexibly adjust the resolution (1280×720~3840×2160). Attached Figure Description

[0020] Figure 1 This is the overall architecture diagram of a multi-exposure HDR image real-time processing system based on FPGA; Figure 2 This is the overall flowchart of a real-time multi-exposure HDR image processing method based on FPGA. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present application and with reference to the accompanying drawings.

[0022] Example 1 This embodiment provides a multi-exposure HDR image real-time processing system based on FPGA, suitable for real-time image processing scenarios requiring high dynamic range and high color fidelity, such as security monitoring, vehicle surround view, professional cameras, and high-definition displays. (See also...) Figure 1 As shown, it includes: The DDR3 image storage control module is integrated inside the FPGA and communicates with external DDR3 chips using the AXI4 bus protocol. It includes a write FIFO buffer submodule, a read FIFO buffer submodule, and a dual-state machine read / write control submodule. It supports high-speed storage and parallel reading of 8-channel multi-exposure images and outputs synchronized multi-exposure image data and frame, line, and data valid signals, solving the problems of large data volume and high read latency of multi-exposure images.

[0023] The multi-exposure HDR fusion module is connected to the output of the DDR3 image storage control module. It adopts a 6-stage hardware pipeline architecture with a delay of 1 clock cycle for each stage to ensure real-time performance. It includes an input parsing submodule, an RGB-to-grayscale submodule, a pixel mean and variance calculation submodule, a basic weight allocation submodule, an overexposure and underexposure weight adjustment submodule, and a weighted summation submodule. It realizes pixel-level weighted fusion of 8 multi-exposure images, preserves the dark details of the high-exposure image and the highlight details of the low-exposure image, and outputs a 24-bit RGB format HDR fusion image.

[0024] The Reinhard tone mapping module, connected to the output of the multi-exposure HDR fusion module, includes a synchronization signal processing submodule, a frame-level brightness statistics submodule, a 6-level delay chain mapping pipeline submodule, and an output synchronization submodule. Based on the optimized Reinhard algorithm, it realizes dynamic range compression from HDR image to LDR image, and outputs a 24-bit RGB format LDR image that meets the requirements of LDR display devices (dynamic range 100:1), ensuring uniform brightness and color fidelity in the output image.

[0025] The DDR3 image storage control module, multi-exposure HDR fusion module, and Reinhard tone mapping module are integrated based on Xilinx Artix-7 series FPGA (or equivalent FPGA chip). They are designed based on the same clock domain of the FPGA (e.g., 125MHz). The data and control signals of the three modules are strictly aligned through delay chain synchronization, and the overall processing delay is 6 clock cycles (48ns).

[0026] In a preferred embodiment of this invention, the dual-state machine read / write control submodule of the DDR3 image storage control module includes a write state machine (DDR3_WR_STATE) and a read state machine (DDR3_RD_STATE): The write state machine includes six states: IDLE, write configuration, write enable, FIFO data read, burst write count, and write complete. It supports allocating storage addresses for eight multi-exposure images, HDR fusion images, and LDR mapping images according to exposure (e.g., address range 0~180000 stores eight multi-exposure images, and address range 220000~240000 stores HDR fusion images and LDR mapping images respectively). It adopts a fixed burst length (e.g., 64 burst length, with each burst transmitting 256 bytes of data). The corresponding write FIFO buffer submodule meets a storage depth of not less than a specified threshold (e.g., ≥1024) to ensure that there is no packet loss buffering for the eight multi-exposure images. The read state machine includes five states: IDLE, address switching, read enable, burst read count, and read complete. It supports parallel reading of eight multi-exposure images. The read FIFO buffer submodule is set with a "near full" threshold to trigger DDR3 read pause and prevent the read FIFO buffer submodule from overflowing. The output terminal is set with CH_flag channel flag to realize channel identification and synchronous output of eight multi-exposure image data.

[0027] In a preferred embodiment of this invention, the functions of each level in the 6-level hardware pipeline architecture of the multi-exposure HDR fusion module are as follows: The first-level input parsing submodule is used to extract the 8-channel multi-exposure image RGB channel data output by the DDR3 module, buffer it into the pipeline register, and transmit frame, line, and data validity signals.

[0028] The second-level RGB-to-grayscale submodule is used to convert the brightness based on the selected brightness conversion formula Y=a*R+b*G+c*4B, where Y is the grayscale value, R, G, and B are the blue, red, and blue channel components, respectively, and a, b, and c are the weight coefficients corresponding to each component. The brightness conversion formula can preferably be the BT.601 grayscale formula Y=0.299R+0.587G+0.114B. After multiplying each weight coefficient by 256, the multiplication operation of the grayscale conversion formula is optimized by shifting and adding (77=64+8+4+1, 150=128+16+4+2, 29=16+8+4+1). Finally, the result is shifted 8 bits to the right to output an 8-bit grayscale value.

[0029] The third-level pixel mean and variance calculation submodule is used to calculate the mean ((R+G+B)×85>>8, 85≈256 / 3) and standard deviation (sqrt((ΔR²+ΔG²+ΔB²) / 3)) of the RGB three channels of each image, using Newton's iteration method to implement the square root operation. The mean reflects the overall brightness level of the image. Since the brightness baseline of images with different exposures varies greatly, the mean can be used to normalize the brightness of each image, avoiding overly bright or dark images due to brightness shifts during weight allocation. The standard deviation reflects the dispersion of pixel values. Flat areas with small standard deviations can smoothly transition between multi-exposure images; detailed areas with large standard deviations should be prioritized for higher weights from normally exposed images. The goal of weight allocation is to give higher weights to pixels with "normal exposure, high detail, and low noise," with the mean and standard deviation being the most critical quantification criteria.

[0030] The fourth-level basic weight allocation submodule is used to prioritize the brightest part in the middle of the gray value. When the pixel gray value is <127, the weight = gray value + d, where d is a fixed constant, such as 1; when the gray value is ≥127, the weight = 256 - gray value, and the 8-bit basic weight is output.

[0031] The Level 5 overexposure and underexposure weight adjustment submodule is used to adjust the basic weights based on preset overexposure thresholds (e.g., >0.8) and underexposure thresholds (e.g., <0.1), reducing the weight of overexposed images, reducing the weight of underexposed images, and increasing the weight of intermediate exposure images, and calculating the total weight.

[0032] The sixth-level weighted summation submodule is used to normalize the weights through the division IP core. The RGB channels of the eight multi-exposure images are multiplied by the normalized weights and then summed to output a 24-bit RGB format HDR fusion image.

[0033] In a preferred embodiment of this invention, the frame-level luminance statistics submodule of the Reinhard tone mapping module adopts a 5-state machine design: IDLE state: Waiting for the rising edge of the HDR fused image frame start signal.

[0034] FSTART state: Waiting for the rising edge of the row start signal, preparing to enter row statistics.

[0035] LINE Status: When the data is valid, accumulate the pixel grayscale value (Gray) (e.g., based on the BT.601 grayscale formula) and count the total number of pixels.

[0036] CALC status: Calculate the frame average brightness mean_value (mean_value = total image brightness / total number of pixels) and white point brightness Lwhite (Lwhite = mean_value × fixed value, preferably 0.95).

[0037] READY state: Outputs a statistics completion flag (map_ready_flag), keeps mean_value and Lwhite stable, and waits for the next frame of statistics to be used for calculation.

[0038] The 6-stage delay chain mapping pipeline submodule implements an optimized Reinhard algorithm based on fixed-point arithmetic, expressed by the formula: L_scaled = (0.3 × L) / mean_value, where L is the original pixel brightness, mean_value is the average brightness of the frame image, L_out = (L_scaled × (1 + L_scaled / Lwhite²)) / (1 + L_scaled), where L_out is the brightness value after nonlinear mapping of the original brightness in the optimized Reinhard algorithm, Lwhite is the white point brightness value of the entire frame image, i.e., the brightness of the brightest pixel in the frame, Ratio = L_out / Gray, where Ratio is the proportion of brightness after mapping corresponding to each unit of original grayscale, and Gray is the original grayscale value of the pixel; the three channels maintain color balance, R_out = In_pixel_r * Ratio, where In_pixel_r is the original red channel value of the pixel after HDR fusion, and R_out is the final displayed red component; the calculation of the R, G, and B channels is similar, with each stage delayed by 1 clock cycle to ensure alignment with the input synchronization signal.

[0039] As a preferred embodiment of this example, a channel synchronization submodule is set between the DDR3 image storage control module and the multi-exposure HDR fusion module, including an 8-channel flag register (CH_flag[7:0]) and a channel address mapping table. The 8-channel flag registers correspond to 8 multi-exposure images respectively. When the read address of the DDR3 image storage control module is in the storage range of the corresponding exposure image, the corresponding channel flag CH_flag bit is pulled high at the output of the read state machine. The multi-exposure HDR fusion module identifies images with different exposure levels based on the channel flag CH_flag and realizes pixel-level alignment fusion.

[0040] As a preferred embodiment of this example, the weight adjustment submodule of the multi-exposure HDR fusion module uses a smooth step function (smooth_step) to optimize the transition: when the pixel grayscale is within a set ratio transition zone of the overexposure threshold or underexposure threshold (e.g., set to the overexposure / underexposure threshold ±10% transition zone), the weight is smoothly adjusted according to the smooth step function (x³×(x×(x×6-15)+10)) to avoid artifacts in the fused image caused by sudden weight changes.

[0041] As a preferred embodiment of this example, the output synchronization submodule of the Reinhard tone mapping module includes three sets of six-level delay chains, which delay the frame valid signal, line valid signal, and data valid signal respectively, so that the control signal of the output LDR image is strictly consistent with the delay of the mapped pixel data.

[0042] To verify the effectiveness and superiority of the method provided in this embodiment, some specific examples are provided below: Example 1 (System Hardware Configuration): This example uses a Xilinx Artix-7 series XC7A100T-FFG676 FPGA chip, externally configured with a 2GB DDR3 chip (NT5CC256M16EP-EK), a multi-exposure image input interface of MIPICSI-2 (4 channels, 1.5Gbps per channel), and an LDR image output interface of HDMI (supporting 4K@60fps). The clock configuration is as follows: 1. FPGA system clock: 125MHz (generated by external crystal oscillator + PLL); 2. DDR3 working clock: 400MHz (AXI4 interface clock, divided by 1:4 to 100MHz user clock); 3. Image input clock: 74.25MHz (MIPICSI-2 decoded clock); 4. Image output clock: 371.25MHz (HDMI encoding clock).

[0043] Verification of DDR3 image storage control module: 1. Input 8 channels of 1280×720 resolution 24-bit RGB images (60fps per channel), write FIFO cache depth of 1024, no data loss; 2. Read FIFO cache depth of 4096, "near full" threshold (rdusedw>4095), trigger read pause and resume normal operation, no overflow; 3. Address allocation test: Exp0~Exp7 are stored in eight different address segments respectively, read address error ≤1Byte.

[0044] Verification of the multi-exposure HDR fusion module: 1. Input 8 different exposure images (exposure time 10ms~100μs), the fused image has no overexposure in the highlight area (brightness>200) and the details in the dark area (brightness<50) are discernible; 2. Pipeline delay test: the single pixel processing delay is 6 clock cycles (48ns), which meets the design requirements; 3. Weight allocation test: the weight ratio of the intermediate exposure image (Exp4, Exp5) is ≥30%, and the weight ratio of the overexposed (Exp7) and underexposed (Exp0) images is ≤15%.

[0045] Reinhard tone mapping module verification: 1. Frame-level statistics test: mean_value calculation error ≤2%, Lwhite calculation error ≤3%; 2. Dynamic range compression test: input HDR image dynamic range 10 4 1. Output LDR image dynamic range 100:1, compression ratio 100:1; 3. Color fidelity test: the color difference between the output image and the original scene is CIE1976 color difference ΔE<3, which meets the requirements of high-definition display.

[0046] The overall system performance test results are as follows: 1. Frame rate: 60fps at 1280×720 resolution, 15fps at 3840×2160 resolution; 2. Power consumption: FPGA core power consumption 4.2W, DDR3 power consumption 1.8W, total power consumption ≤6W; 3. Stability: After 72 hours of continuous operation, there was no data packet loss, frame tearing, or synchronization misalignment; 4. Environmental adaptability: No performance degradation at operating temperatures of -40℃ to 85℃ and humidity of 10% to 90%.

[0047] Example 2 Accordingly, this embodiment provides a real-time multi-exposure HDR image processing method based on FPGA, implemented based on the system described in any embodiment of the present invention, such as... Figure 2 As shown, it includes the following steps: S1: Multi-exposure image storage. The DDR3 image storage control module caches 8 multi-exposure images (each with a 1280×720 resolution and 24-bit RGB format) through the write FIFO cache submodule. The write state machine allocates storage addresses according to the exposure (Exp0~Exp7 correspond to addresses 0~180000). It uses AXI4 bus burst write operations to write to the external DDR3 chip, with a storage rate of 125M pixels / second.

[0048] S2: Synchronous reading of multi-exposure images. The read state machine is based on the CH_flag channel flag. It reads 8 multi-exposure images in parallel through the read FIFO buffer submodule (set to 1 when the address is in the corresponding range), and outputs the synchronized image data and frame, line and data valid signals.

[0049] S3: HDR pixel-level fusion. The multi-exposure HDR fusion module completes input analysis through a 6-stage pipeline, including RGB to grayscale conversion, pixel mean and variance calculation, basic weight allocation, overexposure and underexposure weight adjustment, and weighted summation. Stages 1-2: Parse image data and convert to grayscale values; Stage 3: Calculate mean and standard deviation to determine pixel brightness distribution; Stages 4-5: Assign and adjust weights, reducing the weight of overexposure / underexposure images; Stage 6: Weighted summation outputs the HDR fused image; Outputs a 24-bit RGB format HDR fused image with a fusion accuracy of 8 bits and a detail retention rate of ≥95%.

[0050] S4: HDR to LDR conversion. The Reinhard tone mapping module first calculates the average brightness and white point brightness of the frame through frame-level brightness statistics, with a statistical time of ≤1 frame period (1280×720 / 125M=7.37ms); then, it achieves dynamic range compression through a 6-level mapping pipeline, with a fixed-point calculation error of ≤1%, and outputs a 24-bit RGB format LDR image that meets display requirements.

[0051] S5: Synchronous output. The output synchronization submodule performs 6-level delay alignment on the LDR image control signal to ensure that the data and control signals are transmitted synchronously. The synchronously output LDR image is transmitted to the display device through the FPGA output interface. The synchronization error between the frame, line, and data valid signals and the pixel data is ≤1 clock cycle, with no line misalignment or frame tearing.

[0052] In a preferred embodiment of this example, step S3 employs a "prioritize intermediate exposure, optimize detail preservation" strategy for weight allocation: the exposure differences of the eight multi-exposure images are divided into three levels: low exposure, intermediate exposure, and high exposure. Each level corresponds to a preset upper limit threshold for weight, with the upper limit threshold for the intermediate exposure level being higher than that for the high and low exposure levels. This ensures that details in highlight areas come from the low-exposure image, while details in shadow areas come from the high-exposure image. For example, the upper limit for weight of the two images with the highest exposure (Exp7, Exp6) is 0.8, the upper limit for weight of the two images with the lowest exposure (Exp1, Exp0) is 0.7, and the upper limit for weight of the four intermediate images (Exp5~Exp2) is 1.0.

[0053] As a preferred implementation of this embodiment, in step S4, the fixed-point operation of the optimized Reinhard algorithm adopts an 18-bit format, with 8 bits for integers and 10 bits for fractions. The division operation is implemented through the FPGA's built-in division IP core, and the multiplication operation is optimized by shifting and adding (e.g., 0.3×L is implemented by L×1229>>12, 1229=0.3×2¹²), avoiding the precision loss and delay caused by floating-point operations.

[0054] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0055] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0057] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A real-time multi-exposure HDR image processing system based on FPGA, characterized in that, include: The DDR3 image storage control module is integrated inside the FPGA and communicates with the external DDR3 chip using the AXI4 bus protocol. It includes a write FIFO buffer submodule, a read FIFO buffer submodule, and a dual-state machine read / write control submodule. It supports high-speed storage and parallel reading of 8-channel multi-exposure images and outputs synchronized multi-exposure image data and frame, line, and data valid signals. The multi-exposure HDR fusion module is connected to the output of the DDR3 image storage control module. It adopts a 6-level hardware pipeline architecture, including an input parsing submodule, an RGB to grayscale submodule, a pixel mean and variance calculation submodule, a basic weight allocation submodule, an overexposure and underexposure weight adjustment submodule, and a weighted summation submodule. It realizes pixel-level weighted fusion of 8 multi-exposure images and outputs a 24-bit RGB format HDR fusion image. The Reinhard tone mapping module, connected to the output of the multi-exposure HDR fusion module, includes a synchronization signal processing submodule, a frame-level luminance statistics submodule, a 6-level delay chain mapping pipeline submodule, and an output synchronization submodule. Based on the optimized Reinhard algorithm, it realizes dynamic range compression from HDR image to LDR image and outputs a 24-bit RGB format LDR image that meets the requirements of the display device.

2. The FPGA-based multi-exposure HDR image real-time processing system according to claim 1, characterized in that, The dual-state machine read / write control submodule of the DDR3 image storage control module includes a write state machine and a read state machine: The write state machine includes six states: IDLE, write configuration, write enable, FIFO data read, burst write count, and write complete. It supports allocating storage addresses for eight multi-exposure images, HDR fusion images, and LDR mapping images according to exposure. It adopts a fixed burst length transmission method, and the corresponding write FIFO cache submodule meets a storage depth of not less than a specified threshold to ensure that there is no packet loss in the eight multi-exposure images. The read state machine includes five states: IDLE, address switching, read enable, burst read count, and read complete. It supports parallel reading of eight multi-exposure images. The read FIFO buffer submodule is set with a "near full" threshold to trigger DDR3 read pause and prevent the read FIFO buffer submodule from overflowing. The output terminal is set with a channel flag to realize channel identification and synchronous output of eight multi-exposure image data.

3. The FPGA-based multi-exposure HDR image real-time processing system according to claim 1, characterized in that, The functions of each level in the 6-level hardware pipeline architecture of the multi-exposure HDR fusion module are as follows: The first-level input parsing submodule is used to extract the 8-channel multi-exposure image RGB channel data output by the DDR3 module, buffer it into the pipeline register, and transmit frame, line, and data valid signals. The second-level RGB to grayscale submodule is used to optimize the multiplication operation of the grayscale conversion formula Y=a*R+b*G+c*4B based on the selected brightness conversion formula, where Y is the grayscale value, R, G, and B are the blue, red, and blue channel components, respectively, and a, b, and c are the weight coefficients corresponding to each component. After multiplying each weight coefficient by 256, the multiplication operation of the grayscale conversion formula is optimized by shifting and adding. Finally, the result is shifted 8 bits to the right to output an 8-bit grayscale value. The third-level pixel mean and variance calculation submodule is used to calculate the mean and standard deviation of the three RGB channels of each image, and uses Newton's iteration method to implement the square root operation. The fourth-level basic weight allocation submodule is used to prioritize the bright parts in the middle of the gray value. When the pixel gray value is <127, the weight = gray value + d, where d is a fixed constant. When the gray value is ≥127, the weight = 256 - gray value. The output is an 8-bit basic weight. The Level 5 overexposure and underexposure weight adjustment submodule is used to adjust the basic weights based on preset overexposure and underexposure thresholds, reduce the weight of overexposed images, reduce the weight of underexposed images, and increase the weight of intermediate exposure images, and calculate the total weight. The sixth-level weighted summation submodule is used to normalize the weights through the division IP core. The RGB channels of the eight multi-exposure images are multiplied by the normalized weights and then summed to output a 24-bit RGB format HDR fusion image.

4. The FPGA-based multi-exposure HDR image real-time processing system according to claim 1, characterized in that, The frame-level luminance statistics submodule of the Reinhard tone mapping module adopts a 5-state machine design: IDLE state: Waiting for the rising edge of the HDR fused image frame start signal; FSTART state: Waiting for the rising edge of the row start signal, preparing to enter row statistics; LINE Status: When data is valid, accumulate the pixel grayscale value (Gray) and count the total number of pixels; CALC status: Calculate the frame average brightness mean_value and white point brightness Lwhite; READY state: Outputs a statistics completion flag, keeps mean_value and Lwhite stable, and waits for the next frame of statistics to be used for calculation; The 6-stage delay chain mapping pipeline submodule implements the optimized Reinhard algorithm based on fixed-point arithmetic, expressed by the formula: L_scaled=(0.3×L) / mean_value, where L is the original pixel brightness; L_out=(L_scaled×(1+L_scaled / Lwhite²)) / (1+L_scaled), where L_out is the brightness value after nonlinear mapping of the original brightness in the optimized Reinhard algorithm; Ratio=L_out / Gray, where Ratio is the proportion of brightness after mapping corresponding to each unit of original grayscale, and the three channels maintain color balance; R_out=In_pixel_r*Ratio, where In_pixel_r is the original red channel value of the pixel after HDR fusion, and R_out is the final displayed red component; the calculation of the R, G, and B channels is similar, with each stage delayed by 1 clock cycle to ensure alignment with the input synchronization signal.

5. The FPGA-based multi-exposure HDR image real-time processing system according to claim 2, characterized in that, A channel synchronization submodule is set between the DDR3 image storage control module and the multi-exposure HDR fusion module, including an 8-channel flag register and a channel address mapping table. The 8 channel flag registers correspond to 8 multi-exposure images. When the DDR3 image storage control module reads the address in the storage range of the corresponding exposure image, the corresponding channel flag bit is pulled high at the output of the read state machine. The multi-exposure HDR fusion module identifies images with different exposure levels based on the channel flags and achieves pixel-level alignment and fusion.

6. The FPGA-based multi-exposure HDR image real-time processing system according to claim 1, characterized in that, The weight adjustment submodule of the multi-exposure HDR fusion module uses a smooth step function to optimize the transition: when the pixel grayscale is within the set ratio transition zone of the overexposure threshold or underexposure threshold, the weight is smoothly adjusted according to the smooth step function to avoid artifacts in the fused image caused by sudden weight changes.

7. The FPGA-based multi-exposure HDR image real-time processing system according to claim 1, characterized in that, The output synchronization submodule of the Reinhard tone mapping module includes three sets of six-level delay chains, which delay the frame valid signal, line valid signal, and data valid signal respectively, so that the control signal of the output LDR image is strictly consistent with the delay of the mapped pixel data.

8. A real-time processing method for multi-exposure HDR images based on FPGA, characterized in that, The method is implemented based on the system as described in any one of claims 1 to 7, and includes the following steps: S1: The DDR3 image storage control module caches 8 multi-exposure images through the write FIFO cache submodule, allocates storage addresses according to the exposure of the write state machine, and uses AXI4 bus burst write operation to write to the external DDR3 chip. S2: The read state machine is based on the channel flag. It reads 8 multi-exposure images in parallel through the read FIFO buffer submodule and outputs the synchronized image data and frame, line and data valid signals. S3: The multi-exposure HDR fusion module completes input analysis, RGB to grayscale conversion, pixel mean and variance calculation, basic weight allocation, overexposure and underexposure weight adjustment and weighted summation through a 6-stage pipeline, and outputs a 24-bit RGB format HDR fusion image. S4: The Reinhard tone mapping module first calculates the average brightness and white point brightness of the frame through frame-level brightness statistics, and then achieves dynamic range compression through a 6-level mapping pipeline to output a 24-bit RGB format LDR image that meets display requirements. S5: The output synchronization submodule performs 6-level delay alignment on the LDR image control signal to ensure that the data and control signal are transmitted synchronously.

9. The FPGA-based real-time multi-exposure HDR image processing method according to claim 8, characterized in that, In step S3, the weight allocation adopts the strategy of "prioritizing intermediate exposure and optimizing detail preservation": the exposure difference of the 8 multi-exposure images is divided into three levels, namely low exposure level, intermediate exposure level and high exposure level. Each level corresponds to a preset upper limit threshold of weight. The upper limit threshold of weight for intermediate exposure level is higher than that for high exposure level and low exposure level, so as to ensure that the details in the highlight area come from the low exposure image and the details in the shadow area come from the high exposure image.

10. The FPGA-based real-time multi-exposure HDR image processing method according to claim 8, characterized in that, In step S4, the division operation is implemented through the FPGA's built-in division IP core, and the multiplication operation is optimized through shifting and adding to avoid the precision loss and delay caused by floating-point operations.

Images