Infrared image bilateral filtering implementation method based on FPGA optimization

Abstract

The invention relates to the field of image processing, in particular to an infrared image bilateral filtering method based on field programmable gate array (FPGA) optimization, which comprises the following steps: acquiring data through a hierarchical architecture of'on-chip BRAM cache + distributed register cache ', intercepting pixels in combination with a sliding window, calculating a spatial weight by using an LUT look-up table, and logically calculating a gray scale weight in real time. And carrying out weighted summation and normalization output. According to the method, the processing delay of the infrared image with the resolution of 1024 * 1024 is reduced to be less than 50ms, the frame rate is increased to be more than 60fps, and the method is suitable for a real-time target recognition scene. Through the combination of the optimized bilateral filtering algorithm and the efficient hardware architecture, the processing delay is remarkably reduced, the real-time processing of the high-resolution infrared image is realized, and the real-time performance is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an infrared image bilateral filtering method based on field programmable gate array (FPGA) optimization implementation, aiming to solve the real-time performance and resource efficiency problems in infrared image processing. BACKGROUND

[0002] Infrared images have important applications in medical imaging, industrial detection and other fields. However, due to the characteristics of the sensor and the complex imaging environment, the infrared imaging system is often disturbed by noise, resulting in a significant decrease in image quality. Traditional image denoising algorithms are mainly based on smoothing, which can easily lead to loss of edge information. Bilateral filtering, as a non-linear edge-preserving filtering method, can effectively smooth noise while preserving image details and edge structures, and has shown good application prospects in infrared image processing.

[0003] Although the bilateral filtering algorithm has good performance in theory, its high computational complexity hinders its direct application in real-time applications. Especially for high-resolution infrared images, the amount of calculation increases exponentially. Therefore, how to efficiently implement bilateral filtering on hardware has become a problem to be solved. FPGA is an ideal platform for implementing high-performance image processing due to its parallel processing capability, flexible programmable characteristics and low power consumption. When existing FPGAs implement bilateral filtering, the high-frequency data read-write delay is more than 100ns due to the use of a single cache structure; more than 50% of the logic units are occupied by real-time exponential operations in weight calculation, resulting in a frame rate of less than 20fps for 1024x1024 resolution images. Through structural optimization of the bilateral filtering algorithm and customized design of the FPGA hardware architecture, the efficiency and real-time performance of infrared image processing can be significantly improved. SUMMARY

[0004] The purpose of the present application is to provide an infrared image bilateral filtering implementation method based on FPGA optimization to solve the problems of low real-time processing performance, high resource occupancy and insufficient algorithm adaptability in the prior art. Through the decomposition and optimization of the bilateral filtering algorithm and the design of an efficient architecture based on the hardware characteristics of FPGA, the present application finally realizes high-quality real-time processing of infrared images.

[0005] The technical solution adopted by the present application to achieve the above purpose is:

[0006] An infrared image bilateral filtering implementation method based on FPGA optimization, comprising the following steps:

[0007] 1) obtaining infrared image data through FPGA;

[0008] 2) obtaining pixel values of infrared image data in the filter kernel region based on line buffer registers through a sliding window method;

[0009] 3) sequentially performing weight calculation and weighted summation processing on the pixel values to obtain the filtered infrared image.

[0010] The FPGA processes the real-time read-write demand of high-frequency data by constructing a distributed register cache with nanosecond-level read-write response through a lookup table (LUT) and a flip-flop; relies on a Block RAM hard core to realize a large-capacity BRAM cache supporting multi-port parallel access, and stores the whole row or frame of image data through the BRAM cache; when the image is loaded, the data is first stored in the BRAM cache and then transmitted to the distributed register through the line buffer register.

[0011] The step 2) is specifically:

[0012] The sliding window is based on the multiple rows of data stored in the line buffer register, and slides row by row according to the set window size, and each time a pixel area of the corresponding size is intercepted, and finally the center pixel value I(i,j) and the difference I(i,j)-I(k,l) between the center pixel and the 8 neighboring pixels are obtained, where (k,l) represents the neighboring pixel coordinates.

[0013] The step 3) includes the following steps:

[0014] 3.1) calculating the Gaussian spatial weight and the gray weight of the pixel value;

[0015] 3.2) using a multiplication accumulator to perform weighted summation on the pixel value;

[0016] 3.3) performing normalization processing on the summed pixel value to obtain the filtered infrared image.

[0017] The step 3.1) includes the following steps:

[0018] 3.1.1) constructing a lookup table (LUT) through the mapping relationship between the pixel position distance and the corresponding Gaussian spatial weight, using the bilateral filtering rule, and calculating the spatial weight w s (i,j,k,l) of the current neighboring pixel relative to the center pixel I(i,j) based on the LUT;

[0019] 3.1.2) dynamically calculating the pixel difference to obtain the gray weight w γ (i,j,k,l).

[0020] The bilateral filtering rule is specifically

[0021] The spatial domain kernel weight g d (i,j,k,l) and the value domain kernel weight g t (i,j,k,l) are calculated respectively:

[0022]

[0023]

[0024] wherein (i,j) is the coordinate of the center pixel in the filter window, (k,l) is the coordinate of other pixel points in the filter window, σ d is the standard deviation in the spatial domain kernel, σ t is the standard deviation in the value domain kernel, f(i,j) is the gray value of the center pixel in the filter window, f(k,l) is the pixel value of other pixel points in the filter window, f t (i,j) is the gray value of the center pixel in the filter window in the value domain kernel, f t (k,l) is the pixel value of other pixel points in the filter window in the value domain kernel;

[0025] The total weight of the bilateral filter is calculated as follows:

[0026] ω BF (i,j,k,l)=g d (i,j,k,l)×g t (i,j,k,l)

[0027] The bilateral filter is performed based on the total weight:

[0028]

[0029] wherein I'(i,j) represents the filtered pixel data in the filter window, i.e. the output pixel after filtering, M s represents the sum of the pixel gray value and the product of the bilateral filter weight in the filter window, W s represents the sum of the bilateral filter weight.

[0030] The pixel difference value is dynamically calculated, specifically as follows:

[0031] The absolute value of the pixel difference value I(i,j)-I(k,l) is calculated and substituted into the gray domain Gaussian weight formula, and the gray weight w γ (i,j,k,l) is dynamically calculated by the FPGA logic unit, wherein the gray domain Gaussian weight formula is as follows:

[0032]

[0033] wherein σ r is the gray domain standard deviation.

[0034] The step 3.2) is specifically as follows:

[0035] The filtered center pixel value I'(i,j) is first multiplied by the spatial weight w s and then multiplied by the gray weight w rMultiplication, generate a single neighborhood pixel pair center pixel bilateral weighted value I'(i,j) x w s x w r .

[0036] The present application has the following advantages and benefits:

[0037] 1. The present application significantly reduces the processing delay by combining the optimized bilateral filtering algorithm with the efficient hardware architecture, realizes the real-time processing of high-resolution infrared images, and significantly improves the real-time performance.

[0038] 2. The present application reduces the consumption of FPGA logic units and storage resources through algorithm decomposition optimization and lookup table operation, and improves the resource utilization.

[0039] 3. The present application preserves the image edge and detail information while effectively removing noise through the optimized bilateral filtering algorithm, significantly improves the visual effect of infrared images, and is suitable for key applications such as target identification and tracking. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 Principle diagram of bilateral filtering;

[0041] Figure 2 Overall architecture diagram;

[0042] Figure 3 Row buffer register architecture diagram;

[0043] Figure 4 Weight calculation module architecture diagram;

[0044] Figure 5 Bilateral filtering FPGA internal pipeline operation flowchart. DETAILED DESCRIPTION

[0045] The present application will be further described in detail below in combination with the drawings and examples.

[0046] The present application uses FPGA as the core computing platform, and realizes the bilateral filtering algorithm by combining the bilateral filtering algorithm principle shown in Figure 1 with the Verilog hardware description language. The specific FPGA internal architecture and pipeline operation process are shown in Figure 2 FPGA realizes the real-time read-write demand of high-frequency data through the distributed register buffer constructed by lookup table (LUT) and flip-flop, which has nanosecond-level read-write response; relies on the implementation of Block RAM hard core to realize the large-capacity BRAM buffer supporting multi-port parallel access, which stores the whole row or whole frame image data.

[0047] Firstly, the input image data is stored in a line buffer register, which can adopt a specific commonly used type, such as an asynchronous FIFO with a FIFO structure, and the data buffer and reading are controlled by read and write pointers, which are tools for adapting to the input rhythm of image data stream. The pixel values of the filter kernel region are obtained by a sliding window method (the sliding window can be based on the multiple rows of data stored in the line buffer register, and the window size is set to slide row by row, and each time a pixel region of the corresponding size is intercepted).

[0048] Then, the calculation of the Gaussian spatial weight and the gray weight is realized in the hardware logic, wherein the spatial weight is obtained by a lookup table (LUT), and the gray weight is dynamically calculated according to the input pixel difference value. After all the weight calculation is completed, a multiply-accumulator (MAC) unit is used for weighted summation, and normalization processing is performed to obtain the final filtering result. In addition, in order to improve the calculation throughput of the FPGA, a pipeline structure is adopted, and the image loading, weight calculation and filtering calculation are modularized. When the image is loaded, the data is first stored in the BRAM buffer, and then transmitted to the distributed register through the line buffer register; the weight calculation module reads the pixel values from the register, and the results are transmitted to the normalization module through the MAC unit, and each link is synchronized through the read and write pointers to realize no waiting delay, so as to improve the processing efficiency.

[0049] In addition, the application also adopts an optimized data flow control strategy to ensure the continuity and efficiency of data transmission. Through the multi-level cache structure in the FPGA, the memory access delay is effectively reduced, and the overall throughput of the system is improved. The multi-level cache structure can be designed as a "on-chip BRAM cache + distributed register cache" hierarchy, the distributed register cache is used to process high-frequency small data reading and writing, and the BRAM cache is used to store whole row / whole frame image data; during data processing, the data is preferentially taken from the distributed register, and the missing data is taken from the BRAM cache. At the same time, in the FPGA logic design, parallel calculation and pipeline operation are used to further improve the calculation speed, so that it can meet the demand of high frame rate image processing.

[0050] The pipeline operation process of the bilateral filter FPGA is roughly as follows: the input image is taken as the starting data, and first enters the 5*5 filter window module. The window is not simply intercepted, but based on the multiple rows of data stored in the FPGA line buffer register (as shown in Figure 3 The line buffer register caches the continuous row pixels of the image in advance to ensure that the neighborhood data can be obtained in real time when the window slides), and works according to the logic of "sliding row by row and covering point by point". The target pixel is taken as the center, moves horizontally by 1 pixel each time, and vertically synchronously cooperates with the line switching of the line buffer, dynamically intercepts a 5 row*5 column pixel matrix, outputs the center pixel value I(i,j), and the difference value I(i,j)-I(k,l) of the center pixel and the eight neighborhood pixels, wherein (k,l) represents the neighborhood pixel coordinates.

[0051] The obtained I(i,j) and I(i,j)-I(k,l) flow to different branches, and the bilateral filtering core calculation is started, as shown in the following formula: Figure 5

[0052] The spatial weight calculation branch: the difference I(i,j)-I(k,l) first enters the spatial weight module. According to the bilateral filtering rule that “the closer the spatial distance, the greater the weight”, the module realizes efficient calculation through a pre-constructed LUT (lookup table). During the development stage, the mapping relationship between the pixel position distance (converted by the coordinate difference) and the corresponding Gaussian spatial weight is calculated offline and stored in the LUT. When the difference is input, the module quickly looks up the table and outputs the spatial weight w s (i,j,k,I) of the current neighborhood pixel relative to the center pixel I(i,j), reflecting the influence of the spatial domain on the filtering result.

[0053] The gray weight calculation branch: also based on the difference I(i,j)-I(k,l), the gray weight module performs the logical operation of “the smaller the gray difference, the higher the weight”. It does not rely on table lookup, but calculates in real time through hardware logic: the pixel difference is processed by absolute value (eliminating the positive and negative effects), and then substituted into the gray domain Gaussian weight formula (such as σ r is the gray domain standard deviation, which can be flexibly adjusted through parameter configuration), and the gray weight w γ (i,j,k,l) is dynamically calculated by the FPGA logic unit (such as multiplier, exponential operation optimization circuit), reflecting the contribution of the gray domain to the filtering.

[0054] When the spatial weight w s and the gray weight w r are calculated, the two branches are integrated into the weight fusion and filtering calculation link: the center pixel value I(i,j) is first multiplied by the spatial weight w s (the pixel contribution after spatial domain weighting is obtained), and then multiplied by the gray weight w r (the gray domain weighting effect is superimposed), to generate the “bilateral weighted value” I(i,j)×e s ×w r of a single neighborhood pixel to the center pixel.

[0055] Since the filtering window contains 8 neighborhood pixels, the above-mentioned “calculation of a single neighborhood weighted value” operation will be performed 8 times in parallel (with the help of FPGA parallel computing advantage, 8 groups of multipliers and weight modules work synchronously). Subsequently, the 8 weighted values and the “weighted value” of the center pixel itself (the center pixel spatial weight is 1 and the gray weight is 1, that is, I(i,j)×1×1) flow into the accumulation and normalization module together.

[0056] ​The filtered pixel values are re-sequenced and output in image row and column order to form a complete image after bilateral filtering processing. The entire process relies on the FPGA pipeline architecture, from image input, window extraction, bilateral weight calculation, to accumulation and normalization. Each link is like a station on the assembly line, and the output of the previous link directly flows into the next link without waiting for delay, ensuring that high frame rate images (such as high-definition video streams) can be continuously and efficiently processed, and realizing the hardware acceleration of bilateral filtering on the FPGA platform.

[0057] Such process design not only conforms to the principle of bilateral filtering algorithm, but also deeply combines the hardware characteristics of FPGA parallel computing and pipeline execution. Through hardware logic optimization (such as LUT acceleration and adder tree parallelism), the problem of high computational delay in traditional software implementation is solved, and high-efficiency bilateral filtering denoising and enhancement processing can be achieved in scenes with high real-time requirements such as infrared images.

[0058] The application proposes an optimization strategy based on kernel function decomposition, which independently calculates the spatial domain Gaussian kernel and the value domain kernel to reduce operation coupling and improve computational efficiency.

[0059] As shown in Figure 4 , the formula for spatial domain kernel calculation is represented by equation 1, and the formula for value domain kernel calculation is represented by

[0060] equation 2.

[0061]

[0062]

[0063] where: (i,j) is the center pixel coordinate in the filter window, (k,l) is the coordinate of other pixel points in the filter window, σ d is the standard deviation in the spatial domain kernel, σ t is the standard deviation in the value domain kernel. f(i,j) is the gray value of the center pixel of the filter window (range 0-255), and f(k,l) is the pixel value of other pixel points in the filter window.

[0064] From the above two equations, the value domain kernel weight and the spatial domain kernel weight can be obtained, and the total bilateral filtering weight is shown in equation 3.

[0065] ω BF (i,j,k,l)=g d (i,j,k,l)×g t (i,j,k,l) (3)

[0066] The bilateral filtering function formula is shown in equation 4:

[0067]

[0068] wherein: I(i, j) represents the output data in the filter window, i.e. the pixel output after filtering, and M s and W s are respectively shown in formula (5) and formula (6):

[0069]

[0070]

[0071] It can be seen from the above formula that the bilateral filtering calculation depends on the exponential calculation, the calculation amount is too large for the FPGA platform, a large number of chip resources are occupied, and the image processing speed of other modules is affected.

[0072] The present application adopts a lookup table (LUT, i.e. the results of exponential function calculation are stored therein in advance, the corresponding value is directly obtained by looking up the table, and real-time exponential operation is avoided) to replace the exponential function calculation, so as to reduce the occupation of hardware resources, and in combination with a multi-stage pipeline architecture based on FPGA, the data processing is carried out in a parallel mode, so as to improve the throughput;

[0073] The present application introduces a hierarchical cache mechanism, optimizes the on-chip storage resources, reduces the access delay of external storage, and through a block processing strategy, the input image can be quickly calculated in a local area, and the real-time performance is further improved.

Claims

1. A method for implementing bilateral filtering of infrared images based on FPGA optimization, characterized in that, Includes the following steps: 1) Acquire infrared image data via FPGA; 2) Obtain the pixel values ​​of the infrared image data of the filtering kernel region using a sliding window method based on the row buffer register; 3) Perform weight calculation and weighted summation on the pixel values ​​in sequence to obtain the filtered infrared image.

2. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 1, characterized in that, The FPGA constructs a distributed register cache with nanosecond-level read / write response using lookup tables (LUTs) and flip-flops to handle the real-time read / write requirements of high-frequency data. It also implements a large-capacity BRAM cache that supports multi-port parallel access by relying on the Block RAM hard core. The BRAM cache stores the entire line or frame of image data. When the image is loaded, the data is first stored in the BRAM cache and then passed to the distributed register via the line buffer register.

3. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 1, characterized in that, Step 2) specifically refers to: The sliding window is based on multiple rows of data stored in the row buffer register. It slides row by row according to the set window size, and each time it captures a pixel area of ​​the corresponding size. Finally, it obtains the center pixel value I(i,j) and the difference between the center pixel and 8 neighboring pixels I(i,j)-I(k,l), where (k,l) represents the coordinates of the neighboring pixels.

4. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 1, characterized in that, Step 3) includes the following steps: 3.1) Calculate Gaussian space weights and grayscale weights for pixel values; 3.2) Use a multiply-accumulate method to perform a weighted summation of the pixel values; 3.3) Normalize the summed pixel values ​​to obtain the filtered infrared image.

5. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 4, characterized in that, Step 3.1) Includes the following steps: 3.1.1) Construct a lookup table (LUT) based on the mapping relationship between pixel location distance and corresponding Gaussian spatial weights. Using a bilateral filtering rule, calculate the spatial weight w of the current neighboring pixels relative to the center pixel I(i,j) based on the LUT. s (i,j,k,l); 3.1.2) Dynamically calculate the pixel difference to obtain the grayscale weight w. γ (i,j,k,l).

6. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 5, characterized in that, The bilateral filtering rule is specifically as follows: Calculate the spatial domain kernel weight g respectively d (i,j,k,l) ​​and the range kernel weight g t (i,j,k,l): Where (i,j) are the coordinates of the center pixel in the filtering window, (k,l) are the coordinates of other pixels in the filtering window, and σ d σ is the standard deviation in the spatial domain kernel. t Let f(i,j) be the standard deviation in the range kernel, f(i,j) be the gray value of the center pixel of the filter window, and f(k,l) be the pixel values ​​of other pixels in the filter window. t (i,j) represents the gray value of the center pixel of the filter window in the value range kernel, f t (k,l) represents the pixel values ​​of other pixels in the filtering window within the value range kernel; Calculate the total weights of the bilateral filter: ω BF (i,j,k,l)=g d (i,j,k,l)×g t (i,j,k,l) Bilateral filtering based on total weights: Where I'(i,j) represents the filtered pixel data output in the filtering window, i.e., the filtered output pixels, M s W represents the sum of the products of pixel grayscale values ​​and bilateral filter weights within the filter window. s This represents the sum of the weights in the bilateral filter.

7. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 5, characterized in that, The dynamic calculation of pixel differences specifically involves: Calculate the absolute value of the pixel difference I(i,j)-I(k,l) and substitute it into the Gaussian weighting formula in the grayscale domain. Then, dynamically calculate the grayscale weight w through the FPGA logic unit. γ (i,j,k,l), where the Gaussian weighting formula for the grayscale domain is: Where, σ r is the standard deviation of the grayscale range.

8. The method for implementing bilateral filtering of infrared images based on FPGA optimization according to claim 4, characterized in that, Step 3.2) specifically refers to: First, the filtered center pixel value I'(i,j) is compared with the spatial weight w. s Multiply by, then multiply by the grayscale weight w r Multiply to generate a bilateral weighted value I'(i,j)×w for each neighboring pixel with respect to the center pixel. s ×w r .

Images