An FPGA-based image preprocessing method and system

Abstract

The application discloses an image preprocessing method and system based on FPGA, relates to the technical field of image preprocessing, and realizes mean filtering, Gaussian filtering and median filtering in parallel on FPGA based on RTL, provides two deployment strategies of resource-efficient RET and delay-efficient LET for the mean filtering and Gaussian filtering respectively, accelerates window operation through Triangle FIFO cache lines and redundancy elimination, and adopts a hierarchical model library for the median filtering and selects an optimal sorting submodule according to an input size. All algorithms are packaged into parameterizable IP cores, support online configuration of window size, bit width, image size and the like, and are integrated into an image processing pipeline through an AXI4-Stream interface, so that the frame rate can be effectively improved, the resource occupancy rate is reduced, and the application is suitable for edge automatic driving, unmanned aerial vehicles and remote sensing imaging scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image preprocessing, and in particular to an image preprocessing method and system based on FPGA. BACKGROUND

[0002] Affected by factors such as shooting environment, camera failure, voltage fluctuation, etc., image data may have noise interference in the transmission process, causing image details to be blurred, edges to be distorted, etc., which seriously affects subsequent image processing steps; therefore, it is necessary to preliminarily denoise the collected image to reduce noise interference; common image denoising algorithms include traditional filtering (such as mean filtering, median filtering), wavelet transform, convolutional neural network, etc., the traditional filtering algorithm balances the noise interference around each pixel point based on window sliding, and the principle is simple and fast to calculate; the wavelet transform realizes signal-noise separation by utilizing the difference in frequency domain distribution of noise and signal; the convolutional neural network learns the noise distribution rule and filters out noise layer by layer with the help of convolution operation; in practical application, it is necessary to select a suitable denoising algorithm according to project requirements.

[0003] With the development of hardware end devices and the diversification of user end requirements, image processing functions are gradually transferred from the cloud to the edge, becoming an important part of emerging fields such as autonomous driving and satellite remote sensing; these fields often have very high requirements for processing efficiency, and the characteristics of limited resources on the edge require the deployed algorithm to be as lightweight as possible; the traditional CPU deployment scheme cannot meet the real-time requirements; GPU is suitable for large-scale parallel computing scenarios, but also has high energy consumption; ASIC has the best energy efficiency, but its highly customized characteristics make it difficult to be compatible with rapidly iterating algorithms; FPGA has a shorter development cycle and high reconfigurability, supports arbitrary data precision, and parallel processing of data streams between logic units, becoming an ideal choice for edge image processing hardware acceleration.

[0004] Existing works generally regard image denoising as a step of image processing system, only provide a principle flow description, lack of specific optimization strategy for image denoising algorithm; at the same time, some works deploy image denoising algorithm on FPGA using high-level synthesis tools, this way has high development efficiency, but lacks fine-grained optimization of underlying circuits; in addition, as an important step of image preprocessing, image denoising is usually integrated into a complete image system as a sub-module; however, existing image denoising modules generally adopt customized design mode for specific parameters, have poor scalability, small applicable scope, etc., and are difficult to integrate into a general module. SUMMARY

[0005] In view of the above existing problems, the present application is proposed.

[0006] The application provides an FPGA-based image preprocessing method and system, and solves the problems of low circuit performance, insufficient optimization degree and poor universality in FPGA hardware acceleration of an image denoising algorithm in the prior art.

[0007] To solve the above technical problems, the application provides the following technical solutions.

[0008] In a first aspect, the application provides an FPGA-based image preprocessing method, which comprises,

[0009] In step S1, three image denoising algorithms of mean filtering, Gaussian filtering and median filtering are deployed in parallel on an FPGA based on an RTL development mode, a bottom layer optimization circuit is designed for each algorithm, and a corresponding filtering algorithm is selected according to a noise type;

[0010] In step S2, a denoising algorithm module is packaged as a parameterized IP core, and configuration parameters include a filtering type, a sliding window size, data precision, an image size and a pixel delay;

[0011] In step S3, a circuit is optimized by using a synthesis tool, and a netlist and an IP core calling template are generated;

[0012] In step S4, an IP core is called to complete image denoising processing;

[0013] The mean filtering and the Gaussian filtering adopt a resource-efficient RET or a delay-efficient LET deployment strategy, the median filtering is constructed based on a hierarchical model library, and all modules support parameterized configuration.

[0014] As a preferred scheme of the FPGA-based image preprocessing method, the resource-efficient RET deployment strategy comprises:

[0015] For the mean filtering and the Gaussian filtering, a single shift register, a register array, a binary tree adder and a divider are used to realize weighted calculation in a sliding window;

[0016] The binary tree adder accumulates all pixel points in the window, and the divider completes normalization.

[0017] The pixel point duration needs to be greater than the accumulation operation period to avoid data overlap.

[0018] As a preferred scheme of the FPGA-based image preprocessing method, the delay-efficient LET deployment strategy comprises:

[0019] A row sum cache unit (Triangle FIFO Group) is constructed to store row sums of the first K-1 rows in the sliding window.

[0020] The current row and is calculated for the new row and is accumulated with the cache row and to eliminate transverse redundant calculation;

[0021] For Gaussian filtering, the cache row and is scaled and accumulated according to the ratio between rows.

[0022] As a preferred scheme of the image pre-processing method based on FPGA, the construction of the hierarchical median filter model library comprises:

[0023] The median filter is divided into multiple layers of sub-modules, and each layer selects an optimal sorting structure according to an input condition;

[0024] The sub-modules include 2-input to 9-input sorting units, which adopt different structures for input order to reduce resource consumption;

[0025] For 3*3 window nine-input median filtering, a 3+3+3 or 4+1+4 sub-module combination is used to realize pipeline optimization.

[0026] As a preferred scheme of the image pre-processing method based on FPGA, the parameterized configuration comprises:

[0027] The filter type, window size, image size and data precision are dynamically set through an external interface;

[0028] According to the configuration parameters, the adaptive shift register length, adder level and sorting module structure are automatically generated;

[0029] Supporting multi-channel input and output and different padding type expansion.

[0030] In a second aspect, the application provides an image pre-processing system based on FPGA, comprising,

[0031] An image acquisition module is configured to acquire original image data;

[0032] An image denoising module is configured to perform denoising processing on the collected data;

[0033] A data transceiver module is configured to realize data transmission and protocol conversion between the FPGA and the host computer;

[0034] A result display module is configured to output the denoised image to a display screen;

[0035] The image denoising module is encapsulated as an IP core and integrated in the FPGA development board and connected to each module through an AXI4 bus.

[0036] As a preferred scheme of the image pre-processing system based on FPGA, the IP core interface of the image denoising module comprises:

[0037] Input port: clock signal, reset signal, data valid pulse and multi-channel pixel data;

[0038] Output port: denoised data valid pulse and multi-channel result data;

[0039] Configuration port: filter mode, window size, sliding step and filling type parameters.

[0040] As a preferred scheme of the image pre-processing system based on FPGA, the image acquisition module adopts a 200 million pixel CMOS image sensor, and temporarily stores data through a DDR3 chip; the result display module realizes level and format conversion through an HDMI interface.

[0041] As a preferred scheme of the image pre-processing method based on FPGA, the row and scaling is specifically:

[0042] When the inter-row ratio of the Gaussian kernel is a power of 2, a shift operation is used instead of multiplication and division;

[0043] When the ratio is not a power of 2, a pipeline divider is inserted to complete the row and scaling.

[0044] As a preferred scheme of the image pre-processing method based on FPGA, the timing optimization of the nine-input median filter includes:

[0045] The overlapping window part retains ordered input, reducing the sorting level;

[0046] The Median-3-impUnit and Median-7-simp-1Unit are combined to reduce resource consumption and improve the clock frequency to 406.7MHz.

[0047] The present application has the following advantages: the present application adopts two filtering algorithms, resource-efficient and delay-efficient deployment strategies: for mean filtering and Gaussian filtering with similar denoising principles, resource consumption and inference efficiency are optimized respectively, and users can select the corresponding strategy according to available resources and speed requirements;

[0048] For the median filtering algorithm, based on the modular design idea, the sorting sub-module is constructed in layers according to the input conditions, and the optimal sorting structure is selected in each layer to build the top-level filtering module. Compared with other circuit design schemes, the median filter of the present application has optimal resource consumption and inference delay, and users can build more complex median filters according to the model library;

[0049] All modules are packaged as IP cores by parameterized design, support multiple parameters such as image size, data precision, pixel delay, filter type, etc., users can change parameters as needed in the configuration interface, realize flexible design of image denoising system;

[0050] The packaged IP core is integrated into the FPGA development board, a data transmission interface is reserved for the upper computer, and a complete FPGA image denoising system including four modules of image acquisition, image denoising, result display, data transceiving and format conversion is formed by matching a high-speed bus.

[0051] Compared with the prior art, the FPGA circuit design of the three image denoising algorithms is optimized, the hardware resource consumption is saved, and the inference efficiency of the image denoising process is improved; the packaged IP core can be integrated as a sub-module into other image processing systems, and users can flexibly configure external interfaces according to application requirements. DETAILED DESCRIPTION

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 The flowchart of the image preprocessing method based on FPGA of the present application.

[0054] Figure 2 The structure diagram of the shift register and the register array of the present application.

[0055] Figure 3 The structure diagram of the binary tree adder of the present application.

[0056] Figure 4 The pseudo code diagram of the divider of the present application.

[0057] Figure 5 The processing flowchart of the RET of the present application.

[0058] Figure 6 The schematic diagram of five selection domain overlap modes of the present application.

[0059] Figure 7 The design flowchart of the LET of the present application.

[0060] Figure 8 The Gaussian kernel diagram of three sliding window sizes of the present application.

[0061] Figure 9 The hierarchical median filter model library schematic diagram of the present application.

[0062] Figure 10 Example pictures before and after adding two kinds of noise of the present application.

[0063] Figure 11 Denoising effect pictures of three filtering algorithms of Gaussian noise of the present application.

[0064] Figure 12 Denoising effect pictures of three filtering algorithms of salt and pepper noise of the present application.

[0065] Figure 13 Fps acceleration ratio pictures of three filtering algorithms of the present application for different sliding windows.

[0066] Figure 14 Resource consumption comparison pictures of RET and LET under four sliding windows of the present application.

[0067] Figure 15 Fps comparison pictures of LET and RET in mean filtering and Gaussian filtering of the present application.

[0068] Figure 16 Image denoising module packaging IP and interface description picture of the present application.

[0069] Figure 17 Image denoising system architecture schematic diagram of the present application. DETAILED DESCRIPTION

[0070] In order to make the above objectives, features and advantages of the present application more apparent and comprehensible, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0071] In the following description, a lot of specific details are set forth in order to facilitate a full and complete understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0072] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments.

[0073] Image data may be disturbed by noise during transmission due to factors such as shooting environment, camera failure, voltage fluctuation, etc., causing image details to be blurred and edges to be distorted, which seriously affects subsequent image processing steps; therefore, it is necessary to preliminarily denoise the collected image to reduce noise interference; common image denoising algorithms include traditional filtering (such as mean filtering and median filtering), wavelet transform, and convolutional neural network, etc., traditional filtering algorithms balance the noise interference around each pixel point based on window sliding, and the principle is simple and fast to calculate; wavelet transform realizes signal-noise separation by utilizing the difference in frequency domain distribution of noise and signal; convolutional neural network learns noise distribution rules and filters noise layer by layer with convolution operation; in practical application, it is necessary to select appropriate denoising algorithm according to project requirements.

[0074] With the development of hardware devices and the diversification of user needs, image processing functions are gradually transferred from the cloud to the edge, becoming an important part of emerging fields such as autonomous driving and satellite remote sensing; these fields often have very high requirements for processing efficiency, and the characteristics of limited resources on the edge require the deployed algorithm to be as lightweight as possible; traditional CPU deployment schemes cannot meet real-time requirements; GPU is suitable for large-scale parallel computing scenarios, but also has high energy consumption; ASIC has the best energy efficiency, but its highly customized features make it difficult to be compatible with rapidly iterating algorithms; FPGA has a shorter development cycle and high reconfigurability, supports arbitrary data precision, and parallel processing of data streams between logic units, becoming an ideal choice for image processing hardware acceleration on the edge.

[0075] Existing work generally takes image denoising as a step of image processing system, only provides a principle flow description, lacks specific optimization strategies for image denoising algorithms; at the same time, some works deploy image denoising algorithms on FPGA using high-level synthesis tools, this way has high development efficiency, but lacks fine-grained optimization of underlying circuits; in addition, image denoising as an important step of image preprocessing, is usually integrated into a complete image system as a sub-module; however, existing image denoising modules generally adopt customized design mode for specific parameters, have poor scalability, small applicable scope, etc., and are difficult to integrate into general modules.

[0076] In the research of image denoising algorithm, it is found that some works use convolutional neural network as image denoising algorithm, which can achieve good denoising effect, but does not consider the long inference delay of neural network. Considering using traditional filtering algorithm to remove image noise. At the same time, the existing technology only considers the basic deployment of image denoising algorithm on the hardware side, and there is still room for improvement and optimization in circuit design. Considering the optimization design of the underlying circuit combined with the algorithm principle. In addition, the use of high-level synthesis tools only considers the development efficiency of algorithm deployment, and does not consider the performance of the implemented circuit. Considering using the development method of RTL to realize the image denoising algorithm. The customized circuit design only considers the specific needs of the project, and does not consider the scalability of the circuit design and the universality of module integration. Considering adding parameterized design to realize a set of image denoising methods compatible with multiple parameters, and users can customize the circuit flexibly according to their own needs.

[0077] Embodiment 1, refer to Figure 1 、 Figure 2 、 Figure 3 、 Figure 5 、 Figure 6 and Figure 7 , the first embodiment of the present application provides three kinds of denoising algorithm RTL deployment, including:

[0078] A parameter compatible high-performance FPGA image denoising method significantly improves the execution efficiency and resource utilization of hardware image denoising algorithm, and can automatically complete customized circuit design according to user needs, and can be integrated as a sub-module into other image processing systems. The patent design instructions and use process are as shown in Figure 1 .

[0079] Step S1, image denoising method details; the present application uses the development method of RTL to complete the FPGA deployment of image denoising algorithm, which needs to complete the underlying circuit design of denoising algorithm according to the algorithm principle and optimization target (such as resource consumption, inference efficiency); considering that different denoising algorithms have different filtering effects on specific noise types (such as Gaussian filter and median filter, which are good for removing Gaussian noise and salt and pepper noise respectively), the present application realizes three denoising algorithms of mean filter, Gaussian filter and median filter in parallel, which meets the needs of various noise scenes; after simulation verification to determine that the design is correct, define external interface and configuration parameters by yourself, and encapsulate all designs into an IP core. Users can specify the filter type to select the corresponding function;

[0080] Step S2, IP core parameter configuration; users can configure the image denoising IP core according to their needs. The optional parameters include filter type, data precision, sliding window size, image size, image channel number, input pixel point delay, etc.

[0081] Step S3, system comprehensive optimization; after determining the IP core parameters, the underlying circuit details are automatically optimized by using the Vivado synthesis tool to further improve the circuit performance, and finally the circuit netlist and IP core calling template are generated;

[0082] Step S4, the user calls the generated image denoising IP core in the design to complete the image denoising process.

[0083] The circuit design of the filtering algorithm is composed of sub-modules, and some key sub-modules are introduced.

[0084] (1) Shift register and register array

[0085] The traditional filtering algorithm uses a fixed-size sliding window to obtain the data selection domain around each pixel point in turn, and uses a weighted (such as mean filtering, Gaussian filtering) or median selection (such as median filtering) algorithm in the selection domain to reduce noise interference. The pixel points are input uniformly in row and column order, and the application designs a shift register and register array for sequentially obtaining each data selection domain, supporting any window size, and the structure is as shown in Figure 2 . K is the length of the sliding window, each shift register stores a row of pixel points, and (K-1) shift registers are cascaded, and the internal pixel points move one position when the enable signal is valid. The K pixel points output are delayed by the register array to obtain the entire data selection domain.

[0086] (2) Binary adder tree (BAT)

[0087] Mean filtering and Gaussian filtering both involve weighted operations, and multiplication can be replaced by more efficient shift addition. The accumulation of the multiplication result involves multiple input data, and a corresponding adder module needs to be customized. The application designs a binary adder tree to improve the accumulation efficiency, supporting any number of addend inputs, and the adder structure is as shown in Figure 3 . N addends are arranged from the root node of the binary tree upwards and added two by two, the height of the binary tree is H=log2N, and the bottom layer needs to delay (2 H -N) addends to align with other addition results, and the processing delay is the height H of the binary tree.

[0088] (3) Divider

[0089] The divider uses a pipeline design, which is the same as decimal division. The difference between the binary dividend and the divisor is taken as a pipeline level each time to get the quotient process, which can realize the parallel reasoning of multiple division operations. The pseudo code of the division operation is as shown in Figure 4 .

[0090] The embodiment also provides two deployment strategies of resource-efficient and delay-efficient for the filtering algorithm:

[0091] (1)Design idea

[0092] The mean filter and the Gaussian filter have similar calculation principles, and the difference lies in the different weights assigned to each position. The two filtering algorithms can adopt the same deployment strategy. The present application optimizes the hardware deployment of the filtering algorithm from the resource consumption and the inference delay, and proposes a resource-efficient type (RET, Resource-Efficient Type) and a latency-efficient type (LET, Latency-Efficient Type) deployment strategy, the specific details of which are described below.

[0093] RET uses as few resources as possible to implement the two filtering algorithms, and is suitable for scenarios with limited resources and low requirements for processing delay. The processing flow is shown in Figure 5 After determining the data selection domain, the BAT is directly entered to complete the accumulation operation, and the BAT is reused in each window during filtering. The entire RET only includes a shift register and a register array, a BAT, and a divider. The pixel point duration should not be less than the operation time of the BAT, otherwise data overlap will occur, resulting in incorrect operation results.

[0094] LET has higher processing efficiency than RET and is suitable for scenarios with high requirements for processing delay. In RET, the input of the BAT is always K 2 pixel points, which requires clock cycles. It is observed that the overlapping part between the selection domains constitutes redundant calculations of different sizes, Figure 6 and five selection domain overlap modes are summarized. In fact, most pixel points contribute K 2 times in the adjacent window, so only the third and fifth overlap modes need to be processed, and the redundant calculations in both horizontal and vertical directions in these two modes are completely eliminated. The core idea of eliminating redundant calculations is "multiple use of partial sums", and a suitable size of the cache unit is constructed to temporarily store the partial row sum or partial column sum of the current window for reuse by the next row or column window. Considering that it is difficult to implement redundant calculations in both horizontal and vertical directions in engineering, and the benefit of implementing redundant calculations in the second direction is relatively low (the "marginal benefit" problem, the benefit from K -1 to K -2 is higher than the benefit from 1 to K -1 , especially when K increases), the present application only eliminates the horizontal redundant calculations in the third and fifth modes, i.e., reusing the row sum in the column direction, which can reduce the proportion of redundant calculations from 1 to K -1 . Each sliding window only needs to calculate the row sum of the new row, and the row sums of the previous (K-1) rows in the window have been generated in the previous rows and passed to the window through the buffer. The addend entering the BAT is reduced to (2K-1), achieving the effect of accelerating the mean filter.

[0095] The design flow of LET is as followsFigure 7 As shown, taking a 5x5 sliding window as an example, the Triangle FIFO Group outputs 5 row sums in parallel in a sliding window, and each FIFO is used to store all row sums of a single row in the sliding window. In the first row of the sliding window, each BAT is responsible for calculating a row sum, and the rest of the BATs input the results into the (K-1) FIFOs in the first column, which are used for subsequent sliding window reuse. The number of single-row FIFOs in the Triangle FIFO Group represents the reuse times of the row sum, and the reuse times of the first (K-1) rows gradually increase, and the reuse times of each row thereafter are (K-1). When a new row arrives, the last BAT calculates a new row sum and inputs it into the FIFO in the last row for reuse. The valid FIFO outputs of the sliding window in each row are given in the upper right corner of the Triangle FIFO Group, the black circles represent invalid outputs, the red circles output the old (K-1) row sums, and the green circles represent the new row sum, which is input into the BAT for accumulation and division to obtain the final result.

[0096] Embodiment 2, refer to Figure 8 As a second embodiment of the present application, the embodiment provides a Gaussian kernel scaling strategy, which includes:

[0097] The elements in the Gaussian kernel follow a two-dimensional Gaussian distribution (with a mean of 1 and a standard deviation σ specified by the user), as shown in equation (1), where the integers x and y are the distances from the center pixel in the horizontal and vertical directions, respectively, and the Gaussian kernel is obtained after normalization. According to the instructions provided by OpenCV, when the standard deviation is not specified, the standard deviation can be automatically calculated according to equation (2), where kernel_size is the sliding window size, Figure 8 Gaussian kernels of three window sizes are given. It can be seen that there is a certain proportional relationship between the elements in the rows, so the LET strategy can still be used for row sum reuse. The special thing is that before reuse, a multiplier or divider is needed to complete the row sum scaling according to the proportional relationship.

[0098]

[0099] σ = 0.3 × ((kernel_size-1) × 0.5-1) + 0.8 (2)

[0100] Embodiment 3, refer to Figure 9 As a third embodiment of the present application, the embodiment provides a hierarchical median filter model library, which includes:

[0101] The pixel peak or valley appears in the area polluted by noise in the image, the mean filter and the Gaussian filter assign fixed weight to each pixel in the sliding window by averaging, which reduces the influence of noise, and the median filter selects the median to reduce the noise interference. With the increase of the size of the sliding window, the number of pixel points in the window increases, and the selection of the pixel median needs to consume more time and resources. In practical application, the 3*3 size window can meet the denoising needs of most scenes, and the application establishes a hierarchical median filter model library based on the modular design idea, divides the median filter into multiple sub-modules according to the input, and each sub-module contains multiple structures, which are respectively used for input number sorting in different scenes. For example, the four-input sorting module contains three kinds of four-input unordered, two-input ordered and three-input ordered, which need to take different structures to optimize resource consumption.

[0102] The hierarchical median filter model library is as shown in Figure 9As shown, each median filter is divided into multiple pipeline stages to improve the timing performance of the circuit. Specifically, the Sort-2 Unit is composed of a comparator, a NOT gate, and two 2-1 MUXs, and has a simple structure and only one type of construction. The Median-3 Unit has two types of construction, i.e., three-input unordered and two-input ordered, and the latter reduces one Sort-2 Unit. The Median-3-imp Unit is an optimized construction of the Median-3 Unit, and this module has less resource consumption. The Sort-4 Unit has three types of construction, i.e., four-input unordered, two-by-two ordered, and three-input ordered, and the four-input unordered is the basic construction. The Sort-4-simp-2 Unit reduces one Sort-2 Unit, and the Sort-4-simp-1 Unit has a simpler structure and only needs one Sort-2 Unit and one Median-3-simp Unit. The Median-5 Unit includes two types of construction, i.e., “2+1+2” (Median-5-1 Unit) and “3+2” (Median-5-2 Unit), and the better module is selected according to the performance. The Median-7 Unit has two types of construction, i.e., “4+3” and “2+3+2”, and two types of input arrangement, i.e., seven-input unordered and two groups of two-input ordered, and the four types of construction are used for different scenarios. The final Meidan-9 Unit can use the schemes of “3+3+3”, “4+1+4”, “5+4”, and “7+2” according to the internal construction, and the optimal sub-module construction is composed of the Median-3-imp Unit and the Median-7-simp-1 Unit (“4+3” with two groups of two-input ordered).

[0103] Embodiment 4, with reference to Figure 13 、 Figure 14 and Figure 15 , is a fourth embodiment of the present application, which provides a comparison between the RET and the LET performance, including:

[0104] (1) Denoising effect demonstration

[0105] Uniformly inject Gaussian noise and salt and pepper noise into the example picture, as shown in Figure 10 . Taking a 3x3 sliding window as an example, Figure 11 and Figure 12The denoising effect figures of three filtering algorithms for Gaussian noise and salt and pepper noise are respectively given. For Gaussian noise, the denoising effect of Gaussian filtering is better, and the picture processed by mean filtering has partial distortion (the white area is balanced by the surrounding pixels); for salt and pepper noise, the denoising effect of median filtering is better, and most of the noise influence can be removed.

[0106] (2) Hardware acceleration effect demonstration

[0107] The program running time of the software end can be obtained by using the time function. Since the pixel points flow uniformly, ALU_TIME is the inference delay of a single pixel point (the ALU_TIME of the three filtering algorithms is shown in formula (3), which is composed of BAT, divider and register, and DATA_WD is the data precision). ΔT is the transmission time of a single pixel point. In order to not produce data overlap, the minimum value of ΔT is shown in formula (4). H and W are the length and width of the picture respectively, and the transmission time of all pixel points is ΔT×H×W, which is divided by the clock frequency f to obtain the single frame inference delay Latency (shown in formula (5)). fps can be obtained from Latency, as shown in formula (6). According to the test picture size, H and W are set to 937 and 968 respectively.

[0108]

[0109] Latency=(ALU_TIME+ΔT×H×W) / f (5)

[0110]

[0111] Taking Gaussian noise as an example, the fps acceleration ratio (hardware fps / software fps) of the three filtering algorithms for different sliding windows is shown in formula (7). The present application only implements 3*3 median filtering, and the acceleration ratios of median filtering with K=5 / 7 / 9 are all set to 1. The implementation of hardware Gaussian filtering only contains hardware-friendly splicing and shift calculation, and the timing performance is the best, and the acceleration ratio is more than 2.00x, and the acceleration effect is more obvious with the increase of K. The mean filtering contains a divider IP, and the inference efficiency is not as good as that of Gaussian filtering, but the acceleration effect is still improved with the increase of K. Figure 13

[0112] (3) Performance evaluation of RET and LET

[0113] The advantages of RET and LET are resource consumption and inference efficiency respectively, and the resource consumption and inference efficiency of the two are compared as follows. Figure 14 ​The resource consumption of RET and LET under four sliding windows is given, and the four modules are named X_Y (X and Y represent the filtering algorithm and the use strategy, respectively). For the same filtering type, the resource consumption of RET is significantly less than that of LET, and no BRAM is consumed. For Gaussian filtering using LET, when K = 5 / 7 / 9, the scaling coefficients are not all powers of 2, and a divider is needed to complete the row and scaling. The scaling coefficient increases with the value of K, and the LUT resource consumption of the divider increases exponentially.

[0114] The fps comparison of LET and RET in mean filtering and Gaussian filtering is shown in Figure 15 For mean filtering, the fps of LET is higher than that of RET, but the acceleration ratio gradually decreases as the value of K increases. For Gaussian filtering, when K = 3, there is still an acceleration ratio of 1.73x, but as the value of K increases, the inference efficiency of LET is worse than that of RET. The reason is that the consumption of a large number of LUT resources blocks the clock constraint, reducing the timing performance. According to the measurement results, when K = 5 / 7 / 9, the maximum clock frequency is only 10MHz, which is difficult to meet the daily use demand of FPGA.

[0115] In summary, the expansion of the sliding window offsets the inference efficiency advantage of the LET strategy. Considering that the filtering algorithm with K = 3 is used most frequently in practical engineering applications, the filtering optimization strategy proposed in this application can meet the application demand to some extent, and the corresponding strategy can be selected according to the actual demand. If you want to use the LET strategy to deploy Gaussian filtering with a larger sliding window, you need to approximate the row scaling coefficients of the Gaussian kernel to powers of 2, and replace the multiplication and division of row and scaling with hardware-friendly shift operations.

[0116] (4) Performance analysis of hierarchical median filtering model library

[0117] According to the input number and the applicable conditions, the sub-modules are classified, and the performance analysis results obtained by Vivado synthesis are shown in Table 1. For the timing performance of the two-input comparator, when the clock frequency is set to 100MHz, WNS and WHS both show NA. After viewing the layout of the comparator, it can be known that all the registers are absorbed into the combinatorial logic, and the result has no path to be analyzed for cycle constraint, so it shows NA. For the median filtering with a 3x3 window size, due to the overlapping part in the window sliding process, the input number of the overlapping part remains unchanged in the adjacent window. To speed up the median filtering sorting, the optimal sub-module design is used in the final nine-input median filter, and the resource consumption and timing performance are shown in Table 1.

[0118] Table 1: Performance analysis of hierarchical median filtering model library

[0119]

[0120]

[0121] Embodiment 5, refer to Figure 16 and Figure 17 , which is the fifth embodiment of the present application, provides FPGA image denoising system integration, including:

[0122] (1) IP encapsulation and interface description

[0123] As shown in Figure 16 , three image filtering algorithms are integrated into a module and encapsulated as an IP core. The left side is the input port and the right side is the output port. The external interface and IP core parameter configuration information are shown in Table 2. Users can configure IP core parameters according to actual needs, and after protocol conversion through the transceiver port, they can be integrated into other systems, significantly improving development efficiency.

[0124] Table 2: External interface and IP core parameter configuration information

[0125]

[0126] (2) Image denoising system architecture

[0127] The image denoising module is integrated into the FPGA development board, and the data transceiver interface is reserved for the host computer, and other peripherals and data transceiver modules are added to form a complete image denoising system as shown in Figure 17 . The denoising module input data can be collected by the FPGA sensor or directly sent by the host computer, and the output data can be directly presented on the display screen or sent to the host computer. The FPGA part includes four modules: image acquisition, image denoising, result display, data transceiver and format conversion. OV2640 is used to collect images, and the collected image data is temporarily stored in the external DDR3 chip. After the collection is completed, the data is read from the DDR3, and the format conversion unit is converted to AXI4 data format. The "user-AXI4 interface conversion" module is used to convert the AXI4 format data to the IP core interface format, and input the image denoising module to complete the denoising process. The output data is stored in the cache unit, waiting for the whole image processing to be completed. The display screen uses HDMI interface, which needs to convert the output data format and level.

[0128] The present application is applicable to edge vision preprocessing, such as automatic driving, remote sensing, etc., and has promotional value for related industries.

[0129] In summary, the present application adopts two kinds of filtering algorithm resource efficient and delay efficient deployment strategy: for mean filtering and Gaussian filtering with similar denoising principle, respectively from resource consumption and inference efficiency optimization, users can select the corresponding strategy according to the available resources and speed requirements;

[0130] For the median filtering algorithm, based on the modular design idea, the sorting sub-modules are constructed in layers according to the input conditions, and the optimal sorting structure is selected in each layer to build the top filtering module. Compared with other circuit design schemes, the median filter of the application has optimal resource consumption and reasoning delay, and users can build more complex median filters according to the model library;

[0131] All modules are packaged as IP cores by parameterized design, supporting multiple parameters such as image size, data precision, pixel delay, filtering type, etc. Users can change parameters as needed in the configuration interface to realize flexible design of the image denoising system.

[0132] The packaged IP cores are integrated into the FPGA development board, and a data transmission interface is reserved for the host computer. A complete FPGA image denoising system including four modules of image acquisition, image denoising, result display, data transmission and format conversion is formed by matching a high-speed bus.

[0133] Compared with the prior art, the FPGA circuit design of three image denoising algorithms is optimized, the hardware resource consumption is saved, and the reasoning efficiency of the image denoising process is improved. The packaged IP core can be integrated as a sub-module into other image processing systems, and users can flexibly configure the external interface according to the application requirements.

[0134] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. An image preprocessing method based on FPGA, characterized in that, include, Step S1: Based on the RTL development approach, three image denoising algorithms—mean filtering, Gaussian filtering, and median filtering—are deployed in parallel on the FPGA. Underlying optimization circuits are designed for each algorithm, and the corresponding filtering algorithm is selected according to the noise type. Step S2: The denoising algorithm module is encapsulated as a parameterized IP core, and the configuration parameters include filter type, sliding window size, data precision, image size and pixel delay. Step S3: Optimize the circuit using synthesis tools and generate a netlist and IP core calling template; Step S4: Call the IP core to complete the image denoising process; Among them, mean filtering and Gaussian filtering are optimized in terms of resource consumption and inference efficiency, respectively. Users can select the corresponding deployment strategy according to available resources and speed requirements. The deployment strategy includes resource-efficient RET and latency-efficient LET deployment strategies. The mean filtering and Gaussian filtering adopt the resource-efficient RET or latency-efficient LET deployment strategy. The median filtering is built based on a hierarchical model library, and all modules support parameterized configuration. The resource-efficient RET deployment strategy includes: For mean filtering and Gaussian filtering, a single shift register, register array, binary tree adder and divider are used to implement weighted calculation within the sliding window; The binary tree adder accumulates all pixels within the window, and the divider performs normalization. The duration of the pixel is greater than the accumulation operation cycle; The time-efficient LET deployment strategy includes: Construct row and cache units to store the row sums of the first K-1 rows within the sliding window; Calculate the sum of the current row for each newly added row and sum it with the cached row sum to eliminate horizontal redundant calculations; For Gaussian filtering, the cached lines are scaled and then accumulated according to the line ratio. The construction of the hierarchical median filtering model library includes: The median filter is divided into multiple sub-modules, and the optimal sorting structure is selected for each layer based on the input conditions. The submodule includes sorting units with 2 to 9 inputs, and different structures are used for input ordering to reduce resource consumption.

2. The FPGA-based image preprocessing method as described in claim 1, characterized in that, The parameterized configuration includes: The filter type, window size, image size, and data precision can be dynamically set through an external interface. The appropriate shift register length, adder hierarchy, and sorting module structure are automatically generated based on the configuration parameters. Supports multi-channel input / output and expansion with different fill types.

3. The FPGA-based image preprocessing method as described in claim 1, characterized in that, Linearity and scaling are specifically as follows: When the Gaussian kernel row spacing ratio is a power of 2, a shift operation is used instead of multiplication and division. When the ratio is not a power of 2, a pipelined divider is inserted to complete the row and scaling.

4. An FPGA-based image preprocessing system, based on the FPGA-based image preprocessing method according to any one of claims 1 to 3, characterized in that, include: The image acquisition module is used to acquire raw image data; The image denoising module performs denoising processing on the acquired data; The data transceiver module enables data transmission and protocol conversion between the FPGA and the host computer. The result display module outputs the denoised image to the display screen. The image denoising module is packaged as an IP core, integrated on an FPGA development board, and connected to other modules via an AXI4 bus.

5. The FPGA-based image preprocessing system as described in claim 4, characterized in that, The IP core interface of the image denoising module includes: Input ports: clock signal, reset signal, data valid pulse, and multi-channel pixel data; Output port: Valid pulses of the denoised data and multi-channel result data; Configure the port: filter mode, window size, sliding step size, and fill type parameters.

6. The FPGA-based image preprocessing system as described in claim 4, characterized in that, The image acquisition module uses a 2-megapixel CMOS image sensor and temporarily stores data through a DDR3 chip; the result display module achieves level and format conversion through an HDMI interface.

Images