Real-time image fusion method and system based on FPGA chip

Abstract

The application discloses a real-time image fusion method and system based on an FPGA chip, and the scheme gives a sliding window buffer circuit based on a pipeline real-time operation, and innovatively designs two filters suitable for implementation in the FPGA, namely, an image feature extraction circuit, from the characteristics that the FPGA is high in fixed-point operation efficiency, and converts time-consuming and resource-consuming operations in the filter into more efficient equivalent operations.The scheme realizes real-time parallel image fusion processing in the FPGA chip, and can guarantee image quality while making the operation of high-resolution and high-frame-rate image fusion achieve high real-time performance.

Description

Technical Field

[0001] This invention relates to a method for processing image data, and more specifically to a real-time image fusion scheme. Background Technology

[0002] In existing image processing technologies, image fusion is mostly achieved using DSP coprocessor modules or by using external DDR chips to implement a full-page burst caching mode. These existing solutions cannot meet the latency requirements of real-time image processing under high resolution and high frame rate conditions; furthermore, the use of DDR chips as frame buffers in existing solutions also affects the maximum data bandwidth.

[0003] In addition, most existing solutions are designed for specific image sensor resolutions and frame rates (i.e., fixed data bandwidth), resulting in poor scalability. Summary of the Invention

[0004] To address the problems of real-time performance and scalability in existing image fusion technologies, this invention provides a real-time image fusion method based on an FPGA chip, and a real-time image fusion system based on an FPGA chip. The solution of this invention implements real-time parallel image fusion processing based on an FPGA, which can achieve high real-time performance for high-resolution and high-frame-rate image fusion operations while ensuring image quality.

[0005] To achieve the above objectives, the present invention provides a real-time image fusion method based on an FPGA chip, comprising:

[0006] First, the pixel streams of the first and second images to be fused are input into the corresponding filters in parallel;

[0007] Next, median filtering and Gaussian filtering are performed simultaneously on the first image to be fused to obtain the first base layer and the second base layer of the first image; at the same time, median filtering and Gaussian filtering are performed simultaneously on the second image to obtain the first base layer and the second base layer of the second image.

[0008] Next, calculations are performed on the original image of the first image and the first base layer and second base layer of the first image to form a first detail layer and a second detail layer for the first image; at the same time, calculations are performed on the original image of the second image and the first base layer and second base layer of the second image to form a first detail layer and a second detail layer for the second image.

[0009] Finally, the base layer and detail layer of the first image are fused with the base layer and detail layer of the second image to achieve the fusion between the first image and the second image.

[0010] In some embodiments of the present invention, the first image is a visible light image and the second image is an infrared image.

[0011] To achieve the above objectives, the present invention provides a real-time image fusion system based on an FPGA chip. The real-time image fusion system forms a first sliding window buffer, a second sliding window buffer, and a first FPGA chip within the FPGA chip. B Filter, first F M Filter, second F B Filter, second F M Filters, multi-stage arithmetic units, and parallel adders;

[0012] The first sliding window buffer and the second sliding window buffer respectively acquire the pixel streams of the first image and the second image to be fused, and input them in parallel into the first F... B Filter, first F M Filter, second F B Filter, second F M In the filter;

[0013] The first F B Filter and first F M The filters are connected to the first sliding window buffer and simultaneously perform Gaussian filtering and median filtering on the input first image pixel stream to obtain the second base layer and the first base layer of the first image.

[0014] The second F B Filter and second F M The filters are connected to the second sliding window buffer respectively, and Gaussian filtering and median filtering are performed synchronously on the input second image pixel stream to obtain the second base layer and the first base layer of the second image.

[0015] The multi-level computing unit and the first F B Filter, first F M Filter, second F B Filter, and the second F M The filter data connection performs multi-level calculations on the first base layer and the second base layer of the synchronously formed first image, and the first base layer and the second base layer of the second image to form a linear detail layer and a non-linear detail layer for the first image, and a linear detail layer and a non-linear detail layer for the second image.

[0016] The parallel adder is connected to the multi-level arithmetic unit to synchronously acquire the linear detail layer and non-linear detail layer of the first image and the linear detail layer and non-linear detail layer of the second image processed by the multi-level arithmetic unit, and performs fusion processing on the first image and the second image.

[0017] In some embodiments of the present invention, the first F B Filter or second F B The filter performs Gaussian convolution operations by shifting the input image pixel stream.

[0018] In some embodiments of the present invention, the first sliding window buffer or the second sliding window buffer consists of two FIFOs and nine buffers.

[0019] In some embodiments of the present invention, the first F M Filter or second F M The filter is composed of several sorting circuit units.

[0020] In some embodiments of the present invention, the multi-level arithmetic unit includes a first arithmetic unit, a second arithmetic unit, a third arithmetic unit, and a fourth arithmetic unit;

[0021] The first arithmetic unit and the first F B Filter, first F M Data connections to the filter, the third arithmetic unit, and the parallel adder;

[0022] The second arithmetic unit and the second F B Filter, second F M Data connections to the filter, the third arithmetic unit, and the parallel adder;

[0023] The third arithmetic unit is data-connected to the first arithmetic unit and the second arithmetic unit, respectively.

[0024] The fourth arithmetic unit and the first F B Filter, second F B Data connections for filters and parallel adders.

[0025] The solution provided by this invention implements real-time parallel image fusion processing entirely within an FPGA chip, which can ensure image quality while achieving high real-time performance for high-resolution and high-frame-rate image fusion operations.

[0026] The solution provided by this invention can avoid the problems that arise when using other devices such as DSP, GPU, and DDR in core algorithms and image processing operations, such as low efficiency, higher energy consumption, limited bandwidth, and high replacement costs.

[0027] The solution provided by this invention has the advantages of high precision, low latency, good scalability, strong implementability, and wide application compared with the prior art. Attached Figure Description

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 This is an example diagram illustrating the configuration of a real-time image fusion system based on an FPGA chip in this invention.

[0030] Figure 2 This is an example diagram illustrating the structure of the sliding window buffer in an embodiment of the present invention;

[0031] Figure 3 This is an example diagram of the window formed by the sliding window buffer in an embodiment of the present invention;

[0032] Figure 4 In the example of this invention, F M Example diagram of filter circuit structure;

[0033] Figure 5 This is a structural example diagram of the sorting circuit unit in an embodiment of the present invention;

[0034] Figure 6 In the example of this invention, F B Example diagram of filter circuit structure;

[0035] Figure 7 In the example of this invention, F B Example diagram of convolution kernel in a filter. Detailed Implementation

[0036] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0037] The real-time image fusion method based on FPGA chip presented in this invention is based entirely on FPGA chip to perform real-time parallel image fusion processing. It can achieve high real-time performance of high-resolution and high-frame-rate image fusion operation while ensuring the quality of output image.

[0038] This real-time image fusion method based on FPGA chip performs median filtering and Gaussian filtering on the image to be fused using the FPGA chip. Based on this, multiple base layers and detail layers of the image to be fused are formed. Then, based on the multiple base layers and detail layers of the image to be fused, high-quality fusion of the images is completed.

[0039] Based on the above mechanism, the real-time image fusion based on the FPGA chip in this invention is mainly implemented through the following process:

[0040] First, the pixel streams of the first image and the second image to be fused are input into the corresponding filters in parallel. Median filtering and Gaussian filtering are performed simultaneously on the first image to be fused to obtain the first base layer and the second base layer of the first image. At the same time, median filtering and Gaussian filtering are performed simultaneously on the second image to obtain the first base layer and the second base layer of the second image.

[0041] Next, based on this, calculations are performed on the original image of the first image and the first base layer and second base layer of the first image to form a first detail layer and a second detail layer for the first image;

[0042] At the same time, calculations are performed simultaneously on the original image of the second image and the first and second base layers of the formed second image to form the first and second detail layers of the second image.

[0043] Finally, the base layer and detail layer of the first image are fused with the base layer and detail layer of the second image to achieve the fusion between the first image and the second image.

[0044] To address the real-time image fusion method based on FPGA chips presented in this invention, a real-time image fusion system is constructed. This system is entirely based on FPGA chips and can implement the aforementioned real-time image fusion scheme to perform real-time parallel image fusion processing, enabling the computation to achieve the highest real-time performance.

[0045] Specifically, this real-time image fusion system innovatively presents a sliding window buffer circuit based on pipelined real-time computation, thereby solving the problem of existing DDR chips being used as frame buffers, which affects the maximum data bandwidth.

[0046] Furthermore, this real-time image fusion system innovatively designs two filters suitable for implementation in FPGA, specifically addressing the characteristics of image fusion. These filters, namely image feature extraction circuits, leverage the high efficiency of FPGAs for fixed-point computations to transform the time-consuming and resource-intensive operations in the filters into more efficient equivalent operations. This further enhances real-time performance and saves computational resources.

[0047] In practical applications, this real-time image fusion system can effectively fuse infrared and visible light images, and avoid using multiple layers of image feature extraction and related processing algorithms on the FPGA, thereby achieving the highest real-time performance.

[0048] See Figure 1 The diagram shows an example configuration of the real-time image fusion system provided by the present invention.

[0049] As shown in the figure, this real-time image fusion system forms a first sliding window buffer 110, a second sliding window buffer 120, and a first F in the FPGA chip. B Filter 130, First F M Filter 140, Second F M Filter 150, Second F B Filter 160, multi-stage arithmetic unit and parallel adder 199.

[0050] Specifically, the first sliding window buffer 110 acquires the pixel stream of the first image to be fused and inputs it in parallel into the first F... B Filter 130 and the first F M Filter 140.

[0051] The second sliding window buffer 120 in this system acquires the pixel stream of the second image to be fused and inputs it in parallel to the second F. B Filter 160 and the second F M Filter 150.

[0052] In this system, based on the cooperation of the first sliding window buffer 110 and the second sliding window buffer 120, the input between the pixel streams of the first image and the second image is adjusted, and the obtained pixel streams of the first image and the second image to be fused are synchronously input into the corresponding F. B Filter and F M In the filter, to ensure the quality of subsequent image fusion processing.

[0053] In practical implementation, the sliding window size of the two sliding window buffers in this system can be determined according to actual needs. Preferably, the sliding window of the sliding window buffer in this system is set to a large size, which can ensure the high quality of the image base layer during subsequent processing, thereby ensuring the "smoothness" of the image. The large size here can be determined based on the maximum size that the sliding window buffer can have.

[0054] For ease of subsequent description and understanding of the implementation process of this solution, as an example, the first image here can be a visible light image and the second image can be an infrared image.

[0055] The first F in this system B Filter 130 and the first F M Filter 140 is also data-connected to the first sliding window buffer 110.

[0056] The first F here B Filter 130 acquires the first image pixel stream transmitted by the first sliding window buffer 110 and performs Gaussian filtering to obtain the second base layer of the first image.

[0057] At the same time, the first F M Filter 140 relative to the first F B The filter 130 synchronously acquires the first image pixel stream transmitted by the first sliding window buffer 110 and performs median filtering to obtain the first base layer of the first image.

[0058] The second F in this system M Filter 150 and the second F B Filter 160 is also connected to the second sliding window buffer 120 for data transfer.

[0059] The second F here M Filter 150 acquires the second image pixel stream transmitted by the second sliding window buffer 120 and performs median filtering to obtain the first base layer of the second image.

[0060] Meanwhile, the second F B Filter 160 relative to the second F M The filter 150 synchronously acquires the second image pixel stream transmitted by the second sliding window buffer 120 and performs Gaussian filtering to obtain the second base layer of the second image.

[0061] The multi-level arithmetic units in this system and the first F B Filter 130, First F M Filter 140, Second F B Filter 160, and the second F M The filter 150 performs data connection and performs multi-level calculations on the first base layer and the second base layer of the synchronously formed first image, as well as the first base layer and the second base layer of the second image, to form a linear detail layer and a non-linear detail layer for the first image, and a linear detail layer and a non-linear detail layer for the second image.

[0062] The parallel adder 196 in this system is connected to the multi-level arithmetic unit and synchronously acquires the linear detail layer and non-linear detail layer of the first image and the linear detail layer and non-linear detail layer of the second image processed by the multi-level arithmetic unit, and performs fusion processing on the first image and the second image.

[0063] In some embodiments of the present invention, the first sliding window buffer 110 and the second sliding window buffer 120 in this system are specifically implemented by consisting of 2 FIFO (First In First Out) buffers and 9 buffers, respectively.

[0064] Specifically, such as Figure 2As shown, the sliding window buffer here consists of two FIFOs (First-In-First-Out buffers) and nine buffers. The nine buffers are divided into three groups of three, with the three buffers in each group connected in series. The inlet of the first FIFO is connected to the inlet of the first group of buffers, the outlet of the first FIFO is connected to the inlet of the second FIFO and the inlet of the second group of buffers, and the outlet of the second FIFO is connected to the inlet of the third group of buffers. Simultaneously, the nine buffers in the three groups output in parallel.

[0065] The sliding window buffer thus formed can output a 3×3 pixel value window in parallel and input it to the subsequent filters.

[0066] As an example, let the current time be t=T, then the pixel value about to enter the line buffer is p(t=T). The pixels output simultaneously form a window, such as... Figure 3 As shown; at the same time, the sliding window buffer inputs pixels in parallel to each filter circuit.

[0067] In some embodiments of the present invention, the first F in this system M Filter 140 and the second F M The filter 150 is composed of 19 sorting circuit units (SEs).

[0068] Specifically, such as Figure 4 As shown, this F M The filter has 19 sorting circuit units, each with two input terminals and two output terminals. Taking the illustrated scheme as an example, the input terminal located at the top of each sorting circuit unit is the first input terminal, the input terminal located at the bottom is the second input terminal, the output terminal located at the top is the first output terminal, and the output terminal located at the bottom is the second output terminal.

[0069] Accordingly, the two input terminals of the first sorting circuit unit 1 are set independently, the first output terminal is connected to the first input terminal of the second sorting circuit unit 2, and the second output terminal of the first sorting circuit unit 1 is connected to the first input terminal of the third sorting circuit unit 3.

[0070] The second input terminal of the second sorting circuit unit 2 is connected to the first output terminal of the third sorting circuit unit 3. The first output terminal and the second output terminal of the second sorting circuit unit 2 are respectively connected to the first input terminal of the tenth sorting circuit unit 10 and the first input terminal of the twelfth sorting circuit unit 12.

[0071] The second input terminal of the third sorting circuit unit 3 is set independently, and its second output terminal is connected to the first input terminal of the sixteenth sorting circuit unit 16.

[0072] The two input terminals of the fourth sorting circuit unit 4 are set independently, the first output terminal is connected to the first input terminal of the fifth sorting circuit unit 5, and the second output terminal of the fourth sorting circuit unit 4 is connected to the first input terminal of the sixth sorting circuit unit 6.

[0073] The second input terminal of the fifth sorting circuit unit 5 is connected to the first output terminal of the sixth sorting circuit unit 6. The first and second output terminals of the fifth sorting circuit unit 5 are respectively connected to the second input terminal of the tenth sorting circuit unit 10 and the second input terminal of the twelfth sorting circuit unit 12.

[0074] The second input terminal of the sixth sorting circuit unit 6 is set independently, and its second output terminal is connected to the first input terminal of the fifteenth sorting circuit unit 15.

[0075] The two input terminals of the seventh sorting circuit unit 7 are set independently, the first output terminal is connected to the first input terminal of the ninth sorting circuit unit 9, and the second output terminal of the seventh sorting circuit unit 7 is connected to the first input terminal of the eighth sorting circuit unit 8.

[0076] The second input terminal of the ninth sorting circuit unit 9 is connected to the first output terminal of the eighth sorting circuit unit 8. The first and second output terminals of the ninth sorting circuit unit 9 are respectively connected to the second input terminal of the eleventh sorting circuit unit 11 and the second input terminal of the fourteenth sorting circuit unit 14.

[0077] The second input terminal of the eighth sorting circuit unit 8 is set independently, and its second output terminal is connected to the second input terminal of the fifteenth sorting circuit unit 15.

[0078] The first output terminal of the tenth sorting circuit unit 10 is disconnected, and its second output terminal is connected to the first input terminal of the eleventh sorting circuit unit 11.

[0079] The first output terminal of the eleventh sorting circuit unit 11 is disconnected, and its second output terminal is connected to the first input terminal of the seventeenth sorting circuit unit 17.

[0080] The first output terminal and the second output terminal of the twelfth sorting circuit unit 12 are respectively connected to the first input terminal of the thirteenth sorting circuit unit 13 and the first input terminal of the fourteenth sorting circuit unit 14.

[0081] The first output terminal of the thirteenth sorting circuit unit 13 is disconnected, and its second output terminal is connected to the second input terminal of the seventeenth sorting circuit unit 17.

[0082] The second output terminal of the fourteenth sorting circuit unit 14 is disconnected, while the first output terminal is connected to the second input terminal of the thirteenth sorting circuit unit 13.

[0083] The second output terminal of the fifteenth sorting circuit unit 15 is disconnected, while the first output terminal is connected to the second input terminal of the sixteenth sorting circuit unit 16.

[0084] The second output terminal of the sixteenth sorting circuit unit 16 is disconnected, while the first output terminal is connected to the second input terminal of the eighteenth sorting circuit unit 18.

[0085] The first output terminal and the second output terminal of the seventeenth sorting circuit unit 17 are respectively connected to the first input terminal of the nineteenth sorting circuit unit 19 and the first input terminal of the eighteenth sorting circuit unit 18.

[0086] The second output terminal of the eighteenth sorting circuit unit 18 is disconnected, while the first output terminal is connected to the second input terminal of the nineteenth sorting circuit unit 19.

[0087] The first output terminal of the nineteenth sorting circuit unit 19 is disconnected, and its second output terminal is set independently.

[0088] For further details on the configuration of the sorting circuit unit SE in this example, please refer to [link / reference]. Figure 5 Each sorting circuit unit SE here is mainly composed of a corresponding comparator. Given two input pixel values ​​A and B, such a sorting circuit unit will output the larger pixel value at H and the smaller pixel value at L.

[0089] This constitutes F M The filter selects the median from 9 parallel input pixel values ​​using 19 sorting elements (SEs) and outputs it, as shown in the following formula:

[0090]

[0091] In some embodiments of the present invention, the first F in this system B Filter 130 and the second F B Filter 160 is specifically a corresponding Gaussian filter.

[0092] Specifically, such as Figure 6 As shown, this F B The filter is mainly composed of corresponding parallel adders, and the specific configuration scheme will not be described in detail here.

[0093] This constitutes F B The filter takes the nine pixel values ​​that are sequentially and parallelly input into the sliding window buffer, performs Gaussian convolution operations through shifting operations, and outputs the data.

[0094] As an example, this can be achieved using the following formula:

[0095]

[0096] Where a = b = 1, the F B The convolution kernels in the filter are all composed of powers of 2, such as... Figure 7 As shown, this can transform the multiplication and division operations in convolution into more efficient shift operations, which can greatly improve computational efficiency and reduce the demand for computing power.

[0097] In some embodiments of the present invention, the multi-level arithmetic unit in this system includes a first buffer 171, a second buffer 172, a third buffer 173, a fourth buffer 174, a fifth buffer 175, a sixth buffer 176, a first subtraction operator 177, a second subtraction operator 178, a third subtraction operator 179, a fourth subtraction operator 180, a first addition operator 181, a seventh buffer 182, an eighth buffer 183, a ninth buffer 184, a first division operator 185, a second division operator 186, a tenth buffer 187, an eleventh buffer 188, a twelfth buffer 189, a thirteenth buffer 190, a first multiplication operator 191, a second multiplication operator 192, a third multiplication operator 193, a fourth multiplication operator 194, a fourteenth buffer 195, a fifteenth buffer 196, a sixteenth buffer 197, and a seventeenth buffer 198.

[0098] The first buffer 171 and the sixth buffer 176 are connected to the system's data port, respectively, and are used to store the input original first image pixel stream and second image pixel stream to be fused.

[0099] The second buffer 172 and the third buffer 173 are respectively connected to the first F B Filter 130 and the first F M Filter 140 is used for data connection, and is used to store data after passing through the first F. B The second base layer of the first image formed by Gaussian filtering with filter 130, and the image after passing through the first F... M The first base layer of the first image is formed by median filtering of filter 140.

[0100] The fourth buffer 174 and the fifth buffer 175 are respectively connected to the second F M Filter 150 and the second F B Filter 160 is used for data connection, and is used to store data after passing through the second F. M The first base layer of the second image, formed by median filtering of filter 150, is then processed by the second F... B The first base layer of the second image is formed by a 160 Gaussian filter.

[0101] The first subtraction operator 177 is connected to the first buffer 171 and the second buffer 172. It performs a subtraction operation based on the original image data of the first image in the first buffer 171 and the second base layer data of the first image in the second buffer 172. That is, the second base layer of the first image is subtracted from the original image of the first image to obtain the linear detail layer of the first image.

[0102] The second subtraction operator 178 is connected to the second buffer 172 and the third buffer 173. The subtraction operation is performed based on the second base layer data of the first image in the second buffer 172 and the first base layer data of the first image in the third buffer 173 to obtain the non-linear detail layer of the first image.

[0103] The third subtraction operator 179 is connected to the sixth buffer 176 and the fifth buffer 175. It performs a subtraction operation based on the original image data of the second image in the sixth buffer 176 and the second base layer data of the second image in the fifth buffer 175. That is, the second base layer of the second image is subtracted from the original image of the second image to obtain the linear detail layer of the second image.

[0104] The fourth subtraction unit 180 is connected to the fourth buffer 174 and the fifth buffer 175. It performs a subtraction operation based on the first base layer data of the second image in the fourth buffer 174 and the second base layer data of the second image in the fifth buffer 175 to obtain the non-linear detail layer of the second image.

[0105] The first adder 181 is connected to the second subtractor 178 and the fourth subtractor 180 respectively, and is used to add the nonlinear detail layer of the first image to the nonlinear detail layer of the second image.

[0106] The seventh buffer 182, the eighth buffer 183, and the ninth buffer 184 are connected to the second subtraction unit 178, the first addition unit 181, and the fourth subtraction unit 180, respectively. They are used to store the data calculated by the second subtraction unit 178, the first addition unit 181, and the fourth subtraction unit 180, and to perform clock control to ensure the time synchronization of pixel values.

[0107] The first division unit 185 is connected to the seventh buffer 182 and the eighth buffer 183 respectively to perform division operations on the data cached in the seventh buffer 182 and the eighth buffer 183, wherein the data stored in the seventh buffer 182 is used as the dividend and the data stored in the eighth buffer 183 is used as the divisor, and the result of the operation is output to the eleventh buffer 188.

[0108] The second division unit 186 is connected to the eighth buffer 183 and the ninth buffer 184 respectively to perform division operations on the data cached in the eighth buffer 183 and the ninth buffer 184, wherein the data stored in the ninth buffer 184 is used as the dividend and the data stored in the eighth buffer 183 is used as the divisor, and the result of the operation is output to the twelfth buffer 189.

[0109] The tenth buffer 187 is connected to the first subtraction unit 177 to store the linear detail layer of the first image calculated by the first subtraction unit 177.

[0110] The thirteenth buffer 190 is connected to the third subtraction unit 179 to store the linear detail layer of the second image calculated by the third subtraction unit 179.

[0111] Eleventh buffer 188 and twelfth buffer 189 are connected to the first division unit 185 and the second division unit 186 respectively, and store the pixel data after division operations by the first division unit 185 and the second division unit 186 respectively. The corresponding delay is set according to the data processing period of the first division unit 185 and the second division unit 186, so that the data in the eleventh buffer 188 and the twelfth buffer 189 are output synchronously.

[0112] The first multiplier 191 is connected to the tenth buffer 187 and the eleventh buffer 188 respectively to perform multiplication on the pixel data buffered in the tenth buffer 187 and the eleventh buffer 188, and output the result to the fourteenth buffer 195.

[0113] The second multiplier 192 is connected to the second buffer 172, performs the corresponding multiplication operation on the pixel data stored in the second buffer 172, and outputs the result of the operation to the fifteenth buffer 196.

[0114] The third multiplier 193 is connected to the fifth buffer 175 to perform the corresponding multiplication operation on the pixel data stored in the fifth buffer 175, and outputs the result of the operation to the sixteenth buffer 197.

[0115] The fourth multiplication unit 194 is connected to the twelfth buffer 189 and the thirteenth buffer 190 respectively to perform multiplication operations on the pixel data buffered in the twelfth buffer 189 and the thirteenth buffer 190, and output the result of the operation to the seventeenth buffer 198.

[0116] The fourteenth buffer 195, the fifteenth buffer 196, the sixteenth buffer 197, and the seventeenth buffer 198 are connected to the first multiplication unit 191, the second multiplication unit 192, the third multiplication unit 193, and the fourth multiplication unit 194, respectively, to store the pixel data processed by the multiplication operations of the first multiplication unit 191, the second multiplication unit 192, the third multiplication unit 193, and the fourth multiplication unit 194. The corresponding delay is set according to the data processing period of the first multiplication unit 191, the second multiplication unit 192, the third multiplication unit 193, and the fourth multiplication unit 194, so that the data in the fourteenth buffer 195, the fifteenth buffer 196, the sixteenth buffer 197, and the seventeenth buffer 198 are output synchronously.

[0117] The buffer here consists of shift registers, controlled by a unified global clock, used to keep the pixel values ​​synchronized across different computational stages.

[0118] In some embodiments of the present invention, the parallel adder 199 in this system is connected to the fourteenth buffer 195, the fifteenth buffer 196, the sixteenth buffer 197, and the seventeenth buffer 198 respectively, and synchronizes the pixel data sent synchronously by the fourteenth buffer 195, the fifteenth buffer 196, the sixteenth buffer 197, and the seventeenth buffer 198; thereby, the received four sets of pixel data are further added to achieve image fusion processing.

[0119] The specific configuration of this parallel adder 199 will not be detailed here, and can be determined according to actual needs. As an example, two sets of adders can be stacked to form the corresponding parallel adder.

[0120] The resulting real-time image fusion system is based entirely on FPGA real-time parallel pipelined computation, enabling the computation to achieve the highest real-time performance. In practical applications, it can effectively achieve the fusion of infrared and visible light images.

[0121] The following example illustrates its process of fusing infrared and visible light images.

[0122] Here, the input image size is set to m×n, then the pixel value p(x,y) at coordinates (x,y) is denoted as:

[0123] p(x,y), 1≤≤, 1≤≤;

[0124] In an FPGA, new pixels are input sequentially at unit time intervals Δt, forming a pixel stream timing sequence, where f is the video frame rate.

[0125]

[0126] Taking a video stream with a resolution of 1920×1080 and a frame rate of 60 frames per second as an example, Δt≈8 nanoseconds. Let the pixel stream timing be denoted as p(t), representing the pixel value at a certain unit of time. Then the following relationship holds:

[0127] t = (x-1)n + y;

[0128]

[0129] y = tmodn.

[0130] Based on this, the visible light pixel input is denoted as p1(x,y), and the infrared pixel input is denoted as p2(x,y). The calculation of the final fused image pixel pF(x,y) obtained by this real-time image fusion system is as follows:

[0131] Combination Figure 1 As shown, the optical pixel stream p1(x,y) and the infrared pixel stream p2(x,y) are respectively input to the first sliding window buffer 110 and the second sliding window buffer 120 in the system.

[0132] The first sliding window buffer 110 and the second sliding window buffer 120 respectively input pixels in parallel to the first F B Filter 130, First F M Filter 140, Second F M Filter 150, Second F B In filter 160.

[0133] First F M Filter 140, Second F M Filter 150, First F B Filter 130, Second F B Filter 160 processes the input pixels synchronously, obtaining the following data:

[0134]

[0135]

[0136]

[0137]

[0138] The first F here M Filter 140 and the second F M Filter 150 synchronously performs median filtering on visible light and infrared images based on the above formulas (1) and (2), thereby obtaining the first base layer of visible light and infrared images.

[0139] Here, when performing median filtering on visible light and infrared images, the value of a point in a digital image or digital sequence is replaced by the median value of all points in a neighborhood of that point, so that the surrounding pixel values ​​are close to the true values, thereby eliminating isolated noise points.

[0140] At the same time, the first F B Filter 130 and the second F B Filter 160 synchronously performs Gaussian filtering on visible light and infrared images based on the above formulas (3) and (4), thereby obtaining the second base layer of visible light and infrared images.

[0141] This method can eliminate Gaussian noise points in visible light and infrared images when applying Gaussian filtering.

[0142] Here, two filters are used simultaneously for processing both visible light and infrared images. This allows for the simultaneous application of linear filtering (Gaussian) and nonlinear filtering (median), resulting in two different base layers of the image.

[0143] Based on this, the data is processed by the first subtraction operator 177 and the third subtraction operator 179 to obtain the following data:

[0144]

[0145]

[0146] The data is processed by the second subtraction unit 178 and the fourth subtraction unit 180 to obtain the following data:

[0147]

[0148]

[0149] Here, synchronous operations between the first subtraction operator 177, the third subtraction operator 179, the second subtraction operator 178, and the fourth subtraction operator 180 are used to subtract the corresponding base layers from the visible light and infrared original images, thereby obtaining the corresponding detail layers for the visible light and infrared images. The corresponding detail layers here are as follows:

[0150] Linear detail layer of a visible light image;

[0151] Linear detail layer of infrared image;

[0152] Nonlinear detail layers in visible light images;

[0153] Non-linear detail layer of infrared images.

[0154] Based on this, the data is processed by the first adder 181 to obtain the following data:

[0155]

[0156] Accordingly, the detail layer pixel values ​​calculated after median filtering of the visible light image are added to the detail layer pixel values ​​calculated after median filtering of the infrared image, so as to facilitate further calculations.

[0157] Next, the data is processed by the first division unit 185 and the second division unit 186 to obtain the following data:

[0158]

[0159]

[0160] This method performs L2-Norm (L2 normalization) processing on visible light and infrared images to determine the respective weights of the visible light and infrared images, thereby ensuring the quality of subsequent image fusion.

[0161] Next, the data is processed by the first multiplication unit 191, the second multiplication unit 192, the third multiplication unit 193, and the fourth multiplication unit 194 to obtain the following data:

[0162]

[0163]

[0164]

[0165]

[0166] Finally, the parallel adder 199 processes the data to obtain the final fused image pixel p. F (x,y):

[0167]

[0168] Accordingly, the corresponding pixel values ​​of the visible light image and the corresponding pixel values ​​of the infrared image are organically added together, thereby achieving the fusion of visible light and infrared images.

[0169] When performing base layer fusion, an averaging strategy is used; simultaneously, during the fusion process, the weights of the base layer superposition in the visible light and infrared images are controlled based on the parameter w. When performing detail layer fusion, a weighted averaging strategy is used.

[0170] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A real-time image fusion method based on FPGA chip, characterized in that, include: First, the pixel streams of the first and second images to be fused are input into the corresponding filters in parallel; Next, median filtering and Gaussian filtering are performed simultaneously on the first image to be fused to obtain the first base layer and the second base layer of the first image; at the same time, median filtering and Gaussian filtering are performed simultaneously on the second image to obtain the first base layer and the second base layer of the second image. Next, calculations are performed on the original image of the first image and the first base layer and second base layer of the first image to form a first detail layer and a second detail layer for the first image; at the same time, calculations are performed on the original image of the second image and the first base layer and second base layer of the second image to form a first detail layer and a second detail layer for the second image. Finally, the second base layer and the first and second detail layers of the first image are fused with the second base layer and the first and second detail layers of the second image, thereby achieving the fusion between the first image and the second image.

2. The real-time image fusion method based on an FPGA chip according to claim 1, characterized in that, The first image is a visible light image, and the second image is an infrared image.

3. A real-time image fusion system based on an FPGA chip, characterized in that, The real-time image fusion system forms a first sliding window buffer, a second sliding window buffer, and a first F in the FPGA chip. B Filter, first F M Filter, second F B Filter, second F M Filters, multi-stage arithmetic units, and parallel adders; The first and second sliding window buffers respectively acquire the pixel streams of the first and second images to be fused, and input them in parallel into the first F... B Filter, first F M Filter, second F B Filter, second F M In the filter; The first F B Filter and first F M The filters are connected to the first sliding window buffer and simultaneously perform Gaussian filtering and median filtering on the input first image pixel stream to obtain the second base layer and the first base layer of the first image. The second F B Filter and second F M The filters are connected to the second sliding window buffer and Gaussian filtering and median filtering are performed synchronously on the input second image pixel stream to obtain the second base layer and the first base layer of the second image. The multi-level computing unit and the first F B Filter, first F M Filter, second F B Filter, and the second F M The filter data connection performs multi-level calculations on the first base layer and the second base layer of the synchronously formed first image, and the first base layer and the second base layer of the second image to form a linear detail layer and a non-linear detail layer for the first image, and a linear detail layer and a non-linear detail layer for the second image. The parallel adder is connected to the multi-level arithmetic unit and synchronously acquires the linear and non-linear detail layers of the first image and the linear and non-linear detail layers of the second image processed by the multi-level arithmetic unit. The parallel adder is connected to the first F... B Filter, second F B The filter data connection obtains the second base layer of the first image and the second base layer of the second image, and performs fusion processing on the first image and the second image.

4. The real-time image fusion system based on an FPGA chip according to claim 3, characterized in that, The first F B Filter or second F B The filter performs Gaussian convolution operations by shifting the input image pixel stream.

5. The real-time image fusion system based on an FPGA chip according to claim 3, characterized in that, The first or second sliding window buffer consists of 2 FIFOs and 9 buffers.

6. The real-time image fusion system based on an FPGA chip according to claim 3, characterized in that, The first F M Filter or second F M The filter is composed of several sorting circuit units.

7. The real-time image fusion system based on an FPGA chip according to claim 3, characterized in that, The multi-level arithmetic unit includes a first arithmetic unit, a second arithmetic unit, a third arithmetic unit, and a fourth arithmetic unit; The first arithmetic unit and the first F B Filter, first F M Data connections to the filter, the third arithmetic unit, and the parallel adder; The second arithmetic unit and the second F B Filter, second F M Data connections to the filter, the third arithmetic unit, and the parallel adder; The third arithmetic unit is data-connected to the first arithmetic unit and the second arithmetic unit, respectively. The fourth arithmetic unit and the first F B Filter, second F B Data connections for filters and parallel adders.

Images