A method for implementing multi-channel infrared image parallel acquisition and enhancement by FPGA

Abstract

The present application belongs to the technical field of infrared image processing, and specifically relates to a FPGA implementation method for multi-channel infrared image parallel acquisition and enhancement. The method comprises the following steps: detector driving and configuration, image data reorganization, and image enhancement. Specifically, the image enhancement is realized in parallel for multi-channel infrared image data streams in the FPGA. A layered processing framework is adopted for multi-filter algorithm fusion. The image is decomposed into a background layer and a detail layer for targeted parallel processing. The bilateral and Gaussian filters of the parallel pipeline architecture are reduced in resource occupation and enhanced in real-time performance through a lookup table, threshold setting, frequency multiplication acceleration, and multi-module rotation. The enhanced multi-channel infrared image data is sequentially transmitted to an upper computer. The present application can realize synchronous acquisition, real-time enhancement, and large-bandwidth transmission of multi-channel infrared images, and can overcome problems such as insufficient single-source resolution, limited field of view, large resource consumption, multiple image noises, low contrast, and fuzzy edges, and can realize omnidirectional real-time target detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of infrared image processing, and particularly relates to a FPGA implementation method for multi-channel infrared image parallel acquisition and enhancement. BACKGROUND

[0002] Infrared imaging technology refers to forming an infrared image reflecting the temperature and contour characteristics of a target object through different intensity infrared light radiated by the target object and photoelectric conversion and other operations. The principle of non-contact and passive imaging makes it available day and night, has good concealment, strong detection and anti-interference ability, and is widely applied in the fields of military, automatic driving, security monitoring and the like.

[0003] At present, most existing infrared image systems are limited to single-channel infrared video source acquisition, and the limitations are as follows: 1. Low image resolution, the resolution of domestic infrared detectors is currently concentrated in 640x512, only a few high-end models use 1024x768, and insufficient resolution of single-channel acquisition will limit the application; 2. The field of view provided by the single-channel detector is limited, and real-time target detection in all directions cannot be achieved; 3. Most current systems are developed based on general platforms, and have the disadvantages of a large number of peripherals, resource redundancy and high cost.

[0004] Researchers are inspired by the compound eye visual system of insects, and use an array of multiple sub-eye lenses to simulate the compound eye of insects, thereby obtaining high-resolution and large-field-of-view imaging effect. This bionic curved compound eye visual system can effectively overcome the shortcomings of the above-mentioned infrared system, and has the advantages of high resolution, large field of view, compact structure and small volume.

[0005] Implementing the bionic curved compound eye visual system based on FPGA can obtain better real-time performance by using the parallel pipeline advantage of FPGA. Because FPGA has enough IO ports, it can be connected with a large number of sub-eye lenses, i.e. multi-channel infrared detectors, and is also easy to expand in the future. In addition, the programmability of FPGA can also allow the system to adjust to different application scenarios.

[0006] However, infrared images have the problems of low resolution, complex and diverse noise types, low image contrast and blurred edges, and the visual effect is poor. The existing image processing algorithms have the following three problems: 1. Most of the existing algorithms are for visible light, and cannot effectively solve the defects of infrared images; 2. The complexity of the existing image processing algorithms is high, most of them are implemented on software platforms or only part of the links are accelerated by hardware, and there are few infrared algorithms implemented in FPGA; 3. The commonly used algorithms of the existing single-channel system have large resource consumption and poor real-time performance, and are not suitable for the application scenario of multi-channel detectors.

[0007] Therefore, how to realize parallel collection and enhancement of multi-channel infrared images based on FPGA is an urgent problem to be solved at present. SUMMARY

[0008] To solve the above problems, the purpose of the present application is to provide a FPGA implementation method for parallel collection and enhancement of multi-channel infrared images with less resource consumption, strong real-time performance and good infrared image enhancement effect.

[0009] The FPGA implementation method for parallel collection and enhancement of multi-channel infrared images can achieve the following objectives: (1) maintaining the synchronization between multi-channel infrared video streams; (2) fully utilizing the parallel pipeline characteristics of FPGA to realize real-time enhancement of multi-channel images; (3) obtaining good multi-channel infrared image enhancement effect under resource constraints; (4) full parameterization implementation, making the system have good flexibility, portability and scalability; (5) realizing orderly transmission and scheduling of multi-channel infrared images to the host computer.

[0010] The FPGA implementation method for parallel collection and enhancement of multi-channel infrared images proposed by the present application includes four steps: detector driving and configuration, image data reorganization, image enhancement and communication control; the process is unfolded from image collection, enhancement and transmission, and the specific steps are as follows:

[0011] Step S1, detector driving:

[0012] The multi-channel infrared detectors share the driving interface connected with the FPGA, and the FPGA generates homologous clock and control signal to drive the multi-channel detectors to work synchronously.

[0013] Step S2, detector configuration:

[0014] The FPGA completes the initialization configuration of the multi-channel detectors in sequence by switching the bus chip selection signal and generates a flag signal for starting synchronous collection;

[0015] The specific steps are as follows:

[0016] Step S2-1, the multi-channel detectors share the bus configuration interface, the FPGA acts as the host and the multi-channel detectors act as multiple slaves, and the chip selection signal of a specific slave is selected under the control of the counter and the data selector.

[0017] Step S2-2, the FPGA reads the pre-stored configuration value in the ROM under the control of the state machine and transmits the data to the detector according to the bus timing, and completes the initialization.

[0018] Step S2-3, the sequence detection circuit in the FPGA judges the initialization result of the detector returned by the bus, and switches the selected enable after successful configuration and configures the next detector.

[0019] Step S2-4, the FPGA detects that all the detectors are successfully configured, and then internally generates a flag signal for starting synchronous acquisition, so as to realize synchronous acquisition and reading of multi-channel parallel data.

[0020] Step S3, image data reorganization:

[0021] The multi-channel infrared detector transmits image data to the FPGA multi-channel input subunit through a plurality of groups of data output interfaces, and performs extraction and reorganization of valid information and cross-clock domain timing adjustment in parallel; the specific steps are as follows:

[0022] Step S3-1, the multi-channel infrared detector returns pixel clock, row synchronization signal, frame synchronization signal and image data in parallel, detects the falling edge of the frame synchronization signal, and discards unstable data in the first several frames (which can be set).

[0023] Step S3-2, according to the timing diagram of the infrared detector, the redundant invalid information is removed, and the original image data is spliced and restored, and a corresponding data valid flag signal is generated.

[0024] Step S3-3, the processed data is entered into the multi-channel asynchronous FIFO subunit in the FPGA in parallel for cross-clock domain processing and timing adjustment, and then sent to the image enhancement module to realize synchronization with the system clock.

[0025] Step S4, image enhancement:

[0026] The multi-channel infrared image data stream in the FPGA realizes image enhancement in parallel, and is decomposed into a background layer and a detail layer through a bilateral filter and a Gaussian filter sub-module in a parallel pipeline architecture; the specific implementation steps are as follows:

[0027] Step S4-1, for a given window size and variance, the pixel similarity weight corresponding to the finite difference value is calculated and fixed-pointed in advance using MATLAB, and the weight result in the threshold range is obtained as a pixel similarity lookup table in the FPGA, so as to avoid complex exponential operation in the FPGA.

[0028] Step S4-2, N-1 FIFOs and N groups of shift registers are used to cache N-1 rows and N newly input data, so as to realize a movable filter window with a window size of N*N.

[0029] Step S4-3, the data groups in the movable filter window are entered into a plurality of parallel frequency multiplication processing subunits controlled by a frequency multiplication clock domain, and are processed in parallel under the action of control signals, so as to realize edge-preserving denoising processing by alternately multiplexing the frequency multiplication processing subunits and implementing a pipeline structure bilateral filtering process. The specific sub-steps are as follows:

[0030] Step 4-3-1, N*N-1 pixels in the filter window (except the center pixel) are divided into K groups, and are input into K parallel frequency multiplication processing subunits through asynchronous FIFO for subsequent processing. Each frequency multiplication processing subunit is responsible for (N*N-1) / K pixels (which needs to be an integer), and the frequency is at least (N*N-1) / K times the pixel clock. The enable control signal (denoted as control signal 1) when the FIFO is not empty is used as the starting work mark of the bilateral filtering, and the control signal 1 is delayed and punched in the frequency clock domain to obtain the required control signals (denoted as control signals 2-7) of each step (6 steps) in the pipeline;

[0031] Step 4-3-2, under the control of control signal 1 and its corresponding counter, (N*N-1) / K pixels enter their corresponding processing subunits, and are sequentially filled into the pipeline in (N*N-1) / K frequency multiplication periods to calculate the difference with the center pixel and perform threshold judgment.

[0032] Step 4-3-3, under the action of control signal 2 and its corresponding counter, the value domain and spatial domain weight are sequentially obtained from the lookup table and are input into multiplier 1.

[0033] Step 4-3-4, under the action of control signal 3, the data in the movable filter window is buffered.

[0034] Step 4-3-5, the bilateral filter weight output by multiplier 1 in step 4-3-3 is obtained, and under the action of control signal 4 and its corresponding counter, the corresponding input window pixel data is input into multiplier 2.

[0035] Step 4-3-6, under the action of control signal 5 and its corresponding counter, the bilateral filter weights obtained by the multiple parallel processing subunits in step 4-3-5 are accumulated.

[0036] Step 4-3-7, the weighted pixel value output by multiplier 2 in step 4-3-5 is obtained, and the accumulator is initialized under the action of control signal 6.

[0037] Step 4-3-8, under the action of control signal 7 and its corresponding counter, the pixel weighted values obtained by the multiple parallel processing subunits in step 4-3-7 are accumulated.

[0038] Step 4-3-9, after each (N*N-1) / K period of accumulation in step 4-3-8 is completed, the bilateral filter weight accumulation value obtained in step 4-3-6 is input into the divider to realize normalization processing.

[0039] Step 4-3-10, the normalized result output by the divider in step 4-3-9 is transmitted to the pixel clock domain through the asynchronous FIFO and a data valid mark signal is generated.

[0040] Step S4-4, the processed data enters a movable filter window and is smoothed by a Gaussian filter of a parallel pipeline architecture to remove edge details and residual noise, and a background layer of the image is obtained;

[0041] Step S4-5, the data processed in step S4-3 is buffered, and the background layer of the image is obtained by subtracting the data processed in step S4-4 from the data processed in step S4-3.

[0042] Step S4-6, the background layer and the detail layer of the image data are processed in parallel, that is, the contrast of the background layer is improved, and the detail layer is enhanced and denoised, and the negative gain coefficient generated by the bilateral filter in step S4 is used to weaken the influence of the noise of the detail layer and is superimposed.

[0043] Step S5, communication control

[0044] The multi-channel arbitration module schedules the multi-channel infrared image data after enhancement transmission to be sequentially and orderly transmitted to the upper computer.

[0045] Overall, the infrared image data is processed synchronously by the parallel multiple sub-module circuits, and good real-time and synchronization are achieved.

[0046] In the image enhancement part, a layered processing framework is adopted, the image is decomposed into a background layer and a detail layer by a bilateral filter and a Gaussian filter of a full-parallel pipeline architecture, and the background layer and the detail layer are processed in parallel, so that the real-time of the system is ensured, and the resolution and the visual effect are effectively improved.

[0047] The bilateral filter and the Gaussian filter sub-modules of the parallel pipeline architecture of the image enhancement part are reasonably scheduled by means of lookup table, threshold setting, frequency multiplication acceleration, multi-module rotation multiplexing and the like, resource occupation is effectively reduced, resource consumption is balanced, and the real-time of the enhancement algorithm is balanced. When the detail layer is processed, the negative gain coefficient generated by the bilateral filter is used to weaken the noise, and the detail information of the infrared image is effectively enhanced.

[0048] The multi-channel processing units and the sub-units thereof fully exert the advantages of FPGA in parallel computing and pipeline. All the modules are designed with full parameterization, support modification of the number of detectors, the number of invalid frames, the window size of the filter, the number of processing units, the number of lookup tables, the number of multipliers and the like, have good portability and flexibility, and can balance the resource consumption, the image enhancement effect and the real-time without modifying the design.

[0049] The application also relates to a FPGA implementation system for multi-channel infrared image parallel acquisition and enhancement, which comprises four modules: a detector driving and configuration module, an image data reorganization module, an image enhancement module and a communication control module, which respectively perform the operations of steps S1 and S2, steps S3, steps 4 and steps 5.

[0050] Features and advantages of the present application

[0051] (1) The method of the present application supports the loading of multiple infrared detectors, real-time reading and enhancement of multi-detector data, and meets the functional requirements of synchronous acquisition, real-time enhancement, and large bandwidth transmission. The method applied to the bionic curved surface compound eye vision system can solve the problems of insufficient single-source acquisition resolution, limited field of view, large system resource consumption, complex and diverse types of infrared image noise, low image contrast, and blurred edges, and realizes omnidirectional real-time target detection.

[0052] (2) In the present application, the multiple infrared detectors save the occupation of interface resources by sharing the control interface connected to the FPGA, which is conducive to further expanding the number of multi-channel detectors in the future. At the same time, the driving and configuration circuit involved realizes the synchronous acquisition and reading of multi-channel parallel data, effectively ensuring good data synchronization among the multiple infrared detectors.

[0053] (3) The method of the present application has high parallelism and pipeline characteristics. The multi-channel infrared image data in the FPGA is processed in parallel by multiple sub-units, the background layer and the detail layer in the single enhancement module are processed in parallel, and the filter of the enhancement module is implemented in a parallel pipeline architecture, which fully utilizes the parallel pipeline characteristics of the FPGA, enhances the infrared image effect, and ensures the real-time performance of the multi-channel infrared image acquisition and transmission processing system.

[0054] (4) In the implementation process of the present application, the use of internal hardware resources of the FPGA is fully considered. Through lookup table, threshold setting, frequency multiplication acceleration, and multi-module rotation multiplexing, resource scheduling is reasonably performed and resource consumption is reduced. At the same time, full parameterization enables the present application to flexibly adjust the filter window size and the number of processing units according to the actual number of multi-channel detectors, achieving a good balance between resource consumption, image enhancement effect, and real-time performance without modifying the design and only changing the parameters.

[0055] In summary, the present application fully utilizes the advantages of FPGA parallel computing, pipeline processing, rich interface resources, and good flexibility, breaks through the limitations of single video source acquisition, solves the problems of poor algorithm specificity, high resource consumption, and poor real-time performance, realizes real-time reading and enhancement of multi-detector data, and meets the functional requirements of synchronous acquisition, real-time enhancement, and large bandwidth transmission. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flow chart of the FPGA implementation method of multi-channel infrared image parallel acquisition and enhancement in the embodiment of the present application.

[0057] Figure 2 is a method framework diagram of multi-channel infrared image parallel acquisition and enhancement transmission in the embodiment of the present application.

[0058] Figure 3 is a flow chart of FPGA configuration of multi-channel infrared detector in the embodiment of the present application.

[0059] Figure 4 is a method framework chart of FPGA configuration of multi-channel infrared detector in the embodiment of the present application.

[0060] Figure 5 is a flow chart of multi-channel infrared detector data information extraction and timing adjustment in the embodiment of the present application.

[0061] Figure 6 is a schematic diagram of infrared image enhancement algorithm in the embodiment of the present application.

[0062] Figure 7 is a flow chart of infrared image enhancement algorithm in the embodiment of the present application.

[0063] Figure 8 is a realization schematic diagram of bilateral filter in the embodiment of the present application.

[0064] Figure 9 is a realization flow chart of bilateral filter in the embodiment of the present application.

[0065] Figure 10 is a realization schematic diagram of bilateral filter in the embodiment of the present application.

[0066] Figure 11 is an effect diagram of infrared image layering processing in the embodiment of the present application.

[0067] Figure 12 is a detail diagram of infrared image layering processing effect in the embodiment of the present application.

[0068] Figure 13 is a realization principle schematic diagram of multi-channel communication module in the embodiment of the present application.

[0069] Figure 14 is a simulation effect diagram of multi-channel communication module realization in the embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the FPGA implementation method of multi-channel infrared image parallel acquisition and enhancement of the present application is further described below in combination with embodiments and drawings.

[0071] In this embodiment, Wuhan Gaode Zishen GST117W infrared detector 120 module is selected as the front-end image source, the highest resolution is 120*90, the output frame frequency is controllable in 1-30Hz, and the multi-channel infrared detector generates infrared image data transmission to the FPGA after driving and bus configuration; the Xilinx Spartan 6 series XC6SLX150-2FGG484 type FPGA is selected as the core master, which is responsible for realizing four modules of infrared detector driving and configuration, image data reorganization, image enhancement and communication control (see Figure 1 );The bilateral filter window size is set to 7*7, and 48 pixels except the center pixel are processed by 4 processing subunits and lookup table, each processing subunit is responsible for 12 pixels, and the number of multiplier pipeline stages is 4, which can be flexibly adjusted according to the actual situation.

[0072] The interface between the infrared detector and the FPGA is divided into control and data output interfaces. The control interface transmits the working clock, control and bus configuration signals, and the multi-channel detector shares the interface. The data output interface mainly includes pixel clock, line synchronization signal, frame synchronization signal and infrared image data. The infrared image data processed by the FPGA will be transmitted to the host computer through the interface. This embodiment includes the following steps (see Figure 2 ):

[0073] Step S1, the multi-channel infrared detector shares the driving interface connected with the FPGA, and the FPGA generates the same clock and control signal to drive the multi-channel detector to work synchronously. The multi-detector shared control interface can save the number of IO interfaces and logic resources on the one hand; on the other hand, the same clock and control signal can guarantee the synchronization between the multi-channel detectors.

[0074] Step S2, the FPGA completes the initialization configuration of the multi-channel detector in turn by switching the bus chip selection signal and generating a flag signal to start synchronous acquisition; as shown in Figure 3 and Figure 4 , the specific steps are as follows:

[0075] Step S2-1, the multi-channel detector shares the bus configuration interface, the FPGA is the host, and the multi-channel detector is the slave, under the control of the counter and the data selector, the selected slave is configured in the subsequent steps.

[0076] Step S2-2, the FPGA reads the pre-stored configuration value in the ROM under the control of the state machine and transmits the data to the detector according to the bus timing, and completes the initialization.

[0077] Step S2-3, the sequence detection circuit in the FPGA judges the detector initialization result returned on the bus. If an abnormal configuration occurs, the FPGA resets the infrared detector, restarts the configuration process, and lights the LED on the device to remind the user. If the configuration is successful, the FPGA switches the enablement and configures the next detector under the control of the counter and the data selector circuit.

[0078] Step S2-4, after the FPGA detects that all the detectors are successfully configured, the FPGA internally generates a flag signal indicating the start of synchronous acquisition, realizes synchronous acquisition and reading of multiple parallel data, and ensures that multiple independent parallel interfaces synchronously transmit the pixel clock, the row synchronization signal, the frame synchronization signal, and the image data to the FPGA.

[0079] Step S3, the multiple infrared detectors transmit the image data to the multiple input subunits of the FPGA through multiple data output interfaces, and perform parallel extraction and reorganization of valid information and cross-clock-domain timing adjustment; for details, see Figure 5 , which includes the following substeps:

[0080] Step S3-1, the multiple infrared detectors return the pixel clock, the row synchronization signal, the frame synchronization signal, and the image data in parallel, detect the falling edge of the frame synchronization signal, and discard unstable data in the previous several frames (which can be set).

[0081] Step S3-2, according to the timing diagram of the infrared detector, the redundant invalid information is removed under the control of the pixel clock and the row field synchronization signal, and the information is spliced and restored into original image data and corresponding data valid flag signals. In the embodiment, the infrared detector transmits one pixel value every two clocks, with the high 7 bits in front and the low 7 bits in back, and the FPGA splices and restores the 14-bit image data.

[0082] Step S3-3, the processed data enters the multiple asynchronous FIFO subunits in the FPGA in parallel to perform cross-clock-domain processing and timing adjustment, realizes synchronization with the system clock, and outputs the buffered image to the multiple image enhancement modules instantiated in the subsequent stage in parallel to ensure the matching of the data rates at both ends.

[0083] Step S4, the multiple infrared image data streams in the FPGA realize image enhancement in parallel (see Figure 6 ). To obtain better infrared image enhancement effects, the application adopts a hierarchical processing framework, and the infrared image is decomposed into a background layer and a detail layer by the bilateral filter and the Gaussian filter submodules in the parallel pipeline architecture for targeted enhancement.

[0084] The core principle of the bilateral filter is to replace the center pixel value with the weighted average value of the pixels in the filter window (see Figure 8). Compared with Gaussian filter, bilateral filter considers the influence of spatial position and pixel similarity to the center pixel, which can avoid the edge pixels being blurred by the non-edge pixels in the neighborhood. Its mathematical model is:

[0085]

[0086] In this model, S represents the filter window, I p is the center pixel in the filter window, I q is the neighborhood pixel in the filter window. σ r and σ s are Gaussian variance, which determines the influence degree of the neighborhood pixel to the center pixel in the filtering process.

[0087] is the spatial proximity weight, and ||p-q|| 2 represents the squared Euclidean distance between the neighborhood pixel and the center pixel. From the formula, for the window size and σ s , which are determined, can be calculated in advance in MATLAB.

[0088] is the pixel similarity weight, the more similar the neighborhood pixel and the center pixel, the greater the influence. By calculating the weight value of |I p -I q in advance in MATLAB and setting a certain threshold, the method subsequently obtains the weight through the look-up table.

[0089] is the normalization factor.

[0090] The original video data stream contains background, edge details and noise information. After being filtered by the bilateral filter with edge-preserving denoising effect, except the background information, a large amount of detail information is retained, and only a small amount of noise is left. After further smoothing by the Gaussian filter, the background layer can be obtained. The bilateral filtering result minus the Gaussian filtering result can obtain a large amount of detail information and a small amount of noise information, which is the detail layer.

[0091] The background layer and the detail layer can be processed by different circuit modules through FPGA parallelism, guaranteeing real-time performance and obtaining better infrared image enhancement effect through multi-algorithm fusion. For the background layer information, the visual effect can be enhanced through contrast enhancement. For the detail layer information, direct enhancement will amplify noise due to residual noise information. Therefore, a negative gain coefficient is introduced as a superposition coefficient, which is obtained from the bilateral filtering weight cumulative value of the previous link. For noise information, the difference value with surrounding pixels will be large, so the bilateral filtering weight cumulative value obtained in the bilateral filtering link will also be large. For background or edge information, the pixels in the filtering window will have a certain similarity with them, so the bilateral filtering weight will be relatively small. Therefore, the bilateral filtering weight value is used as the negative gain coefficient of the detail layer enhancement, which can effectively enhance the image edge details and weaken the influence of residual noise.

[0092] As shown in Figure 7 , step S4 includes the following sub-steps:

[0093] Step S4-1, for a given window size and variance, the pixel similarity weight corresponding to the finite difference value is calculated and fixed-pointed in advance by MATLAB, and the weight result in the threshold range is obtained as the pixel similarity lookup table in FPGA, avoiding complex exponential operation in FPGA.

[0094] Step S4-2, assuming N=7, 6 FIFOs and 7 groups of shift registers are used to buffer 6 rows and the latest input of 6 data, to realize a 7*7 movable filtering window.

[0095] Step S4-3, the data groups in the movable filtering window enter multiple parallel frequency multiplication processing subunits controlled by the frequency multiplication clock domain for parallel processing, and under the action of various control signals, the frequency multiplication processing subunits are alternately multiplexed and bilateral filtering processing with pipeline structure is realized, to realize edge-preserving denoising processing. The specific sub-steps are (see Figure 9 ):

[0096] Step 4-3-1, in this embodiment, 48 pixels in the filtering window except the center pixel are divided into 4 groups, which enter 4 parallel frequency multiplication processing subunits for subsequent processing through asynchronous FIFO, and each frequency multiplication processing subunit is responsible for 12 pixels, and the frequency of the processing subunit is at least 12 times the pixel clock; when the FIFO is not empty, the enable control signal 1 is used as the starting work mark of the bilateral filtering, and the control signal 2-7 required by each step of the pipeline is obtained by delaying and punching the control signal 1 under the frequency multiplication clock domain;

[0097] Step 4-3-2, 12 pixels enter their corresponding processing subunits under the control of control signal 1 and its corresponding counter, fill the pipeline in turn in 12 frequency multiplication periods, calculate the difference value with the center pixel and make threshold judgment;

[0098] Step 4-3-3, obtain the value range and spatial domain weight in turn under the action of control signal 2 and its corresponding counter, and send them into multiplier 1;

[0099] Step 4-3-4, under the action of control signal 3, cache the data in the movable filter window;

[0100] Step 4-3-5, in this embodiment, after 4 multiplier pipeline periods, obtain the bilateral filter weight output by multiplier 1 in step 4-3-3, and send it into multiplier 2 together with the corresponding input window pixel data under the action of control signal 4 and its corresponding counter;

[0101] Step 4-3-6, under the action of control signal 5 and its corresponding counter, accumulate the bilateral filter weight obtained by the multiple parallel processing subunits in step 4-3-5, and 4 processing subunit parallel outputs are needed to be accumulated in each period;

[0102] Step 4-3-7, in this embodiment, after 4 multiplier pipeline periods, obtain the weighted pixel value output by multiplier 2 in step 4-3-5, and initialize the accumulator under the action of control signal 6;

[0103] Step 4-3-8, under the action of control signal 7 and its corresponding counter, accumulate the pixel weighted value obtained by the 4 parallel processing subunits in step 4-3-7, and 4 pixel weighted values are needed to be accumulated in each period;

[0104] Step 4-3-9, after the accumulation in step 4-3-8 is completed in 12 periods, send the accumulated value of the bilateral filter weight obtained in step 4-3-6 into the divider to realize normalization processing;

[0105] Step 4-3-10, obtain the normalization result output by the divider in step 4-3-9, transfer it into the pixel clock domain through the asynchronous FIFO and generate a data valid flag signal.

[0106] The above parallel pipeline structure of the bilateral filter is realized by referring to Figure 10 , the effect diagram is shown in Figure 11 and Figure 12 . Figure 11 is the overall effect diagram, Figure 12 is Figure 11 the enlarged detail effect diagram. Among them, Figure 11 a and Figure 12 a are the original images after superimposing random noise.Figure 11 b and Figure 12 b the noise in the image after bilateral filtering is largely removed and the image edges are well preserved; Figure 11 c and Figure 12 c is Figure 11 b and Figure 12 b the background layer obtained after further superimposing Gaussian filtering on the basis of b; Figure 11 d and Figure 12 d is the detail layer obtained by subtracting the background layer from the above-mentioned bilateral filtering result, and the image detail information is well extracted and preserved. The above results show that the bilateral filter implemented under the condition of resource limitation in the present application can obtain a good edge-preserving denoising effect and effectively separate the image background layer and the detail layer.

[0107] Step S4-4, the processed data enters the movable filtering window and is smoothed by the Gaussian filter of the parallel pipeline architecture, removing the edge details and residual noise, and obtaining the image background layer;

[0108] Step S4-5, the data processed in step S4-3 is cached, and the image detail layer is obtained by subtracting the data processed in step S4-4;

[0109] Step S4-6, the background layer and the detail layer image data are processed in parallel, and the negative gain coefficient generated by the bilateral filter in step S4 is used to weaken the noise influence of the detail layer and superimposed.

[0110] Step S5, in the multi-path application scenario, due to the limited data bus bit width of the interface with the host computer, in addition to the need to implement data transmission according to a specific timing, a multi-path arbitration module is also needed to schedule the enhanced multi-path infrared image data transmitted to the host computer in sequence. For example Figure 13 As shown in the implementation principle diagram, taking 14-bit infrared image data in the present embodiment as an example, if the bus width is 32 bits, two channels are shared by two detectors. The enhanced image data value first enters the asynchronous FIFO for caching, and when the cached data is full of one row of data, a handshake request is sent. The frequency-doubled read clock domain arbitrator arbitrates the handshake requests of multiple channels, and adopts a cyclic priority manner to send response signals to each channel in turn. Taking the communication transmission of 10 detectors as an example, the effect diagram is as follows Figure 14The 10-channel infrared image data is input into 5 channels, and a req request (req[4:0]) is sent when a row of data is buffered. Channel 0 has the highest priority and obtains a response, grant[0] is pulled high, and the detector data corresponding to dout0 and dout1 of channel 0 is output to the host computer under the control of the counter cnt_usb_0. When a row of data is sent, the arbiter controls the switching to channel 1 to obtain a response, grant[0] is pulled low, grant[1] is pulled high, and the above sending process is repeated. In this way, the multi-channel data can be sequentially and orderly transmitted to the host computer.

[0111] The multi-channel infrared image parallel acquisition and enhanced FPGA implementation method provided by the above embodiment can realize synchronous acquisition, real-time enhancement and large-bandwidth transmission of multi-channel infrared detectors, solve the problems of insufficient resolution, limited field of view, large system resource consumption, complex and diverse infrared image noise types, low image contrast and fuzzy edges, and realize omnidirectional real-time target detection.

Claims

1. A FPGA implementation method for multi-channel infrared image parallel acquisition and enhancement, characterized in that, The method comprises the following steps: Step S1, detector driving; The multi-channel infrared detector shares a driving interface connected with the FPGA, and the FPGA generates homologous clock and control signals to drive the multi-channel detector to work synchronously; Step S2, detector configuration; the FPGA sequentially completes initialization configuration of the multi-channel detector and generates a flag signal of synchronous acquisition start through switching of a bus chip selection signal; Step S3, image data reorganization; The multi-channel infrared detector transmits image data to the FPGA multi-channel input subunit through a plurality of groups of data output interfaces, and parallelly extracts and reorganizes effective information and adjusts cross-clock domain time sequence; Step S4, image enhancement; The FPGA parallelly realizes image enhancement of the multi-channel infrared image data stream, adopts a hierarchical processing framework of multi-filter algorithm fusion, decomposes the image into a background layer and a detail layer for targeted parallel processing, and reduces resource occupation and enhances real-time performance through a lookup table, threshold setting, frequency multiplication acceleration, multi-module rotation and other means of a bilateral and Gaussian filter of a parallel pipeline architecture; Step S5, communication control; The multi-channel arbitration module sequentially and orderly transmits the multi-channel infrared image data after enhancement transmission to an upper computer; Step S4 specifically comprises: Step S4-1, for a given window size and variance, pixel similarity weights corresponding to limited differences are pre-calculated and fixed-pointed by using MATLAB, a weight result in a threshold range is obtained as a pixel similarity lookup table in the FPGA, and complex exponential operation in the FPGA is avoided; Step S4-2, N-1 rows and N data newly input are buffered by using N-1 FIFOs and N groups of shift registers, and a movable filter window with a window size of N*N is realized; Step S4-3, data groups in the movable filter window enter a plurality of parallel frequency multiplication processing subunits controlled by a frequency multiplication clock domain for parallel processing, the processing subunits are multiplexed and bilateral filtering processing of a pipeline structure is realized under the action of control signals, and edge-preserving denoising processing is realized; Step S4-4, the processed data enter a movable filter window and are smoothed by a Gaussian filter of a parallel pipeline architecture, residual noise and edge details are removed, and an image background layer is obtained; Step S4-5, the data processed in step S4-3 are buffered, and the data processed in step S4-4 are subtracted to obtain an image detail layer; Step S4-6, background layer and detail layer image data are parallelly processed, that is, the background layer image realizes contrast enhancement, the detail layer image realizes detail enhancement and denoising, and a negative gain coefficient generated by the bilateral filter in step S4 is used to weaken the influence of noise in the detail layer and is superimposed; Step S4-3 specifically comprises: Step 4-3-1, N*N-1 pixels in the filter window are divided into K groups, enter K parallel frequency multiplication processing subunits for subsequent processing through an asynchronous FIFO, each processing subunit is responsible for (N*N-1) / K pixels, and the frequency is at least (N*N-1) / K times the pixel clock; when the FIFO is not empty, control signal 1 is enabled as a starting work flag of bilateral filtering, and control signals 2-7 required by each step of the pipeline are obtained by delaying and punching the control signal 1 under the frequency multiplication clock domain. Step 4-3-2, under the control of control signal 1 and its corresponding counter, (N*N-1) / K pixels enter their corresponding processing subunits, fill the pipeline in (N*N-1) / K frequency cycles, calculate the difference with the center pixel and make threshold judgment; Step 4-3-3, under the action of control signal 2 and its corresponding counter, obtain the value range and spatial domain weight from the look-up table in turn and send them into multiplier 1; Step 4-3-4, under the action of control signal 3, cache the data in the movable filter window; Step 4-3-5, obtain the bilateral filter weight output by multiplier 1 in step 4-3-3, under the action of control signal 4 and its corresponding counter, send the corresponding input window pixel data into multiplier 2; Step 4-3-6, under the action of control signal 5 and its corresponding counter, accumulate the bilateral filter weights obtained by multiple parallel processing subunits in step 4-3-5; Step 4-3-7, obtain the weighted pixel value output by multiplier 2 in step 4-3-5, under the action of control signal 6, initialize the accumulator; Step 4-3-8, under the action of control signal 7 and its corresponding counter, accumulate the pixel weighted values obtained by multiple parallel processing subunits in step 4-3-7; Step 4-3-9, after the accumulation of step 4-3-8 is completed every (N*N-1) / K cycles, send the bilateral filter weight accumulation value obtained in step 4-3-6 into the divider to realize normalization processing; Step 4-3-10, obtain the normalization result output by the divider in step 4-3-9, transfer it to the pixel clock domain through asynchronous FIFO and generate a data valid flag signal.

2. The method for multi-channel infrared image parallel acquisition and enhanced FPGA implementation according to claim 1, characterized in that, Step S2 specifically comprises: Step S2-1, the multi-channel detector shares the bus configuration interface, the FPGA acts as the host and the multi-channel detector acts as multiple slaves, under the control of the counter and the data selector, the chip select signal of a specific slave is selected; Step S2-2, under the control of the state machine in the FPGA, the pre-stored configuration value in the ROM is read and the data is transmitted to the detector according to the bus timing, and the initialization is completed; Step S2-3, the sequence detection circuit in the FPGA judges the initialization result returned by the bus, switches the selected enable after successful configuration, and configures the next detector; Step S2-4, after the FPGA detects that all detectors are successfully configured, a synchronization acquisition start flag signal is generated internally, realizing the synchronization acquisition and reading of multiple parallel data.

3. The FPGA implementation method for multi-channel infrared image parallel acquisition and enhancement according to claim 2, characterized in that, Step S3 specifically comprises: Step S3-1, the multi-channel infrared detector returns the pixel clock, the line synchronization signal, the frame synchronization signal and the image data in parallel, detects the falling edge of the frame synchronization signal, and discards the unstable data of the previous several frames; Step S3-2, according to the timing diagram of the infrared detector, remove the redundant invalid information, splice and restore it into the original image data, and generate the corresponding data valid flag signal; Step S3-3, the processed data is entered into the multi-channel asynchronous FIFO subunit in the FPGA in parallel for cross-clock domain processing and timing adjustment, and then sent into the image enhancement module, realizing the synchronization with the system clock.

Images