Infrared dim small target detection method based on vision camera and FPGA hardware acceleration method

Abstract

The application discloses an infrared weak and small target detection method based on a vision camera and a FPGA hardware acceleration method, extracts a pixel unit voltage change rate of infrared data of the vision camera as an infrared scene image sequence, processes the infrared scene image sequence to obtain space-time features, further obtains detection target information, and thus accurately detects the infrared weak and small target; the FPGA hardware acceleration structure comprises an infrared pulse data processing module, a space feature extraction processing module, a time feature extraction processing module, a space feature matrix memory, a background suppression target enhancement module and a target information output module; further comprises a plurality of buffer register groups; the size of the buffer register group is a multiple of the row of the pixel matrix; the categories of the buffer register group comprise infrared data buffering, space feature row buffering, time feature row buffering and space-time feature row buffering; the FPGA hardware acceleration structure is used to realize real-time detection and processing of the infrared weak and small target.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to image target detection and acceleration optimization technology, especially to an infrared weak target detection method based on a Visor camera and an FPGA hardware acceleration method. Background Technology

[0002] In the infrared field, the detection and tracking of small moving targets plays an indispensable role in many fields such as aerospace and drone photography. However, objects at a greater distance are generally captured by infrared cameras as small targets, occupying very few pixels. Accurately detecting and tracking small infrared targets in real time is technically challenging.

[0003] The Visor camera is a new type of camera that mimics the retinal coding principle of primates, capturing light signals individually for each pixel, achieving a frame rate of up to 40,000 frames per second. It accurately and completely records the spatiotemporal changes in light intensity, a feat also achievable in the infrared field. However, the high frame rate inevitably leads to a massive data flow, drastically increasing processing time, which falls far short of real-time requirements and significantly diminishes the advantages of the Visor camera.

[0004] Existing traditional infrared weak target detection algorithms either use future information in the image sequence, resulting in unavoidable delays; or are limited by the two-dimensional features of a single frame, failing to effectively suppress background noise; and due to the limitations of traditional cameras, they are deficient in capturing infrared weak targets and targets with continuous motion. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, this invention provides an infrared weak target detection method based on a Visun camera, as well as an FPGA hardware acceleration design method and system. By proposing a new infrared weak target detection method, the spatiotemporal features of the infrared data from the Visun camera are extracted and further processed to obtain the target information, thereby accurately detecting infrared weak targets. At the same time, hardware acceleration design is performed on the FPGA to maintain real-time processing, which can meet the goal of real-time processing.

[0006] The technical solution provided by this invention:

[0007] A method for detecting weak infrared targets based on a Visor camera includes the following steps:

[0008] 1) Acquire infrared scene image sequences using an infrared vision camera;

[0009] Infrared cameras capture temperature / thermal radiation at each pixel. The voltage value of each pixel decreases over time, with a faster decrease at higher temperatures. The camera samples the voltage values ​​of all pixels at regular intervals to obtain the acquired infrared pulse data. After calculation using an image reconstruction algorithm, an infrared scene image sequence can be obtained. The pixel values ​​of the infrared scene image sequence are obtained by calculating the pixel unit voltage change rate using the image reconstruction algorithm. Therefore, this invention directly extracts the pixel unit voltage change rate matrix in the intermediate process and considers it as the infrared scene image sequence.

[0010] 2) Extract the spatial distribution features in a single frame of an infrared image sequence using a spatial domain convolutional converter to obtain a spatial feature matrix sequence;

[0011] 3) Obtain the temporal feature matrix by the difference between the maximum and minimum values ​​of the corresponding pixel positions in the spatial feature matrix sequence of the selected frames. The spatiotemporal feature matrix can be obtained by superimposing the temporal feature matrix with the spatial feature matrix.

[0012] Further background suppression processing at the pixel level can be applied to the spatiotemporal feature matrix to eliminate blurring caused by edge noise and camera motion, thereby highlighting the target features.

[0013] 4) The feature matrix is ​​compared according to the preset threshold. Pixels with values ​​higher than the threshold are identified as targets, and information such as the coordinates of the detected target points is obtained.

[0014] This method eliminates the need for future frames, effectively extracts the spatiotemporal feature information of the target, and superimposes a pixel-level background suppression module to enhance the contrast between the target area and the background area, reducing the computational load and achieving good detection accuracy.

[0015] This invention also provides an FPGA hardware acceleration structure for the above-mentioned infrared weak target detection method based on a Visor camera, including: an infrared pulse data processing module, a spatial feature extraction processing module, a temporal feature extraction processing module, a spatial feature matrix memory (module), a background suppression and target enhancement module, and a target information output module. The FPGA hardware acceleration method includes six stages, with a buffer register group (size a multiple of the number of rows in the pixel matrix) placed between each two stages. This includes an infrared data buffer, spatial feature row buffers (k1 / k2 / k3), a temporal feature row buffer k1, and a spatiotemporal feature row buffer k1, to facilitate row-level pipelined implementation. The infrared data buffer is n*n rows in size (n*n is the region size requirement of the convolutional unit in the spatial feature extraction processing module), while the remaining buffers are single rows in size. Specifically, it includes the following steps:

[0016] 1) In Stage 0, an infrared pulse data processing module is designed. The infrared pulse data generated by the Vision camera is used as input and input to the infrared pulse data processing module by pixel row to obtain the pixel unit voltage change rate matrix and store it in the infrared data buffer (the hardware implementation of row operation is implemented on the FPGA for row-level pipelined processing); this can save a lot of resources and latency.

[0017] 2) In Stage 1, a spatial feature extraction and processing module and a spatial feature matrix memory are designed. Infrared pulse data is stored in a buffer (infrared data buffer) of n*n rows (n can be 3, 4, or 5, but is generally 3. As the value of n increases, the computational load increases, but the result is more accurate). This satisfies the region size requirement of the convolutional module for spatial feature extraction and processing. The convolutional module consists of two steps: the first step calculates the mean of the n*n pixel region, and the second step performs convolution calculation on the n*n means to extract spatial features. After each row is completed, the result is saved in the spatial feature row buffer k1 and simultaneously written to the spatial feature matrix memory.

[0018] 3) In Stage 2, a temporal feature extraction and processing module is designed to read the historical spatial feature matrix (frame) k2 and spatial feature matrix k3 from the spatial feature matrix memory row by row into the corresponding spatial feature row buffer. The difference between the maximum and minimum values ​​of each pixel position is calculated with the spatial feature row buffer k1 to obtain the corresponding row of the temporal feature matrix k1, which is stored in the temporal feature row buffer k1. The spatial feature row buffer k1 of the current stage is then copied to the spatial feature row buffer k1 of the next stage.

[0019] 4) In Stage 3, the values ​​of the corresponding pixel positions in the spatial feature row buffer k1 and the temporal feature row buffer k1 are added together, and the result is stored in the spatiotemporal feature row buffer k1.

[0020] 5) In Stage 4, a background suppression target enhancement module is designed. After storing p rows of spatiotemporal feature rows in the buffer, background suppression target enhancement calculation is performed pixel by pixel. Similarly, after processing each row, the earliest input row is discarded, and the new input row is superimposed. If p rows are still satisfied, the calculation and processing continue.

[0021] 6) In Stage 5, each pixel is compared with a preset threshold, and the position of a pixel that is higher than the preset threshold is determined as the target, thereby obtaining information such as the coordinates of the detected target.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention proposes an infrared weak target detection method based on a Visun camera, featuring low latency, high accuracy, and high frame rate. Hardware acceleration design is implemented on an FPGA, achieving row-level pipelined processing for higher real-time processing speed. The hardware design forms an IP (Intellectual Property Core, i.e., a partially reusable "chip design module") that can be easily deployed on any other resource-rich FPGA board. Its technical advantages include:

[0024] (I) This invention proposes an infrared weak target detection method based on a Visor camera. It fully extracts the temporal and spatial features from infrared pulse data, eliminating the need for future frame information, but does not ignore prior frames, and performs pixel-level background suppression. It has high accuracy, good real-time performance, and high frame rate.

[0025] (ii) FPGA can easily implement parallel operation. Hardware acceleration design is carried out on FPGA to realize row-level pipelined processing of the proposed method, maximize processing speed, and significantly improve real-time performance without affecting accuracy, so as to obtain information of the detection target. Attached Figure Description

[0026] Figure 1 This is a flowchart of the infrared weak target detection method provided by the present invention.

[0027] Figure 2 This is a block diagram of the FPGA hardware-accelerated infrared weak target detection system provided by the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0029] This invention proposes a novel infrared weak target detection method based on an infrared vision camera. It maintains accuracy while reducing computational load, making it easier to implement FPGA hardware acceleration design. It has shown good application results in infrared weak target detection scenarios.

[0030] Infrared Vision cameras capture temperature / thermal radiation at each pixel. The voltage value of each pixel decreases over time, and the higher the temperature, the faster the decrease. The camera samples the voltage values ​​of all pixels at regular intervals to obtain the collected data. After calculation by an image reconstruction algorithm, an infrared scene image sequence can be obtained.

[0031] refer to Figure 1The design diagram of the infrared weak target detection method of the present invention is given. The pixel values ​​of the infrared scene image sequence are obtained by calculating the voltage change rate of pixel units through an image reconstruction algorithm. Therefore, the present invention directly extracts the change rate matrix in the intermediate process and regards it as the infrared scene image sequence, omitting many calculations in the image reconstruction algorithm, especially the division operation which consumes a lot of resources and has a large delay in hardware implementation. After obtaining the infrared "image" sequence, a spatial domain convolution is used to extract the spatial distribution features in a single frame of the infrared image. The convolution takes an n*n pixel as a unit, selects n*n units, regards the middle as the target area and the others as the background area, and takes the mean to perform n*n convolution to obtain the spatial feature matrix. Spatial feature extraction processing is performed on each selected current frame and prior frame to obtain a spatial feature matrix sequence. The time feature matrix is ​​obtained based on the difference between the maximum and minimum values ​​of the corresponding pixel positions in the spatial feature matrix sequence of the selected frames. The time feature matrix is ​​superimposed on the spatial feature matrix to obtain the spatiotemporal feature matrix. At this point, it is sufficient to obtain accurate feature information, but in order to eliminate edge noise, camera motion and other blur, pixel-level background suppression processing is required on the spatiotemporal feature matrix. Selecting a region of size p*p pixels, where the pixel with the maximum value remains unchanged, and subtracting the difference from the maximum value from the feature values ​​of other pixels, can more clearly highlight target features and suppress background information. Finally, the feature matrix is ​​compared according to a preset threshold (which can be 1.5). Pixels with values ​​higher than the threshold are identified as targets, thus obtaining information such as the coordinates of the detected target points. This method eliminates the need for future frames, effectively extracts the spatiotemporal feature information of the target, and overlays a pixel-level background suppression module to enhance the contrast between the target and background regions, reducing computational load and achieving good detection accuracy.

[0032] refer to Figure 2This invention realizes the proposed method in FPGA hardware acceleration and optimization design. The figure shows the FPGA hardware acceleration structure, which includes an infrared pulse data processing module, spatial feature extraction processing, temporal feature extraction processing, spatial feature matrix memory, and background suppression and target enhancement module, ultimately obtaining target information output. The infrared pulse data processing module takes infrared pulse data as input and outputs a single row of pixel unit voltage change rate. Its function is to receive the raw signal from the camera and calculate the pixel unit voltage change rate for use by the spatial feature extraction module. The spatial feature extraction module takes a single row of pixel unit voltage change rate as input and outputs a single row of spatial feature values ​​in a single matrix. Its function is to calculate spatial feature values, store them in the spatial feature matrix memory, and use them for the temporal feature extraction and background suppression / target enhancement modules. The spatial feature matrix memory takes a single row of spatial feature values ​​as input and outputs three rows of spatial feature values ​​(one row per frame) in three different matrices. Its function is to store the spatial feature matrix for use by the temporal feature extraction module. The temporal feature extraction module takes three rows of spatial feature values ​​as input and outputs a single row of temporal feature values. Its function is to calculate temporal feature values ​​for use by the background suppression / target enhancement module. The background suppression / target enhancement module takes a single row of spatial feature matrix values ​​and a single row of temporal feature matrix values ​​as input and outputs a target threshold matrix. After comparison with the threshold, the detected target information is output. A series of buffer register groups are set between each stage to facilitate row-level pipelined processing. In Stage 0, infrared pulse data generated by the Visor camera is used as input and input to the infrared pulse data processing module by pixel row. The resulting pixel unit voltage change rate matrix is ​​stored in the infrared data buffer. Since the proposed method eliminates a large number of division operations, this choice can save significant resources and latency. In Stage 1, after caching n*n rows (i.e., meeting the region size requirements of the convolutional module for spatial feature extraction processing), convolution calculation is performed. In subsequent convolution calculations, for each row processed, the earliest input row is discarded, and the latest input row is superimposed, still meeting the n*n row requirement. Spatial features are extracted, and after each row is completed, the extracted spatial features are saved in the spatial feature row buffer k1 and simultaneously written to the spatial feature matrix memory. In Stage 2, temporal feature extraction is performed. Historical spatial feature matrices k2 and k3 are read row-by-row from the spatial feature matrix memory and stored in their corresponding spatial feature row buffers. The difference between the maximum and minimum values ​​at each pixel position in the spatial feature row buffer k1 and the spatial feature row buffer k1 is used to obtain the corresponding row of the temporal feature matrix k1, which is then stored in the temporal feature row buffer k1. For use in subsequent stages and to avoid affecting the subsequent use of the preceding modules, the spatial feature row buffer k1 is copied to the spatial feature row buffer k1 of the next stage. In Stage 3, the values ​​at the corresponding pixel positions in the spatial feature row buffer k1 and the temporal feature row buffer k1 are added together, and the result is stored in the spatiotemporal feature row buffer k1.In Stage 4, after caching p rows, background suppression and target enhancement calculations are performed pixel by pixel. Similarly, after processing each row, the earliest input row is discarded, and the new input row is superimposed. If p rows are still satisfied, the calculation and processing continue. In Stage 5, the values ​​of each pixel position are compared with a preset threshold. Pixel positions that are higher than the threshold are determined to be targets, thereby obtaining information such as the coordinates of the detected targets.

[0033] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for detecting infrared dim small target based on a vision camera, characterized in that, By extracting the pixel unit voltage change rate from the infrared data of the Vision camera as an infrared scene image sequence and processing it to obtain spatiotemporal features, further target information is obtained, thereby accurately detecting weak infrared targets; the steps include the following: 1) Acquire infrared scene image sequences using an infrared vision camera; The infrared camera samples the voltage values ​​of all pixels in the image at regular intervals to collect infrared pulse data; the infrared scene image sequence is calculated from the pixel unit voltage change rate. 2) Extract the spatial distribution features in a single frame of an infrared image sequence using a spatial domain convolutional converter to obtain a spatial feature matrix sequence; 3) Obtain the temporal feature matrix by the difference between the maximum and minimum values ​​of the corresponding pixel positions in the spatial feature matrix sequence of the selected frames. The spatiotemporal feature matrix is ​​obtained by superimposing the temporal feature matrix on the spatial feature matrix. 4) By comparing the preset threshold with the spatiotemporal feature matrix, the pixel positions that are higher than the preset threshold are determined as targets, and the information of the detected target points, including the coordinates of the detected target points, is obtained; thus realizing infrared weak target detection based on the Vision camera.

2. The infrared weak target detection method based on a Visor camera as described in claim 1, characterized in that, Step 3) Further perform pixel-level background suppression processing on the spatiotemporal feature matrix to eliminate edge noise and camera motion blur, enhance the contrast between the target area and the background area, and highlight the features of the detected target.

3. The infrared weak target detection method based on a Visor camera as described in claim 1, characterized in that, The preset threshold value is 1.

5.

4. An FPGA hardware acceleration structure for implementing the infrared weak target detection method based on a Visor camera as described in claim 1, comprising: The system includes an infrared pulse data processing module, a spatial feature extraction and processing module, a temporal feature extraction and processing module, a spatial feature matrix storage module, a background suppression and target enhancement module, and a target information output module.

5. The FPGA hardware acceleration structure as described in claim 4, characterized in that, It also includes multiple buffer register groups; the size of the buffer register group is a multiple of the number of rows in the pixel matrix; the types of buffer register groups include infrared data buffer, spatial feature row buffer, temporal feature row buffer and spatiotemporal feature row buffer, which are used for row-level pipelined implementation.

6. The FPGA hardware acceleration structure as described in claim 5, characterized in that, The infrared data buffer is n*n rows in size, where n*n is the region size of the convolutional unit in the spatial feature extraction and processing module; the remaining buffers are all single rows in size.

7. An FPGA hardware acceleration method for infrared weak target detection based on a Visor camera, characterized in that, The FPGA hardware acceleration structure described in claim 5 is used to realize real-time detection and processing of infrared weak targets; Includes the following stages: 1) In Stage 0, an infrared pulse data processing module is designed to take the infrared pulse data generated by the Vision camera as input, process the data by pixel row, obtain the pixel unit voltage change rate matrix, and store it in the infrared data buffer. 2) In Stage 1, a spatial feature extraction and processing module and a spatial feature matrix memory are designed. After the number of rows of infrared pulse data stored in the infrared data buffer meets the region size requirement of the convolutioner used for spatial feature extraction and processing, convolution calculation is performed to extract spatial features. After each row is completed, the result is saved in the spatial feature row buffer and simultaneously written to the spatial feature matrix memory. 3) In Stage 2, a temporal feature extraction and processing module is designed to read the historical spatial feature matrix frames and spatial feature matrix frames row by row from the spatial feature matrix memory into the corresponding spatial feature row buffer. The difference between the maximum and minimum values ​​of each pixel position calculated from the spatial feature row buffer is used to obtain the corresponding row of the temporal feature matrix, which is then stored in the temporal feature row buffer. And copy the spatial feature row buffer of the current stage to the spatial feature row buffer of the next stage; 4) In Stage 3, the values ​​of the corresponding pixel positions in the spatial feature row buffer and the temporal feature row buffer are multiplied, and the result is stored in the spatiotemporal feature row buffer k1. 5) In Stage 4, a background suppression target enhancement module is designed. After storing p rows of spatiotemporal feature rows in the buffer, background suppression target enhancement calculation is performed pixel by pixel. Similarly, after processing each row, the earliest input row is discarded, and the new input row is superimposed. If p rows are still satisfied, the calculation and processing continue. 6) In Stage 5, a preset threshold is set, and the positions of pixels that are higher than the preset threshold are identified as targets, thereby obtaining the coordinate information of the detected targets.

Images