FPGA-based infrared moving target real-time identification method and system
By implementing wavelet pyramid model and central peripheral differential operation of adaptive feature images on FPGA, the problem of slow recognition speed of infrared moving targets at edge devices and unclear edge profiles is solved, and high-precision real-time recognition of infrared moving targets is achieved.
Patent Information
- Application Number
- CN202411547739.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-03-21
AI Technical Summary
The prior art is difficult to realize real-time identification of infrared moving targets at edge devices, especially due to the slow algorithm operation speed and unclear target edge profiles.
The infrared moving target real-time recognition method based on FPGA is adopted to obtain infrared video streams in real time through FPGA, and the wavelet pyramid model of adaptive feature images is realized using HLS high-level comprehensive integration, and the central peripheral differential operation is performed to obtain the target significant feature map.
High-precision real-time recognition of infrared moving targets is achieved, significant graph quality is improved, processing delay is reduced, and detailed outline information of the target is effectively retained.
Smart Images
Figure CN119672289A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image target recognition, and in particular relates to a real-time recognition method and system for infrared moving targets based on FPGA. Background Art
[0002] Modern intelligent weapons and equipment, such as unmanned reconnaissance aircraft, cruise missiles, and satellite detection devices, are equipped with built-in image intelligent processing modules to identify hostile targets. In daily civilian use, more and more imaging devices are embedded with image recognition algorithms to achieve monitoring and alarm purposes. As the focus and difficulty in the field of image processing, target recognition has great uses in daily production and life, and plays an important role in various occasions.
[0003] Infrared images have the advantages of being less affected by weather and being able to work all day, and are widely used in the modern military field. At present, infrared moving target recognition methods can be mainly divided into two categories: those based on deep learning and those based on visual saliency algorithms. Due to the changes in the contour of the target during the movement, the target recognition method based on deep learning shows the problem of insufficient intelligence for such changes, and requires a large number of infrared images of real scenes for training. Due to the large amount of calculation, the running speed on the edge device side is slow, and it is difficult to implement in a real combat environment. The method based on traditional visual saliency detection does not rely on training data, but has the characteristics of incomplete detection of targets and unclear edge contours. At the same time, the algorithms on the edge device side are mostly deployed on the CPU, which makes it difficult to achieve real-time recognition of moving targets in the video. The method based on GPU (Graphics Processing Unit) acceleration can improve the recognition speed, but the high power consumption makes it difficult to apply on the edge device side. Summary of the invention
[0004] In view of this, the present invention provides a real-time identification method and system for infrared moving targets based on FPGA, which can solve the problem in the prior art that the edge device-side saliency recognition algorithm for infrared moving targets has unclear edge contours and slow running speed, making it difficult to identify in real time.
[0005] The technical solution for implementing the present invention is as follows:
[0006] A real-time recognition method for infrared moving targets based on FPGA, comprising:
[0007] FPGA acquires infrared video stream in real time and obtains infrared grayscale image features;
[0008] HLS high-level synthesis realizes the wavelet pyramid model of adaptive feature images, and obtains significant feature maps through central and peripheral difference operations;
[0009] The target area and background are obtained according to the salient feature map, the geometric features of the target area are extracted, and the target is determined.
[0010] Furthermore, the HLS high-level synthesis realizes the wavelet pyramid model of the adaptive feature image, and the central peripheral difference operation obtains the target significant feature map including:
[0011] Perform histogram equalization on the input infrared grayscale features to determine the target index;
[0012] The target index is the ratio of the target pixels to the pixels of the entire image, and the formula is:
[0013]
[0014] Among them, the critical threshold M is determined according to the target and background type, and 5000 is selected for the infrared ship moving target; when the number of target pixels accounts for 1 / 4 or more of the total image area, the target can be considered to be a large target, and a multi-layer scale pyramid is used; when the number of target pixels accounts for a small proportion of the total image area, the target in the image can be identified as a small target. Due to excessive downsampling, the targets in the high layers will disappear, causing some differential operations to become redundant. Therefore, appropriately reducing the number of pyramid layers can better adapt to small target detection. At the same time, for small targets, reducing the number of pyramid layers can not only improve the quality of the saliency map, but also reduce processing delay.
[0015] Furthermore, in the process of calculating the central peripheral difference, wavelet transform is used to perform pyramid modeling on the feature image. In the process of normalizing the feature image, reconstruction is achieved by performing inverse wavelet transform on the low-resolution low-frequency image. The original scale image obtained by this method retains the important information in the image and has the same resolution as the original image. More importantly, it effectively retains the edge information of the image, which is conducive to the extraction and recognition of salient targets in the later image.
[0016] The wavelet pyramid model of adaptive feature images is implemented using the HLS high-level synthesis method and integrated into the FPGA in the form of IP;
[0017] Among them, HLS is developed in C++, and high-level synthesis uses function parallelization optimization to reduce processing cycles; pipelines are used to implement internal parallel optimization of for loops to reduce processing cycles; dataflow optimization reduces the occupancy of variable FIFO to significantly reduce resources.
[0018] Furthermore, it is integrated into FPGA in the form of IP, specifically:
[0019] 1) The design uses function parallelization to reduce the processing cycle; when the current content and result of a function no longer change and can be used for subsequent functions, the next function will be executed quickly, which greatly shortens the processing cycle and realizes parallelism between multiple for loops between different for loops;
[0020] 2) Use pipelines to achieve internal parallelism in for loops and reduce processing cycles; use pipelines to implement logic so that when the first variable in the for loop is calculated, the second variable is read. This independent pipeline allows for parallel execution of calculations in the for loop.
[0021] 3) Reduce the occupancy of variable FIFO to significantly reduce resources; use stream optimization to make data stream variables into FIFO format. Since the grayscale data is 8 bits, the FIFO depth is set to 8, and the read and write speed can be met without the need for full cache.
[0022] An infrared moving target real-time recognition system based on FPGA, comprising: a camera, a logic control module, a hardware acceleration module and a display module;
[0023] The camera is used for real-time acquisition of infrared video streams, with an input format of 640×512 and a frame rate of 100 Hz;
[0024] The logic control module is used for sending and receiving application program instructions, task scheduling and display processing;
[0025] The hardware acceleration module implements a wavelet pyramid model of adaptive feature images, realizes saliency detection of infrared targets and superposition of target borders on FPGA;
[0026] The display module is used to display the recognition result.
[0027] Furthermore, the logic control module and the hardware acceleration module are based on Zynq UltraScale+MPSoC, which is an ARM and FPGA heterogeneous chip. The ARM processor executes application logic control, and the FPGA processor executes algorithm processing functions. The heterogeneous cores of the same chip are closely integrated to achieve complex functions while reducing power consumption.
[0028] Beneficial effects:
[0029] 1. In the solution of the present invention, the algorithm adaptively adjusts the scale of the pyramid according to the target features, thereby improving the quality of the saliency map and reducing the processing delay.
[0030] 2. The present invention uses inverse wavelet transform to replace the linear interpolation of central peripheral difference, effectively retains the detailed contour information of the target, gives full play to the local information expression ability and multi-scale spatial analysis ability of image wavelet transform, and can make the final significant target image have better contour information.
[0031] 3. The algorithm of the present invention is deployed in the FPGA system to improve the recognition speed, achieve the effect of real-time video processing, and ultimately realize real-time recognition of infrared moving targets with high precision. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 The figure is a flow chart of the method of the present invention.
[0033] Figure 2 The present invention is a flow chart for obtaining a salient feature map.
[0034] Figure 3 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION
[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0036] The invention provides a real-time recognition method for infrared moving targets based on FPGA.
[0037] Embodiment 1
[0038] like Figure 1 As shown, the real-time infrared moving target recognition method based on FPGA provided by the embodiment of the present invention includes:
[0039] S101, FPGA acquires infrared video stream in real time and obtains infrared grayscale image features;
[0040] S102, HLS high-level synthesis, realizes the wavelet pyramid model of adaptive feature image, and obtains the target saliency map by central and peripheral difference operation;
[0041] S103, obtaining the target area and the background according to the salient feature map, extracting geometric features of the target area, and determining the target.
[0042] In the method for real-time target recognition in a complex environment described in the embodiment of the present invention, FPGA acquires infrared video stream in real time and obtains infrared grayscale image features; HLS high-level synthesis realizes the wavelet pyramid model of adaptive feature image, and obtains the target saliency map by central peripheral differential operation; obtains the target area and background according to the saliency feature map, extracts geometric features of the target area, and determines the target. In this way, the algorithm adaptively adjusts the scale of the pyramid according to the target features, improves the quality of the saliency map while reducing the processing delay; uses the inverse wavelet transform to replace the linear interpolation of the central peripheral differential, effectively retains the detailed contour information of the target, and gives full play to the local information expression ability and multi-scale spatial analysis ability of the image wavelet transform, so that the final saliency target image can have better contour information. Moreover, the algorithm is deployed in the FPGA system to improve the recognition speed, which can achieve the effect of real-time video processing, and finally can realize the real-time recognition of infrared moving targets with high precision.
[0043] In order to better understand the real-time target recognition method under a complex environment according to an embodiment of the present invention, a detailed description is given. The method may specifically include:
[0044] S101, FPGA acquires infrared video stream in real time and obtains infrared grayscale image features;
[0045] In this embodiment, the video data transmitted by the infrared camera is obtained through a high-speed differential transmission line, and an asynchronous FIFO is used to synchronize with the bus clock, and the row valid and frame valid signals are used to mark the data position;
[0046] The cross-clock domain synchronization frame frequency introduces an external 100MHz clock into the clock management MMCM and multiplies it to 200MHz, dividing it into three related clock domains to drive the underlying data processing modules; the AXI4 bus and the top-level state machine are driven by a 100MHz clock provided by the on-chip ARM system, which uses a synchronous method for accessing the register table, and an asynchronous method for large data transmission;
[0047] S102, HLS high-level synthesis, realizes the wavelet pyramid model of adaptive feature image, and obtains the target saliency map by central and peripheral difference operation. The algorithm flow is as follows Figure 2 shown.
[0048] In this embodiment, firstly, the input infrared grayscale feature is subjected to histogram equalization to determine the target index;
[0049] The target index is the ratio of the target pixels to the pixels of the entire image, and the formula is:
[0050]
[0051] The critical threshold M is determined according to the target and background type, and 5000 is selected for the ship target M. When the number of target pixels accounts for 1 / 4 or more of the total image area, the target can be considered to be a large target, and a multi-layer scale pyramid is used.
[0052] Specifically, the pyramid scale parameter calculation formula is:
[0053] N = target index × target coefficient (N ≥ 3)
[0054] Among them, N is rounded up, and when N<3, it is 3.
[0055] When the number of target pixels accounts for a small proportion of the total image area, the target in the image can be identified as a small target. Excessive downsampling will cause the targets in the high layers to disappear, causing some differential operations to become redundant. Therefore, appropriately reducing the number of pyramid layers can better adapt to small target detection. At the same time, for small targets, reducing the number of pyramid layers can not only improve the quality of the saliency map, but also reduce processing delay. The target coefficients of different targets are different. For ship targets, M is selected as 25.
[0056] Different layers of the pyramid have different resolutions. The bottom layer is the original image, which has the highest resolution. High resolution can reflect more detailed information. The resolution decreases as you go up. Low resolution can better reflect the background information of the image. The upper layers can be regarded as representatives of a region in the original image. Therefore, the difference between a certain point and the background of the local area, that is, the difference between the large-scale and small-scale images in the pyramid, can reflect the contrast of the local area.
[0057] The central difference calculation formula is as follows:
[0058] I(c,s)=|I(c)ΘI(s)|
[0059] Where Θ represents the subtraction operation after adjusting the sizes of the two images to be consistent, and I(c, s) is the obtained feature saliency map.
[0060] Traditional methods use linear interpolation algorithms to normalize the scale of features at different levels. This process leads to a reduction in the resolution of the feature image. The salient targets displayed in the final salient image are incomplete and a large amount of detail information is lost. Therefore, in the process of calculating the central peripheral difference, wavelet transform is used to pyramid model the feature image. In the process of normalizing the feature image, the low-resolution low-frequency image is reconstructed by inverse wavelet transform. The original scale image obtained by this method retains the important information in the image and has the same resolution as the original image. More importantly, it effectively retains the edge information of the image, which is conducive to the extraction and recognition of salient targets in the later image.
[0061] Normalize the initially obtained feature saliency map:
[0062] S′=(SS min ) / (SS max )
[0063] Among them, S is the final feature saliency map, S min is the minimum grayscale value in the feature saliency map, S max is the maximum grayscale value in the feature saliency map.
[0064] Use HLS (High Level Synthesis) high-level synthesis to implement the wavelet pyramid model IP of adaptive feature images and integrate it into FPGA. HLS is an FPGA high-level synthesis tool. The wavelet pyramid model of adaptive feature images is developed in C++ and then synthesized into IP for calling on FPGA and the following method is used to optimize the comprehensive compilation.
[0065] 1) The design uses function parallelization to reduce the processing cycle. When the current content and result of a function no longer change and can be used for subsequent functions, the next function will be executed quickly, which greatly shortens the processing cycle and realizes parallelism between multiple for loops between different for loops.
[0066] 2) Use pipelines to achieve internal parallelism in for loops and reduce processing cycles. Use pipelines to implement logic, such as reading, calculating, and writing values, so that when the first variable in the for loop is calculated, the second variable is read. This independent pipeline allows for parallel execution of calculations within the for loop.
[0067] 3) Reduce the occupancy of variable FIFO to significantly reduce resources. Use stream optimization to make the data stream variable into FIFO format. Since the grayscale data is 8 bits, the FIFO depth is set to 8, and the read and write speed can be met without the need for full cache.
[0068] S103, obtaining the target area and the background according to the salient feature map, extracting geometric features of the target area, and determining the target.
[0069] In this embodiment, the pixels of the fused saliency map are divided into a target area and a background area according to a threshold value W, and then the threshold value W is continuously optimized to obtain an optimized target area and an optimized background area with the purpose of maximizing the inter-class variance.
[0070] The target geometric features include minimum bounding rectangle, aspect ratio, area, etc. For targets such as ships, the target type can be quickly determined based on the minimum bounding rectangle, aspect ratio, area, etc.
[0071] Embodiment 2
[0072] The embodiment of the present invention also provides a real-time infrared moving target recognition system based on FPGA. Figure 3 As shown, it includes: an infrared camera, a logic control module, a hardware acceleration module and a display module;
[0073] The camera is used for real-time acquisition of infrared video streams, with an input format of 640×512 and a frame rate of 100 Hz;
[0074] The logic control module is used for sending and receiving application program instructions, task scheduling and display processing;
[0075] The hardware acceleration module implements a wavelet pyramid model of adaptive feature images, realizes saliency detection of infrared targets and superposition of target borders on FPGA;
[0076] The display module is used to display the recognition result;
[0077] Furthermore, the logic control module and the hardware acceleration module are based on Zynq UltraScale+MPSoC, which is an ARM and FPGA heterogeneous chip. The ARM processor executes application logic control, and the FPGA processor executes algorithm processing functions. The heterogeneous cores of the same chip are closely integrated to achieve complex functions while reducing power consumption.
[0078] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A real-time infrared moving target recognition method based on FPGA, characterized in that: include: FPGA acquires infrared video stream in real time and obtains infrared grayscale image features; HLS high-level synthesis realizes the wavelet pyramid model of adaptive feature images, and obtains significant feature maps through central and peripheral difference operations; The target area and background are obtained according to the salient feature map, the geometric features of the target area are extracted, and the target is determined.
2. The real-time infrared moving target recognition method based on FPGA as claimed in claim 1 is characterized in that: The HLS high-level synthesis realizes the wavelet pyramid model of adaptive feature image, and the central peripheral difference operation obtains the target significant feature map including: Perform histogram equalization on the input infrared grayscale features to determine the target index; The target index is the ratio of the target pixels to the pixels of the entire image, and the formula is: Among them, the critical threshold M is determined according to the target and background type, and 5000 is selected for the infrared ship moving target. When the number of target pixels accounts for 1 / 4 or more of the total image area, the target can be considered to be a large target, and a multi-layer scale pyramid is used. When the number of target pixels accounts for a small proportion of the total image area, the target in the image can be identified as a small target, and the number of pyramid layers is reduced.
3. The real-time infrared moving target recognition method based on FPGA as claimed in claim 1 or 2, characterized in that: In the process of calculating the central peripheral difference, wavelet transform is used to build pyramid model of the feature image. In the process of normalizing the feature image, the low-frequency image with low resolution scale is reconstructed by inverse wavelet transform. The wavelet pyramid model of adaptive feature images is implemented using the HLS high-level synthesis method and integrated into the FPGA in the form of IP; Among them, HLS is developed in C++, and high-level synthesis uses function parallelization optimization to reduce processing cycles; pipelines are used to implement internal parallel optimization of for loops to reduce processing cycles; dataflow optimization reduces the occupancy of variable FIFO to significantly reduce resources.
4. The real-time infrared moving target recognition method based on FPGA as claimed in claim 3 is characterized in that: Integrate into FPGA in the form of IP, specifically: 1) The design uses function parallelization to reduce the processing cycle; when the current content and result of a function no longer change and can be used for subsequent functions, the next function will be executed quickly, which greatly shortens the processing cycle and realizes parallelism between multiple for loops between different for loops; 2) Use pipelines to achieve internal parallelism in for loops and reduce processing cycles; use pipelines to implement logic so that when the first variable in the for loop is calculated, the second variable is read. This independent pipeline allows for parallel execution of calculations in the for loop. 3) Reduce the occupancy of variable FIFO to significantly reduce resources; Using stream optimization, the data stream variables are made into FIFO format. Since the grayscale data is 8 bits, the FIFO depth is set to 8, and the reading and writing speeds can be met without the need for full cache.
5. A real-time infrared moving target recognition system applied to the method of claim 1, characterized in that: include: Camera, logic control module, hardware acceleration module and display module; The camera is used for real-time acquisition of infrared video streams, with an input format of 640×512 and a frame rate of 100 Hz; The logic control module is used for sending and receiving application program instructions, task scheduling and display processing; The hardware acceleration module implements a wavelet pyramid model of adaptive feature images, realizes saliency detection of infrared targets and superposition of target borders on FPGA; The display module is used to display the recognition result.
6. The infrared moving target real-time recognition system according to claim 5, characterized in that: The logic control module and hardware acceleration module are based on Zynq UltraScale+MPSoC, which is an ARM and FPGA heterogeneous chip. The ARM processor performs application logic control, and the FPGA processor performs algorithm processing functions. The heterogeneous cores of the same chip are closely integrated to achieve complex functions while reducing power consumption.