Variable-scale ground target tracking method and system based on FPGA (Field Programmable Gate Array)

Abstract

The invention relates to an FPGA-based variable-scale ground target tracking method and system. The method comprises the following steps: carrying out frame-by-frame preprocessing on an input video stream; constructing a parallel computing window based on the input image size and the target scale, and caching template data; performing normalized cross correlation coefficient calculation by using the to-be-matched template data and the calculation window, and selecting the central point of the corresponding window with the maximum cross correlation coefficient as a tracking result; and estimating and updating the template according to the calculated target scale change, and selecting templates at different intervals to match subsequent frame images. According to the method, the template matching stability under large-scale change of ground target tracking is improved, meanwhile, the method is easy to implement in an image real-time processing system with an FPGA as an accelerator, and the method can be used for real-time tracking application scenes of air-to-ground target scale change.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and particularly to a variable-scale ground target tracking method and system based on FPGA. Background Art

[0002] With the rapid development of the UAV industry and low-cost precision-guided weapons, target tracking algorithms based on optical video have been widely applied in military and civilian fields. At the same time, in the application scenarios of air-ground target tracking where the light intensity changes, the background texture is not rich, and the target scale changes greatly, how to ensure the robustness and real-time performance of the algorithm has become the key to the application of target tracking algorithms.

[0003] In the prior art, a Chinese invention patent with the publication number of CN108550161A discloses a scale-adaptive kernel correlation filtering fast target tracking method. By introducing a two-dimensional variable-bandwidth Gaussian window in the KCF framework to promote the separation of the foreground and background and adding a scale estimation module, the scale change of the target is estimated through the matching result of the feature points of the image block. On this basis, displacement prediction can also be combined to determine whether the target is occluded. Once the target is occluded, it is considered that there is no scale transformation at this time and the template is not updated, thereby enhancing the accuracy of tracking in this way. The advantages of this invention are simple and fast calculation, and it can significantly improve the operation efficiency on the basis of being compatible with scale adaptation.

[0004] A Chinese invention patent with the publication number of CN107516296A discloses a moving target detection and tracking system and method based on FPGA. First, the image data is converted and then output in two paths; one path enters the external DDR3 storage module through the data input DDR3 FIFO module for caching, and the other path enters the DDR3 storage module for caching after RGB to grayscale processing and median filtering processing; the coordinate values of the moving template are obtained by performing frame difference operation, binarization processing, and erosion and dilation processing on two consecutive frames of grayscale image data continuously output from the DDR3 buffer, and then fed back to the RGB format color image to obtain the moving target template, which is stored in the on-chip RAM of the FPGA, and finally used for moving template matching to achieve the tracking of the color moving target. This invention has achieved a good balance in terms of target recognition speed, accuracy, cost, power consumption, etc.

[0005] However, in existing object tracking methods based on detection, the performance of the tracking method highly depends on the effectiveness of object detection. However, in the application scenario of air-ground object tracking, there are often problems such as poor texture information of the tracked object and large changes in light intensity, which greatly affect the performance of the object detector. In object tracking methods based on template matching, due to limitations in computing resources or FPGA pipelined computing, the object tracking is achieved by using a single fixed template, and the update strategy of the tracking template is not reasonably designed, affecting the robustness and accuracy of object tracking.

[0006] In the application fields of UAV platforms and precision-guided weapons, implementing a tracking algorithm on a low-cost computing platform and ensuring the real-time performance of the algorithm are the keys to engineering implementation. Existing methods rarely conduct targeted algorithm design according to the characteristics of the computing platform, and it is difficult to balance the characteristics of low cost and high performance. Summary of the Invention

[0007] To solve the above technical problems existing in the prior art, the object of the present invention is to provide an FPGA-based variable-scale ground target tracking method and system, which can improve the template matching stability under large-scale changes in ground target tracking, and is easy to implement in an FPGA-based real-time image processing system as an accelerator, and can be used in real-time tracking application scenarios of air-to-ground target scale changes.

[0008] To achieve the above object of the invention, the present invention provides an FPGA-based variable-scale ground target tracking method, including the following steps:

[0009] Step S1: Perform frame-by-frame preprocessing on the input video stream;

[0010] Step S2: Construct a parallel computing window based on the input image size and target scale, and cache the template data;

[0011] Step S3: Calculate the normalized cross-correlation coefficient between the template data to be matched and the computing window, and select the center point of the corresponding window with the largest cross-correlation coefficient as the tracking result;

[0012] Step S4: Estimate and update the template according to the calculated target scale change, and select templates at different intervals for matching subsequent frame images.

[0013] According to a technical solution of the present invention, in step S1, the input video stream is received, and each frame of the image is read, and the noise contained in the input image is removed by performing fast median filtering using a 3×3 window.

[0014] According to a technical solution of the present invention, in step S2, the parallel computing window is constructed through a row buffer queue, the window size is 32×32, and the input image is sampled at equal intervals.

[0015] According to a technical solution of the present invention, in the step S3, the normalized cross-correlation calculation is performed using the following formula:

[0016]

[0017] Among them, C(m,n) represents the calculated normalized cross-correlation coefficient value, I(m,n) represents the sum of the image window, T(m,n) represents the sum of the template window, I 2 (m,n) represents the sum of the squares of each pixel of the image window, T 2 (m,n) represents the sum of the squares of each pixel of the template window, and IT(m,n) represents the sum of the matrix dot products of the image window and the template window.

[0018] According to a technical solution of the present invention, in the step S4, the steps of the template update strategy include:

[0019] Step S41, calculate the target scale change value between the previous and current frames, and judge according to the maximum value of the cross-correlation coefficient between the two frames;

[0020] Step S42, if the maximum value of the normalized cross-correlation coefficient of the current frame is lower than the set threshold, select the template corresponding to the sampling scale for update.

[0021] According to a technical solution of the present invention, in the step S41, the target scale change value between the previous and current frames is calculated according to the target scale estimated value, and the calculation formula is:

[0022]

[0023] Among them, C max (T t+1 ) is the maximum value of the normalized cross-correlation coefficient at time t + 1, C max (T t ) is the maximum value of the normalized cross-correlation coefficient at time t, and ρ is the target scale change value.

[0024] According to a technical solution of the present invention, in the step S42, if the maximum value C of the normalized cross-correlation coefficient of the current frame max > C 阈值 , and the current template has not been equally spaced sampled, the template is not updated. If the current template has been equally spaced sampled, the 32×32 template data without sampling is used for template update;

[0025] If the maximum value C of the normalized cross-correlation coefficient of the current frame max ≤ C 阈值, then the target scale change value is judged again. If the target scale change value ρ ≤ 1, the 32×32 template data without sampling is used for template update; if the target scale change value 1 < ρ ≤ 2, the 32×32 template data obtained by equally spaced sampling of the 64×64 image window is used for template update; if the target scale change value ρ > 2, the 32×32 template data obtained by equally spaced sampling of the 128×128 image window is used for template update.

[0026] According to one aspect of the present invention, a variable-scale ground target tracking system based on FPGA is proposed, including:

[0027] A preprocessing module for frame-by-frame preprocessing of the input video stream;

[0028] A cache module for constructing a parallel computing window based on the input image size and the target scale, and caching the template data;

[0029] A matching calculation module for calculating the normalized cross-correlation coefficient by using the template data to be matched and the calculation window, and selecting the center point of the corresponding window with the largest cross-correlation coefficient as the tracking result;

[0030] A template update module for updating the template according to the estimated target scale change obtained by calculation, and selecting templates with different intervals for matching of subsequent frame images.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The variable-scale ground target tracking method and system based on FPGA of the present invention cache the matching templates of multiple sampling intervals while constructing the image calculation window, and update the templates by using the method of estimating the target scale change, which enhances the robustness of the air-to-ground target tracking method and improves the target tracking accuracy and stability under the condition of large scale change in the near target end segment. At the same time, in the matrix dot product calculation unit, the column parallel calculation combined with multi-column pipelining is adopted to find the best balance between the calculation speed and resources, reduce the resource requirements for the algorithm to be transplanted and deployed on low-cost FPGAs, and has the advantages of high real-time performance, good stability and easy implementation in embedded systems.

[0033] The present invention can stably track targets with less texture information for the air-to-ground target tracking scenario, especially in the application requirements of seeker end segment target tracking, and has broad application prospects in unmanned aerial vehicle platforms and infrared seekers of precision guided weapons. Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0035] Figure 1 Schematically showing a flowchart of a variable-scale ground target tracking method based on FPGA according to an embodiment of the present invention;

[0036] Figure 2 Schematically showing a flowchart of a variable-scale ground target tracking method based on FPGA according to another embodiment of the present invention;

[0037] Figure 3 Schematically showing a block diagram of data preprocessing and parallel computing window construction according to an embodiment of the present invention;

[0038] Figure 4 Schematically showing a schematic diagram of a matrix dot product unit according to an embodiment of the present invention. Specific embodiments

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0040] As Figure 1 shown, a variable-scale ground target tracking method and system based on FPGA of the present invention has an overall design scheme including three parts. First, frame-by-frame preprocessing, computing window construction, and template caching are performed on the input video stream. Second, normalized cross-correlation matching calculation based on the template is performed, and the center point of the corresponding window with the largest cross-correlation coefficient is selected as the tracking result output. Finally, different sampling intervals of templates are selected for template update according to the calculated target scale change estimation result. The present invention is conducive to parallel acceleration design and implementation on the FPGA platform, and at the same time can improve the accuracy of tracking targets under variable-scale conditions.

[0041] A variable-scale ground target tracking method based on FPGA of the present invention includes the following steps:

[0042] Step S1: Perform frame-by-frame preprocessing on the input video stream;

[0043] Receive the input video stream, read each frame of the image, and perform fast median filtering using a 3×3 window to remove the noise contained in the input image.

[0044] Step S2: Construct a parallel computing window based on the input image size and the target scale, and cache the template data;

[0045] Establish a 32×32 parallel computing window with the same sampling rate as the template for subsequent normalized cross-correlation template matching calculations.

[0046] As Figure 3 shown, in the process of template matching of the present invention, by comparing the magnitudes of the cross-correlation coefficients of the two consecutive computing windows, if the cross-correlation coefficient of the latter window is greater than that of the former window, then equidistant sampling is performed through image windows of three scales of 32×32, 64×64, and 128×128 to obtain three 32×32 templates to be matched. When all window matching calculations are completed, the template data corresponding to the maximum cross-correlation coefficient at the three scales is obtained. Then, according to the result of the scale estimation module, if the template data needs to be updated, one of the sampled template data at the sampling interval is selected as the template to be matched for the subsequent frame image matching.

[0047] It should be particularly noted that the size of the parallel computing window is 32×32. Equidistant sampling is performed on the input image according to the scale of the template to construct the computing window, and full-image search is performed in the form of a sliding window.

[0048] Step S3: Calculate the normalized cross-correlation coefficient using the template data to be matched and the computing window, and select the center point of the corresponding window with the maximum cross-correlation coefficient as the tracking result;

[0049] The present invention calculates the normalized cross-correlation coefficient between the image window and the template data to obtain the best matching position of the tracked target. The specific calculation formula is as follows.

[0050]

[0051] For the sake of facilitating the pipelining implementation of the FPGA, the formula is simplified to obtain the following formula.

[0052]

[0053] Among them, C(m,n) represents the calculated normalized cross-correlation coefficient value, I(m,n) represents the sum of the image window, T(m,n) represents the sum of the template window, I 2 (m,n) represents the sum of the squares of each pixel of the image window, T 2 (m,n) represents the sum of the squares of each pixel of the template window, and IT(m,n) represents the sum of the matrix dot products of the image window and the template window.

[0054] The adder with multi-stage pipelining in the present invention is used to calculate the sum of the data of the image window I(m,n) and the template T(m,n). The calculation of IT(m,n), I 2 (m,n) and T 2 (m,n) is realized through the dot product calculation unit and accumulator of the matrix, and then the value of C(m,n) is obtained through the square root operation and floating-point division operation.

[0055] The above-mentioned dot product calculation unit of the matrix first realizes the multiplication of column elements, and then realizes the sum of the multiplication results of multiple column elements through the accumulator, so as to balance the resources and speed.

[0056] As Figure 4 shown, in order to match the pipeline structure of the multiply-accumulator, it is necessary to perform ladder delay on the two window row caches of the calculation by supplementing data with a value of 0. The specific implementation process is that the length of the queue in the next row is increased by 1 delay cycle compared with the previous row. The first row data of matrix A and matrix B are input into multiplier 1 at the same time, the second row data of matrix A and matrix B are input into multiplier 2 at the same time, and the Nth row data of matrix A and matrix B are input into multiplier N at the same time. When all the data of the entire calculation window 32×32 (N = 32) enter the multiplier and complete the accumulator calculation, the matrix dot product operation of a corresponding window is ended. The logic control and reset signal generator clears and resets the multiplier and accumulator, and pulls down the enable signal of the row cache queue (FIFO, high level enables effectively) for one clock cycle, and waits for the next clock cycle to start the matrix dot product calculation of the next window.

[0057] Step S4: Update the template according to the calculated target scale change estimation, and select templates with different intervals for the matching of subsequent frame images. The steps of the template update strategy include:

[0058] Step S41: Calculate the target scale change value between the previous and current frames, judge according to the maximum value of the cross-correlation coefficient between the two frames, and calculate the target scale change value between the previous and current frames according to the target scale estimation value. The calculation formula is:

[0059]

[0060] Among them, C max (T t+1 ) is the maximum value of the normalized cross-correlation coefficient at time t + 1, C max (T t ) is the maximum value of the normalized cross-correlation coefficient at time t, and ρ is the target scale change value.

[0061] Step S42: If the maximum value of the normalized cross-correlation coefficient of the current frame is lower than the set threshold, select the template corresponding to the sampling scale for update.

[0062] Under normal circumstances, the threshold setting range is between 0.8 and 0.98, and it can be flexibly configured according to the differences in imaging systems (visible light, infrared), such as 0.92. Taking the threshold setting of 0.9 as an example, the template update strategy steps of the present invention are as follows:

[0063] If the maximum value C of the normalized cross-correlation coefficient of the current frame max > 0.9, and the current template has not been equally spaced sampled, then the template is not updated. If the current template has been equally spaced sampled, then the non-sampled 32×32 template data is used for template update.

[0064] If the maximum value C of the normalized cross-correlation coefficient of the current frame max ≤ 0.9, it is necessary to judge the target scale change value again. If the target scale change value ρ ≤ 1, the non-sampled 32×32 template data is used for template update; if the target scale change value 1 < ρ ≤ 2, the 32×32 template data obtained by equally spaced sampling of the image window 64×64 is used for template update; if the target scale change value ρ > 2, the 32×32 template data obtained by equally spaced sampling of the image window 128×128 is used for template update.

[0065] In the present invention, the matching degree between the current template and the target in the image window is judged by the maximum value of the cross-correlation coefficient. When the maximum value of the cross-correlation coefficient is less than the threshold, according to the change degree of the cross-correlation coefficient, the template with the corresponding sampling scale is selected for update, so that the size of the template data can not only cover the range of the tracking target, but also introduce relatively small background noise into the template data.

[0066] In some embodiments of the present invention, as Figure 2 shown, first, after the image is input, fast median filtering is performed, a parallel computing window is constructed, and a template cache unit is constructed; if it is the first frame for target tracking, only the template data is cached according to the input initial tracking point. If it is not the first tracking frame, the template data with a suitable sampling interval is selected as the template to be matched, the normalized cross-correlation calculation is performed on the established image window and the template data, the window center point coordinates corresponding to the maximum cross-correlation coefficient are obtained as the tracking result through comparison of the calculated cross-correlation coefficients. At the same time, through the scale change estimation of the cross-correlation coefficient, it is determined whether to update the template and the sampling interval of the updated template data. And the corresponding template to be matched is determined, and wait for the next frame of image to be input before performing the matching calculation.

[0067] According to one aspect of the present invention, a variable-scale ground target tracking system based on FPGA is proposed, including:

[0068] A preprocessing module for performing frame-by-frame preprocessing on the input video stream;

[0069] The cache module constructs a parallel computing window based on the input image size and the target scale, and caches the template data;

[0070] The matching calculation module is used to calculate the normalized cross-correlation coefficient between the template data to be matched and the computing window, and select the center point of the corresponding window with the largest cross-correlation coefficient as the tracking result;

[0071] The template update module is used to update the template according to the estimated change of the target scale obtained by calculation, and select templates with different intervals for matching subsequent frame images.

[0072] It should be noted that although the embodiments described above of the present invention are illustrative, this is not a limitation of the present invention. Therefore, the present invention is not limited to the above specific embodiments. Without departing from the principle of the present invention, any other embodiments obtained by those skilled in the art under the inspiration of the present invention are regarded as within the protection scope of the present invention.

Claims

1. A variable-scale ground target tracking method based on FPGA, characterized in that, It includes the following steps: Step S1: Perform frame-by-frame preprocessing on the input video stream; Step S2: Construct a parallel computing window based on the input image size and the target scale, and cache the template data; Step S3: Use the template data to be matched and the computing window to calculate the normalized cross-correlation coefficient, and select the center point of the corresponding window with the largest cross-correlation coefficient as the tracking result; Step S4: Estimate and update the template according to the calculated target scale change, and select templates at different intervals for matching subsequent frame images.

2. The method for variable-scale ground target tracking based on FPGA according to claim 1, wherein In step S1, receive the input video stream, read each frame image, and remove the noise contained in the input image by performing fast median filtering using a 3×3 window.

3. The method for variable-scale ground target tracking based on FPGA according to claim 1, characterized in that, In step S2, the construction of the parallel computing window is implemented through a row cache queue, the window size is 32×32, and the input image is sampled at equal intervals.

4. The method for variable-scale ground target tracking based on FPGA according to claim 1, wherein In step S3, the normalized cross-correlation calculation is performed using the following formula: Among them, C(m,n) represents the calculated normalized cross-correlation coefficient value, I(m,n) represents the sum of the image window, T(m,n) represents the sum of the template window, I 2 (m,n) represents the sum of the squares of each pixel in the image window, T 2 (m,n) represents the sum of the squares of each pixel in the template window, and IT(m,n) represents the sum of the matrix dot products of the image window and the template window.

5. The method for variable-scale ground target tracking based on FPGA according to claim 1, characterized in that In step S4, the steps of the template update strategy include: Step S41: Calculate the target scale change value between the previous and current frames, and make a judgment based on the maximum cross-correlation coefficient of the two frames; Step S42: If the maximum value of the normalized cross-correlation coefficient of the current frame is lower than the set threshold, select the template corresponding to the sampling scale for update.

6. The method for variable-scale ground target tracking based on FPGA according to claim 5, characterized in that, In step S41, calculate the target scale change value between the previous and current frames according to the target scale estimation value, and the calculation formula is: Among them, C max (T t+1 0 is the maximum value of the normalized cross-correlation coefficient at time t + 1, C max (T t ) is the maximum value of the normalized cross-correlation coefficient at time t, and ρ is the target scale change value.

7. The method for variable-scale ground target tracking based on FPGA according to claim 5, wherein In the step S42, if the maximum value C of the normalized cross-correlation coefficient of the current frame max > C 阈值 , and the current template has not been equally-spaced sampled, the template is not updated. If the current template has been equally-spaced sampled, 32 = 32 template data without sampling are used for template update; If the maximum value C of the normalized cross-correlation coefficient of the current frame max ≤C 阈值 , then the target scale change value is judged again. If the target scale change value ρ ≤ 1, the 32×32 template data without sampling is used for template update; If the target scale change value 1 < ρ ≤ 2, use the 32×32 template data sampled at equal intervals from the 64×64 image window for template update; If the target scale change value ρ > 2, use the 32×32 template data sampled at equal intervals from the 128×128 image window for template update.

8. A variable-scale ground target tracking system based on FPGA, characterized in that, It includes: A preprocessing module for performing frame-by-frame preprocessing on the input video stream; A caching module for constructing a parallel computing window based on the input image size and the target scale, and caching the template data; A matching calculation module for using the template data to be matched and the computing window to calculate the normalized cross-correlation coefficient, and selecting the center point of the corresponding window with the largest cross-correlation coefficient as the tracking result; A template update module for estimating and updating the template according to the calculated target scale change, and selecting templates at different intervals for matching subsequent frame images.

Images