Method for extracting moving target under complex background based on FPGA (Field Programmable Gate Array)

Abstract

A method for extracting a moving target under a complex background based on an FPGA (Field Programmable Gate Array) comprises the following steps: acquiring a complex background image video stream as to-be-processed image data in real time through an infrared imaging detection system, resolving by using the infrared imaging detection system to obtain an optical axis offset as priori information, and compensating the amount of movement of an inter-frame complex background, so as to extract a moving target under the complex background; and on the basis, traversing inter-frame motion of an image background through translation of n2 pixels in the upper, lower, left and right directions, compensating offset of a previous frame of image by minimizing a difference value of inter-frame images and obtaining inter-frame offset of the images, and performing difference on the current image and the compensated previous frame of image to achieve the effects of suppressing the background and improving the image quality. And the target effect is enhanced. And carrying out binarization and segmentation on the difference image, and extracting a target according to information between target frames. According to the method, the moving target under the complex background can be rapidly extracted, and compared with the traditional algorithm implementation based on the DSP, the operation time is only 1 / 6 of the original operation time.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and recognition, and relates to a method for extracting a moving target under a complex background. Background Art

[0002] Image processing and image recognition technology is one of the hottest fields today. In particular, target extraction is the prerequisite for advanced pattern recognition and artificial intelligence systems, and has a wide range of applications in civil, military, aerospace and other fields.

[0003] The problem of moving target extraction under complex background has always been a difficult problem to be solved by infrared imaging detection systems. On the one hand, infrared long-distance targets have the characteristics of weak targets, and the downward-looking background is complex and changeable, showing high-frequency characteristics similar to the target, which can easily submerge the target in the background and cause problems such as missed detection or false alarm; on the other hand, the infrared imaging detection system has very high requirements for the real-time implementation of the algorithm, and the traditional DSP platform has too long computing time, which limits the practical application of the algorithm.

[0004] The traditional frame difference method does not fully consider the movement of the complex background itself, and differentiates the moving complex background from the image, thereby increasing the false alarm rate of the target. The frame difference method based on feature point registration requires the extraction of a large number of stable feature points to estimate the movement of the background in the early stage, which is difficult for infrared images themselves: on the one hand, the contour edges of infrared images are not sharp enough, and it is difficult to extract a large number of feature points that can be used for matching; on the other hand, extracting stable feature points itself is very time-consuming. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a method for extracting moving targets in complex backgrounds based on FPGA. Based on prior information, the entire image area is involved in the estimation of background motion to solve the problem of difficult and time-consuming feature point extraction, and the target extraction speed can be greatly improved to meet the real-time requirements of the algorithm. It can be embedded in micro devices for wider application.

[0006] The technical solution of the present invention is: a method for extracting moving targets under complex backgrounds based on FPGA, comprising:

[0007] (1) Using an infrared imaging detection system to collect video streams containing complex background images in real time as images to be processed;

[0008] (2) The optical axis offset calculated by the infrared imaging detection system is used as prior information to compensate for the movement of the complex background between the image frames to be processed;

[0009] (3) Traversing the inter-frame motion of the image to be processed by pixel translation, minimizing the difference between the inter-frame images and obtaining the inter-frame offset of the image to be processed, and compensating for the offset of the previous frame image;

[0010] (4) Differentiate the current image and the compensated previous image, binarize and segment the differential image, and extract the target based on the information between the image frames.

[0011] Preferably, the optical axis offset is in four directions: up, down, left, and right, and the range of each direction is between 0 and n1 pixels. The n1 is 20.

[0012] Furthermore, the movement of the complex background between the image frames to be processed is compensated, specifically: the previous frame image P and the current image g are cached in the on-chip RAM, the image actually involved in the complex background motion estimation is the N×N pixels in the center of the current image g, and the previous frame image P is also based on the N×N pixels in the center; the area where the N×N pixels of the previous frame image P are located is offset according to the optical axis offset (x, y), and its size is expanded by n2 pixels in the upper, lower, left and right directions to obtain an area of ​​(N+2n2)×(N+2n2) size as the compensation result.

[0013] Preferably, n2 is 5, and N is 312. The inter-frame motion of the image to be processed is traversed by pixel translation, and the translation mode is n2 pixels each for the top, bottom, left, and right sides.

[0014] Furthermore, the minimization of the inter-frame image difference and the inter-frame offset of the image to be processed is specifically as follows: constructing a shift array of length 2n2×(N+2n2)+2n2+1, and pushing the data of the previous frame image P into the array from the first address in sequence from the end of the array in FIFO mode until 2n2×(N+2n2)+2n2+1 clocks have passed, and the first (2n2+1)×(2n2+1) data participating in the calculation is pushed as a whole to the upper left corner of the shift array, and (2n2+1)×(2n2+1) accumulators are initialized at the same time; when the next clock arrives, the first valid pixel of the current image g is compared with the (2n2+1)×(2n2+1) data in the upper left corner of the shift array to obtain the absolute value of the difference, assuming that the pixel value of the current image g is g(x1,y1), which is the same as the nth data P n The absolute value of the difference is d(x1,y1,n), which is put into the corresponding (2n2+1)×(2n2+1) accumulators for differential value accumulation to obtain d(n).

[0015]

[0016] d(n)=d(x1,y1,n)+d(n)

[0017] Where n>0 and n≤(2n2+1)×(2n2+1)), M*M is the size of the image to be processed;

[0018] When the next clock arrives, the next pixel of the previous frame image P is pushed into the end of the shift array in FIFO mode, and d(n) is calculated again, and the cycle repeats until the N×N valid pixels of the current image g are all operated.

[0019] Compare the values ​​of the (2n2+1)×(2n2+1) accumulators in pairs and find the index number n with the minimum value, which is the index number corresponding to the offset of the previous frame image P relative to the current image g. The offset (x2, y2) is obtained by reverse calculation through the index number, where x2 is the row offset and y2 is the column offset.

[0020] Furthermore, the offset (x2, y2) is obtained by inverse calculation through the index number, specifically:

[0021]

[0022] Furthermore, the difference between the current image and the compensated previous frame image is specifically as follows:

[0023] Find the pixel with the subscript overall offset (x2, y2) from the previous frame image P, and use it as the starting pixel point (M-2n1-2n2-2)×(M-2n1-2n2-2) pixel area and the center (M-2n1-2n2-2)×(M-2n1-2n2-2) pixel area image of the current image g for difference, and get the differenced image D:

[0024]

[0025] Accumulate the difference value of each pixel into accumulator A:

[0026] A=A+D(x1,y1)

[0027] Among them, x1,y1∈[n1+n2+1,M-n1-n2-2];

[0028] The calculation and accumulation operations of the differential image are completed after (M-2n1-2n2-2)×(M-2n1-2n2-2) clocks, and the differential image is cached in the on-chip RAM.

[0029] Furthermore, the differential image is binarized and segmented, specifically by calculating the segmentation threshold T according to the cumulative sum A.

[0030]

[0031] Among them, Q is the number of pixels participating in the accumulator, which is (M-2n1-2n2-2)×(M-2n1-2n2-2), and α is the coefficient.

[0032] The advantages of the present invention compared with the prior art are:

[0033] (1) The present invention adopts an enumeration method based on prior information to estimate the inter-frame motion of the complex background, and performs differentiation of the background through motion compensation, thereby suppressing the background and enhancing the target. Compared with the traditional complex background motion target extraction algorithm based on filtering, the present invention better suppresses the complex background; compared with the traditional inter-frame difference method, the motion of the complex background itself is taken into account, and the problem of increasing the target false alarm rate by differentiating the moving complex background is solved, which is more reasonable; compared with the inter-frame difference method based on feature point registration, there is no need to extract a large number of stable feature points to estimate the background motion in the early stage, and the whole image area is involved in the estimation of the background motion based on the prior information, which solves the problem of difficulty and time consumption in feature point extraction, and the present algorithm is simple and easier to implement;

[0034] (2) The present invention is implemented based on an FPGA platform and adopts a pipeline working mode, which greatly improves the speed of target extraction and meets the real-time requirements of the algorithm. It can be embedded in micro devices, so that the target extraction algorithm can be more widely used and is suitable for the extraction of moving targets under complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 A schematic diagram of the present invention performing an expansion operation on a previous frame of image;

[0037] Figure 3 It is a schematic diagram of the shift array of the present invention;

[0038] Figure 4 This is a schematic diagram of updating the shift array data of the present invention;

[0039] Figure 5 It is a schematic diagram of the present invention for reversely calculating the offset through the index number. DETAILED DESCRIPTION

[0040] The method of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0041] The present invention provides a method for extracting moving targets in complex backgrounds based on FPGA. Figure 1A flow chart of the method is given. The infrared imaging detection system collects 512*512@100Hz complex background image video stream in real time as the image data to be processed. The FPGA main clock frequency is 100MHz and the clock period is 10ns.

[0042] It should be noted that, in order to highlight the fast processing speed of the method of the present invention, a 512*512@100Hz complex background image video stream is used for illustration in a real-time system with high requirements for processing speed. More generally, an M*M video stream is also applicable.

[0043] The infrared imaging detection system calculates the optical axis offset (four directions, up, down, left, and right, each direction ranges from 0 to n1 pixels, and the selection of n1 is estimated based on the maximum inter-frame motion capability of the optical axis of the real-time system. In this system, n1 is 20) as prior information to compensate for the movement of the complex background between frames. On this basis, the inter-frame motion of the image background is traversed by the translation of n2 pixels up, down, left, and right (usually 5 pixels are sufficient). By minimizing the difference between the inter-frame images and obtaining the inter-frame offset of the image, the offset of the previous frame image is compensated. Differentiating the current image and the compensated previous frame image can suppress the background and enhance the target. Binarize and segment the differential image, and extract the target based on the information between the target frames. The specific steps are as follows:

[0044] S1: Cache the previous frame image P and the current image g in the on-chip RAM. Assume that the size of each image is 512×512×16 bits, and the image actually involved in the complex background motion estimation is the N×N pixels in the center of the current image, and the previous frame image is also based on the N×N pixels in the center. After receiving the external input optical axis offset (x, y), the N×N pixel area of ​​the previous frame image is offset and its size is expanded by a maximum of n2 pixels up, down, left, and right, such as Figure 2 shown.

[0045] The yellow part is the N×N size effective image area involved in the estimation of the current frame, and the red dot is the first effective pixel (upper left corner pixel) involved in the calculation of the current frame. Assuming that the previous frame may move up to n2 pixels in each direction relative to the current frame, then the pixel position corresponding to the upper left corner pixel of the current frame is shown in the green area in the figure. The size of this area is (2n2+1)×(2n2+1), which corresponds to the (2n2+1)×(2n2+1) possibilities of moving n2 pixels to the left horizontally, n2 pixels upward vertically, n2 pixels to the right horizontally, and n2 pixels downward vertically.

[0046] In order to ensure that each pixel from the upper left corner to the lower right corner of the current image has a corresponding (2n2+1)×(2n2+1) area of ​​the previous frame to match it, it will be expanded according to the maximum set of the yellow dotted box, that is, expanded to a size of (N+2n2)×(N+2n2).

[0047] It should be noted that, for the sake of convenience in implementation, n2 takes the maximum value of possible deviations in two directions, and it is most convenient to expand it according to a square.

[0048] According to engineering experience, based on the compromise between algorithm operation efficiency and background estimation effect, N is set to 312, and according to the pixel resolution and motion law of the actual system, n2 is set to 5. Because the actual resolution is 512*512, the larger the value of N, the more accurate the estimation of background motion, and the longer the algorithm takes to run. Specifically, the algorithm time affected by the value of N is N*N clock cycles, and the clock cycle is 10ns as mentioned at the beginning.

[0049] S2: Traverse the inter-frame motion of the image background by translating up and down, left and right by a maximum of n2 pixels.

[0050] The specific method is to construct a shift array of length 2n2×(N+2n2)+2n2+1. Because FPGA calculations are parallel, data is taken one point at a time according to the clock cycle. If you want the algorithm to run with the highest efficiency, you must start calculating when you get the first (2n2+1)×(2n2+1) area. Figure 3 As shown, the shortest time required to obtain the (2n2+1)×(2n2+1) square array in the upper left corner to participate in the operation is to sample (2n2×(N+2n2)+2n2+1) cycles, as shown in the yellow+brown area in the figure.

[0051] In FIFO mode, the data of the previous frame of image is pushed into the array from the first address in sequence from the end of the array until 2n2×(N+2n2)+2n2+1 clocks have passed, and the first (2n2+1)×(2n2+1) data involved in the calculation is pushed to the upper left corner of the shift bit array as a whole (understand the way of storing the two-dimensional array in one dimension), as shown in the attached figure. Figure 2 shown.

[0052] The previous frame image (theoretically has been expanded to (N+2n2)×(N+2n2) pixels, with the first address being the point in the upper left corner, is pushed into the shift array in FIFO mode in the order of rows first and columns later.

[0053] At the same time, (2n2+1)×(2n2+1) accumulators are initialized.

[0054] S3: When the next clock arrives, the first valid pixel of the current frame is compared with the (2n2+1)×(2n2+1) data in the upper left corner of the shift array to obtain the absolute value of the difference. Assuming that the pixel value of the current frame is g(x1,y1), compared with the nth data P n (n>0 and n≤(2n2+1)×(2n2+1)) The absolute value of the difference is:

[0055]

[0056] Put them into the corresponding (2n2+1)×(2n2+1) accumulators to accumulate the difference values ​​and get d(n):

[0057] d(n)=d(x1,y1,n)+d(n)

[0058] in,

[0059] The original image is 512*512 (M*M), and the image involved in background estimation is centered on N*N, so the upper left corner of the N*N image is ((MN) / 2, (MN) / 2), and the lower right corner is ((MN) / 2+N-1, (MN) / 2+N-1).

[0060] S4: When the next clock arrives, the 2n2×(N+2n2)+2n2+2th data (next pixel) starting from the first address of the previous frame image is pushed into the end of the shift array according to the FIFO mode, as shown in the attached figure. Figure 4 As shown, S3 is repeated until the above operation is completed for all N×N valid pixels in the current frame.

[0061] S5: Compare the values ​​of the (2n2+1)×(2n2+1) accumulators in pairs, find the index number n where the minimum value is located, which is the index number corresponding to the offset of the background of the previous frame image relative to the current frame, and calculate it back to the offset (x2, y2), where x2 is the row offset and y2 is the column offset, such as Figure 5 As shown, taking n=5 as an example, the red number is the index number, and the corresponding black number below is the offset. The method is as follows:

[0062]

[0063] Square brackets indicate rounding.

[0064] In the (2n2+1)×(2n2+1) two-dimensional array, the position of the element in the x2th row and y2th column is At this time, the value ranges of x2 and y2 are both [0, 2n2]. In order to more intuitively represent the image offset direction and offset, n2 is subtracted from each of them so that the value ranges of x2 and y2 become [-n2, +n2].

[0065] S6: Compensate the previous frame image according to the offset (x2, y2) inversely calculated in S5.

[0066] The specific method is: find the pixel with the subscript overall offset (x2, y2) from the previous frame image, that is, the pixel with the subscript (n1+n2+1-x2, n1+n2+1-y2) as the starting pixel point (512-2n1-2n2-2)×(512-2n1-2n2-2) pixel area (in order to protect the edge pixels, the maximum translation will not exceed the image size of 512*512) and the center (512-2n1-2n2-2)×(512-2n1-2n2-2) pixel area image of the current frame for difference, and get the differenced image D:

[0067]

[0068] Accumulate the difference value of each pixel into the new accumulator A:

[0069] A=A+D(x1,y1)

[0070] Among them, x1,y1∈[n1+n2+1,512-n1-n2-2]. The pixels in this range are offset by n1 and n2 pixels, and the maximum value range is [1,510], which will definitely not exceed the image value range of [0,511].

[0071] After (512-2n1-2n2-2)×(512-2n1-2n2-2) clocks, the calculation and accumulation operation of the differential image are completed, the differential image is cached in the on-chip RAM, and the cached differential image and the accumulated sum are sent to the DSP in the next clock;

[0072] S7: DSP calculates the segmentation threshold T based on the accumulated sum A:

[0073]

[0074] Among them, Q is the number of pixels participating in the accumulator, which is (512-2n1-2n2-2)×(512-2n1-2n2-2) here, and α is the coefficient, which is selected according to engineering experience, and is selected as 2 here.

[0075] The received differential image is segmented using the segmentation threshold T and the target is extracted based on the information between the target frames. For how to extract, please refer to the article "Detection and Tracking Algorithm for Infrared Small Targets on Missile-borne Platforms" published by Li Dawei et al.

[0076] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A method for extracting moving targets in complex backgrounds based on FPGA, characterized by: include (1) Using an infrared imaging detection system to collect video streams containing complex background images in real time as images to be processed; (2) The optical axis offset calculated by the infrared imaging detection system is used as prior information to compensate for the movement of the complex background between the image frames to be processed; (3) Traversing the inter-frame motion of the image to be processed by pixel translation, minimizing the difference between the inter-frame images and obtaining the inter-frame offset of the image to be processed, and compensating for the offset of the previous frame image; (4) The current image is differentiated from the compensated previous image frame, the differential image is binarized and segmented, and the target is extracted based on the information between the image frames.

2. The method for extracting moving targets under complex background based on FPGA according to claim 1, characterized in that: The optical axis offset is in four directions: up, down, left, and right, and the range of each direction is between 0 and n1 pixels.

3. The method for extracting moving targets under complex background based on FPGA according to claim 2, characterized in that: The n1 is set to 20.

4. The method for extracting moving targets under complex background based on FPGA according to claim 2, characterized in that: The compensation for the movement of the complex background between the image frames to be processed is specifically as follows: the previous frame image P and the current image g are cached in the on-chip RAM, the image actually involved in the complex background motion estimation is the N×N pixels at the center of the current image g, and the previous frame image P is also based on the N×N pixels at the center; The N×N pixel area of ​​the previous frame image P is offset according to the optical axis offset (x, y), and its size is expanded by n2 pixels in the upper, lower, left and right directions to obtain an area of ​​(N+2n2)×(N+2n2) size as the compensation result.

5. The method for extracting moving targets under complex background based on FPGA according to claim 4 is characterized in that: The n2 is 5, and N is 312.

6. The method for extracting moving targets in complex backgrounds based on FPGA according to claim 4, characterized in that: The inter-frame motion of the image to be processed is traversed by pixel translation, and the translation mode is n2 pixels up, down, left, and right.

7. The method for extracting moving targets in complex backgrounds based on FPGA according to claim 6, characterized in that: The method of minimizing the difference between inter-frame images and obtaining the inter-frame offset of the image to be processed is specifically as follows: constructing a shift array of length 2n2×(N+2n2)+2n2+1, and pushing the data of the previous frame image P into the array from the first address in sequence from the end of the array in FIFO mode, until after 2n2×(N+2n2)+2n2+1 clocks, the first (2n2+1)×(2n2+1) data participating in the calculation is pushed as a whole to the upper left corner of the shift array, and the (2n2+1)×(2n2+1) accumulators are initialized at the same time; when the next clock arrives, the first valid pixel of the current image g is compared with the (2n2+1)×(2n2+1) data in the upper left corner of the shift array at the same time, and the absolute value of the difference is obtained. Assuming that the pixel value of the current image g is g(x1,y1), which is the same as the nth data P n The absolute value of the difference is d(x1,y1,n), which is put into the corresponding (2n2+1)×(2n2+1) accumulators for differential value accumulation to obtain d(n). d(n)=d(x1,y1,n)+d(n) Where n>0 and n≤(2n2+1)×(2n2+1)), M*M is the size of the image to be processed; When the next clock arrives, the next pixel of the previous frame image P is pushed into the end of the shift array in FIFO mode, and d(n) is calculated again, and the cycle repeats until the N×N valid pixels of the current image g are all operated. Compare the values ​​of the (2n2+1)×(2n2+1) accumulators in pairs and find the index number n with the minimum value, which is the index number corresponding to the offset of the previous frame image P relative to the current image g. The offset (x2, y2) is obtained by reverse calculation through the index number, where x2 is the row offset and y2 is the column offset.

8. The method for extracting moving targets under complex background based on FPGA according to claim 7, characterized in that: The offset (x2, y2) is obtained by inverse calculation through the index number, specifically:

9. The method for extracting moving targets in complex backgrounds based on FPGA according to claim 8, characterized in that: The difference between the current image and the compensated previous frame image is specifically as follows: Find the pixel with the subscript overall offset (x2, y2) from the previous frame image P, and use it as the starting pixel point (M-2n1-2n2-2)×(M-2n1-2n2-2) pixel area and the center (M-2n1-2n2-2)×(M-2n1-2n2-2) pixel area image of the current image g for difference, and get the differenced image D: Accumulate the difference value of each pixel into accumulator A: A=A+D(x1,y1) Among them, x1,y1∈[n1+n2+1,M-n1-n2-2]; The calculation and accumulation operations of the differential image are completed after (M-2n1-2n2-2)×(M-2n1-2n2-2) clocks, and the differential image is cached in the on-chip RAM.

10. The method for extracting moving targets in complex backgrounds based on FPGA according to claim 9, characterized in that: The binarization and segmentation of the differential image is specifically performed by calculating the segmentation threshold T according to the cumulative sum A. Among them, Q is the number of pixels participating in the accumulator, which is (M-2n1-2n2-2)×(M-2n1-2n2-2), and α is the coefficient.

Images