FPGA (Field Programmable Gate Array) hardware implementation architecture of motion estimation algorithm

Abstract

The invention discloses an FPGA (Field Programmable Gate Array) hardware implementation architecture of a motion estimation algorithm, which is mainly applied to the field of video image stabilization, combines a sub-pixel precision estimation thought, and reduces the window size of Fourier transform by reasonably multiplexing a Fourier transform module, optimizing a hardware structure and performing down-sampling processing before phase correlation calculation. And after calculation, fitting the 3 * 3 neighborhood of the peak value to obtain a sub-pixel precision offset, and then amplifying the down-sampling multiple to obtain a final offset result. Compared with a basic implementation architecture, the improved architecture provided by the invention only uses two configurable Fourier transform modules, required hardware resources are greatly reduced, LUT is reduced by 45%, FF is reduced by 46%, and the detection range is expanded by 64 times.

Description

Technical Field

[0001] The invention relates to the technical field of video stabilization and proposes an FPGA hardware implementation architecture for a motion estimation algorithm for video stabilization. Background Art

[0002] Video stabilization is to process the original video sequence captured by the video device to remove the jitter and make the video picture more stable.

[0003] The current mainstream video stabilization technologies can be roughly divided into three categories: mechanical stabilization, optical stabilization and electronic stabilization. Mechanical stabilization mainly uses a motor to drive the camera for reverse compensation. Optical stabilization mainly uses the refraction of prisms and lenses to compensate for the image projected onto the sensor. Electronic stabilization is based on an embedded CPU, DSP or FPGA, and uses digital image processing technology to achieve motion estimation and motion compensation for video sequences.

[0004] The design of the present invention belongs to electronic image stabilization technology. Figure 1 The structure block diagram of a typical electronic video stabilization system is shown. The core of video stabilization is motion estimation, which aims to calculate the motion vector between image sequences and provide data support for subsequent motion filtering and motion compensation. Common motion estimation methods include block matching, bit plane matching, grayscale projection, feature point matching, phase correlation, etc. The present invention realizes motion estimation based on the phase correlation method and combines sub-pixel fitting to complete the optimization design of the hardware structure.

[0005] The basis of the phase correlation method is the two-dimensional Fourier transform. Let f 1 (x,y) is the reference image, f 2 (x, y) is the detection image. When they only have an offset relationship, that is: f 2 (x,y)=f 1 (xx s ,yy s ) (1)

[0006] In formula (1), (x s ,y s ) is the offset. According to the offset characteristics of Fourier transform, their corresponding Fourier transforms have the following relationship:

[0007] In formula (2), F 1 (u,v),F 2 (u,v) are f 1 With f 2 The two-dimensional discrete Fourier transform function of , then the normalized cross power spectrum can be expressed as formula (3):

[0008] In formula (3) F 2 The complex conjugate of (u,v). Taking the inverse Fourier transform of equation (3), we get the phase correlation function δ(x+x s ,y+y s ), which is a two-dimensional pulse function, and the offset between the two images can be obtained according to its peak position (x s ,y s ). In hardware implementation, there is no need to normalize the cross power spectrum. Obtaining the inverse Fourier transform can reduce the number of calculations and improve efficiency.

[0009] According to the properties of Fourier transform, the two-dimensional Fourier transform can be split into two one-dimensional Fourier transforms, first calculating the row Fourier transform and then the column Fourier transform, or first calculating the column Fourier transform and then the row Fourier transform. The two-dimensional inverse Fourier transform also has this property. Summary of the invention

[0010] The present invention proposes an FPGA hardware implementation structure of a motion estimation algorithm, which can perform motion estimation on an image sequence with a resolution of 1024×1024. The phase correlation method is used as the basis for motion estimation. The method is based on two-dimensional Fourier transform and can accurately calculate the horizontal and vertical offsets between image sequences.

[0011] The present invention draws on the idea of ​​sub-pixel precision estimation, performs downsampling before phase correlation calculation, reduces the window size of Fourier transform, and fits the 3×3 field of the peak after calculation to obtain the sub-pixel precision offset, and then amplifies the downsampling multiple to obtain the final offset result, thereby expanding the detection range. In terms of resource overhead, the required hardware resources are significantly reduced by optimizing the data flow and processing order and rationally reusing some Fourier transform modules.

[0012] Figure 3 The hardware implementation architecture proposed in the present invention includes image cropping, mean downsampling, phase correlation calculation, data splicing, peak location and fitting. The image cropping module crops the image to be processed with a resolution of 1024×1024 from the original image; the mean downsampling module completes 1 / 8 mean downsampling to obtain an image with a resolution of 128×128 to reduce the window size of the Fourier transform, and then uses two Fourier transform modules to complete the calculation of the phase correlation method. The data splicing and peak location modules can reduce the storage occupancy of the phase correlation function and obtain the peak point information. Finally, the sub-pixel precision offset is calculated by the fitting module, and the final offset result is obtained by amplifying 8 times and rounding. The fitting method adopted by the fitting module in the present invention is Gaussian curve fitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a block diagram of the structure of a typical electronic video stabilization system;

[0014] Figure 2 The basic hardware implementation architecture diagram for the motion estimation algorithm;

[0015] Figure 3 An improved hardware implementation architecture diagram for the motion estimation algorithm;

[0016] Figure 4 Compare the resource consumption of the base architecture and the improved architecture. DETAILED DESCRIPTION

[0017] A basic hardware implementation architecture, such as Figure 2 As shown. The detection range is set to the center of the image, with a resolution of 128×128. Figure 2 The hardware architecture in can adapt to a larger detection range, and the corresponding required hardware resources will also increase.

[0018] The basic hardware architecture mainly includes image cropping, phase correlation calculation and peak location. Figure 2 The working process of the hardware structure shown.

[0019] Step 1: The pixel stream data input of the detection frame is first processed by the image cropping module, and a 128×128 image area is cropped in the center for subsequent phase correlation calculation. The phase correlation calculation part is mainly composed of the Fourier transform module, RAM storage space (caching necessary intermediate data) and conjugate point multiplication module. For images with a resolution of 128×128, each row of 128 data is a group of row FFT calculations, and the results are stored in RAM1 or RAM2 (if the result of the previous frame after row FFT calculation is in RAM1, the result of the current detection frame is stored in RAM2; otherwise, it is stored in RAM1, and the switching is cyclic), and it is executed a total of 128 times;

[0020] Step 2: Read the row FFT results of the reference frame and the detection frame from RAM1 and RAM2 by column at the same time. Each column of 128 data is a group of column FFT calculations to obtain the FFTs described in formula (2). 1 (u,v),F 2 (u,v), then calculate F 1 (u,v) and F 2 The conjugate dot product of (u,v) gives the cross power spectrum described in equation (3): Every 128 conjugate point multiplication results are a set of calculation column IFFT, which is executed 128 times in total and the results are stored in RAM3;

[0021] Step 3: Read out the column IFFT results from RAM3 by row, calculate IFFT for each row of 128 data as a group, and execute it 128 times in total. Finally, the two-dimensional inverse Fourier transform result of the cross-power spectrum is obtained, which is the phase correlation function.

[0022] Step 4: Complete the peak location. Use a comparator to find the peak (real part maximum) from the phase correlation function. The horizontal and vertical offsets of the image can be obtained based on the position coordinates corresponding to the peak.

[0023] Each time a new frame of image is input, the aforementioned steps 1-4 are executed.

[0024] Figure 2 The entire hardware architecture shown uses a total of 5 Fourier transform modules (the calculation structures of the forward transform and the inverse transform are basically the same. As for the results, taking the conjugate of the inverse transform result and multiplying it by the number of transformation points is equal to the forward transform result, so the Fourier forward transform module and the Fourier inverse transform module can be uniformly regarded as Fourier transform modules).

[0025] The motion detection range of the basic implementation architecture is 128×128, which is too small for common video resolutions. If the foreground target in the video screen occupies a large number of detection windows, the motion of the foreground target may be mistaken for the jitter of the video screen, resulting in the offset calculation result being far from the actual situation. Increasing the detection range can reduce the probability of misjudgment, but it will greatly increase the consumption of hardware resources.

[0026] In view of this situation, the present invention improves the aforementioned basic hardware implementation architecture: drawing on the sub-pixel accuracy estimation idea of ​​the phase correlation method in the article “Using an improved phase correlation algorithm to achieve image sub-pixel matching”, downsampling is performed before phase correlation calculation, and after completing the phase correlation calculation, the 3×3 field of the peak is fitted to obtain a more accurate offset result, so as to obtain a larger detection range; in addition, according to the spatiotemporal characteristics of the data stream, some Fourier transform modules are reused to reduce the required hardware resources.

[0027] The improved hardware implementation architecture is as follows Figure 3 As shown, only two configurable Fourier transform modules are used, and the data flow direction is changed compared to the basic architecture. The processing order of the data flow is identified by serial numbers 1-11. Since the results of the Fourier forward and inverse transforms are similar, the Fourier transform module can be designed to be dynamically configurable to achieve forward and inverse transforms, and is collectively referred to as a configurable Fourier transform module.

[0028] The improved architecture proposed in the present invention uses only two configurable Fourier transform modules, numbered ① and ② respectively. Figure 3 It is reflected in ① completing row FFT and column FFT, ② completing column FFT, column IFFT and row IFFT, Figure 3 The two muxes in the figure are modules that implement the multi-way selection function. The following describes in detail the working process of the improved architecture.

[0029] Step 1: The detection frame is processed by the image cropping module to obtain a 1024×1024 resolution image, and then 1 / 8 mean downsampling is performed to obtain a 128×128 resolution image. The downsampling result performs 128 128-point row FFTs, and the transformation results are stored in RAM1 or RAM2 (if the result of the row FFT calculation of the previous frame is in RAM1, the result of the current detection frame is stored in RAM2; otherwise, it is stored in RAM1, and the switching is repeated in this way).

[0030] Step 2. Read the row FFT results of the reference frame and the detection frame from RAM1 and RAM2 by column at the same time. Read only one column and send it to ① and ② (① and ② are configured as forward transform) to calculate 128-point column FFT, then calculate the conjugate dot product, send the dot product result to ② (configured as inverse transform) to calculate the column IFFT, and then store the column IFFT result into RAM3. Repeat step 2 128 times.

[0031] Step 3, read out the column IFFT results from RAM3 by row, and send each row of 128 data as a group to ② (configured as inverse transform) to calculate the row IFFT. Perform it 128 times in total, and finally obtain the two-dimensional inverse Fourier transform result of the cross-power spectrum, that is, the phase correlation function.

[0032] Step 4. Since a 1 / 8 downsampling process is performed before the phase correlation method is calculated, the offset corresponding to the peak of the phase correlation function should be the result of rounding off the actual offset after reducing it by 8 times. In order to obtain a result closer to the actual offset, it is necessary to fit the peak field data, find a more accurate peak position from the discrete data, obtain an offset with sub-pixel accuracy, and then amplify it by 8 times and round it off to get the final offset result.

[0033] Each time a new frame of image is read in, the aforementioned steps 1-4 are executed.

[0034] There are various fitting methods, such as interpolation of the 3×3 region of the peak value, or curve or surface fitting. The present invention provides an example of curve fitting using a Gaussian function, and the prototype of the Gaussian function is:

[0035] According to the properties of the Gaussian function, when x=n, the function reaches its maximum value; the integral of a two-dimensional Gaussian function in any dimension is a one-dimensional Gaussian function, and the center of this one-dimensional Gaussian function is the projection of the corresponding two-dimensional Gaussian function in this dimension. Therefore, it is only necessary to calculate the symmetric center point n in both the horizontal and vertical directions to obtain the offset with sub-pixel accuracy (n x,n y ). The 3×3 area around the peak point is defined as matrix P, where A x,y is the peak point, and its position in the phase correlation function is (x, y). The result is calculated through the following two steps.

[0036] Step 1: Calculate the numerical integrals for the horizontal and vertical directions respectively, and obtain two sets of ordered number pairs (x-1, a x-1 ), (x,a x ), (x+1,a x-1 ) and (y-1,b y-1 ), (y,b y ), (y+1,b y+1 )in

[0037] Step 2: Perform Gaussian fitting on the two ordered number pairs to find the center coordinates of the one-dimensional Gaussian function, which is the image offset with sub-pixel accuracy. x-1 ),(x,a x ),(x+1,a x+1 ) is substituted into the one-dimensional Gaussian function (4), and the following system of equations is obtained:

[0038] Solve for parameter n, which is the lateral offset n x (sub-pixel accuracy).

[0039] A similar method can be used to calculate the longitudinal offset n. y (sub-pixel accuracy).

[0040] Formulas (7), (8), (9), and (10) contain complex operations such as division and logarithm. In hardware implementation, mathematical calculation IP can be called and combined with state machine implementation.

[0041] The design and synthesis of the basic architecture and the improved architecture are completed. The data format is set to single-precision floating point numbers. The FPGA chip model is selected from the Xilinx manufacturer's xcku040-ffva1156-2. Finally, the resource consumption of the two architectures is compared as follows: Figure 4 As shown, compared with the basic architecture, the improved hardware implementation architecture has 45% fewer LUTs and 46% fewer FFs.

[0042] In addition, the improved architecture has a larger image detection range. The detection range of the improved solution is 1024×1024 centered on the video screen, while the basic architecture is only 128×128. The detection screen range of the improved architecture is 64 times that of the basic architecture.

Claims

1. An FPGA hardware implementation architecture for a motion estimation algorithm for video stabilization, characterized in that The downsampling and sub-pixel fitting techniques are adopted and the configurable Fourier transform module is reused. The architecture expands the detection range through downsampling processing and sub-pixel fitting, and reduces the occupation of hardware resources by reusing the configurable Fourier transform module.

2. The FPGA hardware implementation architecture according to claim 1, characterized in that: The downsampling module performs 1 / 8 mean downsampling processing on the input 1024*1024 resolution image to obtain a 128*128 resolution image.

3. The FPGA hardware implementation architecture according to claim 1, characterized in that: The sub-pixel fitting module uses a Gaussian function to perform curve fitting on the 3x3 area data around the peak point to obtain the image offset with sub-pixel accuracy.

4. The FPGA hardware implementation architecture according to claim 1, characterized in that: The configurable Fourier transform module can realize the switching between forward transform and inverse transform by configuring the port, and only two configurable Fourier transform modules are used in the whole architecture to complete the calculation of the phase correlation method.

Images