Parallel assembly line type aerial image real-time matching system based on FPGA (Field Programmable Gate Array)
By optimizing the inter-frame matching of aerial images through the FPGA parallel pipeline architecture, the problem of unstable positioning of UAV remote sensing technology in GPS-denied environments is solved, efficient and stable image matching and positioning are achieved, and the mission success rate and endurance of the UAV remote sensing system are improved.
Patent Information
- Application Number
- CN202510862311.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-25
- Publication Date
- 2025-10-17
AI Technical Summary
Existing UAV remote sensing technology has difficulty achieving efficient and stable inter-frame matching of aerial images in GPS-denied environments, resulting in insufficient positioning information, affecting mission success rate and system stability.
The system adopts an FPGA-based parallel pipeline architecture, and improves the parallelism and data throughput of aerial image frame matching through grayscale, filtering, gradient value calculation, image structure direction calculation, feature caching, matching point search and transformation matrix calculation modules. It also uses the low power consumption and high stability of FPGA to accelerate the image matching process.
The efficiency of inter-frame matching of aerial images is improved, the operation time of the UAV remote sensing system is extended, and the stability of the system is improved.
Smart Images
Figure CN120807977A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of unmanned aerial vehicle aerial image processing, and particularly relates to a parallel pipeline aerial image real-time matching system based on FPGA. BACKGROUND
[0002] Unmanned aerial vehicle remote sensing technology uses an unmanned aerial vehicle as a load platform, uses an imaging load carried on the unmanned aerial vehicle platform to obtain aerial images of a target object and a ground environment, and through processing and analysis of the aerial images, spatial remote sensing information such as geometric and physical characteristics of the target object and the ground environment is obtained, and a remote sensing measurement task is efficiently completed. With the increasing demand for spatial remote sensing information in human society, the application field of unmanned aerial vehicle remote sensing technology is gradually expanding. At present, unmanned aerial vehicle remote sensing technology has been widely applied in the fields of forestry monitoring, coastline monitoring, smart agriculture, reservoir security inspection, pipeline line inspection and the like.
[0003] In the process of performing a remote sensing measurement task, the unmanned aerial vehicle platform needs to obtain positioning information in real time to determine its own position. Reliable positioning information can not only ensure that the unmanned aerial vehicle platform successfully carries out the measurement task according to the set task route, but also can guarantee the safety of the unmanned aerial vehicle platform and the imaging load during the task execution process. In the existing unmanned aerial vehicle remote sensing technology, the global positioning system (GPS) is usually used to obtain the positioning information of the unmanned aerial vehicle. The GPS positioning technology receives GPS satellite signals, and then outputs the positioning information of the unmanned aerial vehicle after calculation. This technology is relatively mature, has high system integration, and is widely used in various industries. However, the accuracy and stability of the GPS positioning technology are restricted by the quality of the GPS satellite signals. When the unmanned aerial vehicle performs a remote sensing measurement task in an environment such as a city, a forest, or near a high-voltage power line, the GPS satellite signals are easily blocked or interfered, resulting in GPS denial, so that the unmanned aerial vehicle cannot obtain effective positioning information and the task fails.
[0004] In order to perform the unmanned aerial vehicle remote sensing measurement task in the GPS denial environment, the unmanned aerial vehicle remote sensing system usually adopts the unmanned aerial vehicle positioning technology based on vision to solve the positioning information of the unmanned aerial vehicle. The unmanned aerial vehicle positioning technology based on vision acquires the aerial photograph of the ground and the surrounding environment by the aerial camera carried on the unmanned aerial vehicle, and then acquires the matching points between the adjacent frame aerial photographs by the inter-frame matching processing of the adjacent frame aerial photographs by means of the image matching method, and then solves the pose parameters of the unmanned aerial vehicle platform to complete the positioning of the unmanned aerial vehicle platform. The timeliness of the unmanned aerial vehicle positioning technology based on vision is mainly limited by the inter-frame matching process of the aerial photograph, therefore, the inter-frame matching process of the aerial photograph is accelerated by the graphics processing unit (GPU), but the GPU has large running power consumption and poor stability, which easily causes the decrease of the system endurance and reliability. SUMMARY
[0005] Therefore, the application discloses a parallel pipeline type aerial photograph real-time matching system based on FPGA, which adopts the FPGA with low power consumption and high stability to accelerate the inter-frame matching process of the aerial photograph, improves the parallel degree and data throughput of the inter-frame matching process based on the parallel processing capability of the FPGA, thereby improving the matching efficiency of the inter-frame matching process of the aerial photograph and realizing the real-time inter-frame matching of the aerial photograph of the unmanned aerial vehicle with high efficiency.
[0006] In order to achieve the above purpose, the technical scheme adopted by the application is as follows:
[0007] A parallel pipeline type aerial photograph real-time matching system based on FPGA, comprising a grayscale module, a Gaussian filter module, a gradient value calculation module, an image structure direction calculation module, a feature cache module, a matching point search module and a transformation matrix calculation module realized based on FPGA.
[0008] The grayscale module is used for reading the RGB brightness values in the aerial photograph pixel by pixel, and parallel pipeline processing the R brightness value, the G brightness value and the B brightness value in the RGB brightness value, calculating the grayscale value of the current pixel point in the single precision floating point number format and transmitting the grayscale value to the Gaussian filter module.
[0009] The Gaussian filter module is used for calculating the grayscale value of the pixel point after filtering in a parallel pipeline manner, and then transmitting the grayscale value to the gradient value calculation module.
[0010] The gradient value calculation module is used for calculating the horizontal gradient value Gx of the pixel point and the vertical gradient value Gy of the pixel point, and then transmitting the horizontal gradient value Gx and the vertical gradient value Gy to the image structure direction calculation module. x y The image structure direction calculation module is used for calculating the image structure direction of the pixel point according to the horizontal gradient value Gx and the vertical gradient value Gy of the pixel point received.
[0011] The image structure direction calculation module is used for calculating the image structure direction of the pixel point according to the horizontal gradient value Gx and the vertical gradient value Gy of the pixel point received.x with the vertical gradient value G y , the following calculation is performed:
[0012] t y = 2G x G y , tan2a = t y / t x ;
[0013] Taking a as the image structure direction of the current pixel point, the image structure direction matrix corresponding to the current aerial image is obtained through pixel-by-pixel calculation, and then is transmitted to the feature cache module;
[0014] The feature cache module uses two frame buffers to ping-pong cache the image structure direction matrix of the current frame and the previous frame of the aerial image;
[0015] The matching point search module randomly selects N rectangular regions as feature templates in the image structure direction matrix of the previous frame of the aerial image, and performs a pipeline element-by-element calculation of the matching similarity in the image structure direction matrix of the current frame of the aerial image to obtain N matching points corresponding to the N feature templates; N matching point pairs are constructed according to the N feature templates and the N matching points, and are transmitted to the transformation matrix calculation module;
[0016] The transformation matrix calculation module solves the affine transformation matrix composed of N matching point pairs according to the least square method, calculates the transformation matrix between the previous frame and the current frame of the aerial image, and realizes the real-time matching of the aerial image based on the FPGA parallel pipeline.
[0017] Further, the grayscale module includes a 3x1 array floating point conversion submodule, a 3x1 array multiplier, and a 3-input addition tree.
[0018] The 3x1 array floating point conversion submodule converts the R brightness value, the G brightness value, and the B brightness value from fixed-point number format to single-precision floating point format in parallel through three floating point conversion units;
[0019] The 3x1 array multiplier calculates the product of the R brightness value, the G brightness value, and the B brightness value and the corresponding weighting coefficient in parallel through three floating point multipliers to obtain the weighting results of the RGB three channels respectively; the 3-input addition tree calculates the sum of the weighting results as the grayscale value of the current pixel point in single-precision floating point format in parallel and pipeline through two adders and one register, and transmits it to the Gaussian filter module.
[0020] Further, the Gaussian filter module includes a row-column conversion submodule, a 3x3 array multiplier, and a 9-input addition tree.
[0021] The row-column conversion submodule caches the gray values of each pixel point and judges pixel by pixel:
[0022] If the current pixel point is the last two rows or the last two columns of pixel points in the current frame of aerial image, no operation is performed, and the next pixel point is continuously judged;
[0023] Otherwise, the current pixel point is taken as the first row and the first column element, the pixel points are selected to the right and down, a three-row and three-column pixel point gray value matrix is constructed according to the gray values of the pixel points, and is input into a 3*3 array multiplier;
[0024] The 3*3 array multiplier calculates the product of each element in the three-row and three-column pixel point gray value matrix and the corresponding filter coefficient through nine floating-point multipliers, respectively obtains the Gaussian filtering result of each element in the three-row and three-column pixel point gray value matrix, the filter coefficients of the nine floating-point multipliers conform to the Gaussian filter coefficient, and the 9-input adder tree calculates the sum of the Gaussian filtering results corresponding to the three-row and three-column pixel point gray value matrix through eight adders and three registers, takes the sum as the filtered gray value of the current pixel point, and transmits it to the gradient value calculation module.
[0025] Further, the gradient value calculation module comprises a row-column conversion submodule, a 3*2 array bit operation multiplier, a 2*3 array bit operation multiplier and two 6-input adder trees;
[0026] The row-column conversion submodule caches the filtered gray values of each pixel point and judges pixel by pixel:
[0027] If the current pixel point is the last four rows or the last four columns of pixel points in the current frame of aerial image, no operation is performed, and the next pixel point is continuously judged;
[0028] Otherwise, the current pixel point is taken as the first row and the first column element, the pixel points are selected to the right and down, a three-row and three-column pixel point filtered gray value matrix is constructed according to the filtered gray values of the pixel points;
[0029] The 3*2 array bit operation multiplier calculates the product of the filtered gray values of the first column and the third column elements in the three-row and three-column pixel point filtered gray value matrix and the Sobel operator in parallel by performing addition operation on the sign bit and the exponent in the single-precision floating-point number format, and inputs into the first 6-input adder tree;
[0030] The 2*3 array type bit operation multiplier calculates the product of the filtered gray value of the first row and the third row elements in the three rows and three columns pixel point filtered gray value matrix and the Sobel operator through the addition operation of the sign bit and the exponent in the single precision floating point number format, and inputs into the second 6-input addition tree;
[0031] The two 6-input addition trees each include five adders and one register, and calculate the sum value of the calculation results of the 3*2 array type bit operation multiplier and the 2*3 array type bit operation multiplier, denoted as the horizontal gradient value G of the current pixel point x and the vertical gradient value G y , and then transmits to the image structure direction calculation module.
[0032] Further, the image structure direction calculation module includes a 3*1 array type multiplier, a subtractor, a bit operation multiplier, a divider, a pipeline coordinate rotation digital calculation unit and a bit operation divider;
[0033] The 3*1 array type multiplier calculates the values of G x G y and through three floating point multipliers in parallel, and then calculates the values of and t y = 2G x G y through the subtractor and the bit operation multiplier in parallel; then, the divider is used to calculate t y / t x as the tangent value tan2α of the image structure direction of the current pixel point; finally, the pipeline coordinate rotation digital calculation unit is used to perform the inverse tangent operation to calculate the value of 2α, and then the bit operation divider is used to perform the subtraction operation on the exponent in the single precision floating point number format, and the image structure direction α of the current pixel point is calculated in a pipeline manner, to obtain the corresponding image structure direction matrix and transmit to the feature cache module.
[0034] Further, the matching point search module includes a first region intercepting sub-module, a second region intercepting sub-module, a subtractor, a pipeline coordinate rotation digital calculation unit, an accumulator and a comparator;
[0035] The first region intercepting sub-module is used to select N rectangular regions as feature templates in the image structure direction matrix of the last frame of the aerial image;
[0036] For each feature template, the second region intercepting sub-module is used to perform element-by-element judgment in the image structure direction matrix of the current frame of the aerial image;
[0037] In the image structure direction matrix of the aerial image of the current frame, the current element is taken as the first row and first column element, if the corresponding element can be selected to form a rectangular region consistent with the size of the feature template to the right and down, then the rectangular region corresponding to the current element is recorded as a feature region, otherwise no operation is performed, and the next element is continuously judged.
[0038] For each feature region, the difference between the feature region and the corresponding element in the feature template is calculated by a subtracter in a pipeline, to obtain a corresponding difference matrix, and the difference matrix is transmitted to a pipeline coordinate rotation digital computer unit.
[0039] The pipeline coordinate rotation digital computer unit calculates the cosine value of each difference in the difference matrix as a cosine similarity to obtain a corresponding cosine similarity matrix, and the cosine similarity matrix is transmitted to an accumulator.
[0040] The accumulator adds all the cosine similarities in the cosine similarity matrix to record the matching similarity of the current element, and transmits it to a comparator.
[0041] The comparator compares the maximum value of the matching similarity of the pipeline search, and takes the element corresponding to the maximum value of the matching similarity as the matching point of the feature template. Then, a matching point pair is formed according to the element in the first row and first column of the feature template and the matching point obtained.
[0042] The above process is repeated to obtain N matching points corresponding to N feature templates and N matching point pairs, and they are transmitted to a transformation matrix calculation module.
[0043] Compared with the prior art, the beneficial effects of the present application are as follows:
[0044] The present application uses the parallel processing capability of FPGA to optimize the aerial image inter-frame matching process by using a parallel pipeline architecture, effectively improving the parallelism and data throughput of the aerial image inter-frame matching process, thereby improving the matching efficiency of inter-frame matching. At the same time, FPGA has the advantages of low power consumption and high stability, which is beneficial to improve the operation time and stability of the unmanned aerial vehicle remote sensing system. BRIEF DESCRIPTION OF DRAWINGS
[0045] Figure 1 is the flowchart of the parallel pipeline aerial image real-time matching based on FPGA in the embodiment of the present application.
[0046] Figure 2 is the parallel pipeline architecture diagram of the grayscale module in the embodiment of the present application.
[0047] Figure 3 is the parallel pipeline architecture diagram of the Gaussian filter module in the embodiment of the present application.
[0048] Figure 4 is the parallel pipeline architecture diagram of the gradient value calculation module in the embodiment of the application.
[0049] Figure 5 is the parallel pipeline architecture diagram of the image structure direction calculation module in the embodiment of the application.
[0050] Figure 6 is the pipeline architecture diagram of the matching point search module in the embodiment of the application. DETAILED DESCRIPTION
[0051] The application will be further described in combination with the specific embodiments and the accompanying drawings.
[0052] The application designs a parallel pipeline aerial image real-time matching system based on FPGA, takes FPGA as a processing platform, and realizes high-efficiency, low-power-consumption, high-stability aerial image real-time interframe matching of an unmanned aerial vehicle by designing a parallel pipeline algorithm hardware architecture. Figure 1 is a flowchart of the parallel pipeline aerial image real-time matching system based on FPGA, mainly including seven processing steps, which are parallel pipeline gray-scale processing, parallel pipeline Gaussian filtering, parallel pipeline gradient value calculation, parallel pipeline image structure direction calculation, feature caching, parallel pipeline matching point search and transformation matrix calculation.
[0053] The gray-scale processing module adopts the parallel pipeline architecture designed in the application Figure 2 to process the RGB brightness values in the aerial image pixel by pixel. Firstly, the 3x1 array floating-point conversion submodule converts the R brightness value, the G brightness value and the B brightness value in the RGB brightness value into floating-point format by using three floating-point conversion units. Then, the 3x1 array multiplier multiplies the converted R brightness value, G brightness value and B brightness value with the weighting coefficient 1, the weighting coefficient 2 and the weighting coefficient 3 respectively by using three floating-point multipliers to obtain the weighted results of the three channels of RGB. Then, the 3-input addition tree calculates the sum of the weighted results of the three channels of RGB in parallel pipeline by using two adders and one register to obtain the gray-scale value in single-precision floating-point format, wherein the adder 1 receives the outputs of the floating-point multiplier 1 and the floating-point multiplier 2; the register receives the output of the floating-point multiplier 3; the adder 2 receives the outputs of the adder 1 and the register and outputs the gray-scale value in single-precision floating-point format; and the register is used to adjust the delay time of data to align the time sequence of data.
[0054] The Gaussian filtering module adopts the parallel pipeline architecture designed in the application Figure 3The parallel pipeline architecture designed in the application is used to process the gray value output by the gray scale module. Firstly, the row-column conversion submodule buffers the gray value output by the gray scale module, and when the gray values of the nine pixel points in the Gaussian filter window are buffered, the values are output in parallel to the 3x3 array multiplier. Then, the 3x3 array multiplier uses nine floating-point multipliers to calculate the products of the gray values of the nine pixel points and the filter coefficients in parallel, and obtains the filter components of the nine pixel points. Then, the nine-input addition tree uses nine adders and three registers to calculate the sum of the filter components of the nine pixel points in parallel, and obtains the filtered gray value, wherein the adder 1 receives the outputs of the floating-point multiplier 3 and the floating-point multiplier 2; the adder 2 receives the outputs of the floating-point multiplier 1 and the floating-point multiplier 6; the adder 3 receives the outputs of the floating-point multiplier 5 and the floating-point multiplier 4; the adder 4 receives the outputs of the floating-point multiplier 9 and the floating-point multiplier 8; the register 1 receives the output of the floating-point multiplier 7; the adder 5 receives the outputs of the adder 1 and the adder 2; the adder 6 receives the outputs of the adder 3 and the adder 4; the adder 7 receives the outputs of the adder 5 and the adder 6; the register 2 receives the output of the register 1, and the register 3 receives the output of the register 2; the adder 8 receives the outputs of the adder 7 and the register 3, and outputs the filtered gray value; and the three registers are used to adjust the delay time of the data to align the time sequence of the data.
[0055] The gradient value calculation module adopts Figure 4 The parallel pipeline architecture designed in the application is used to process the filtered gray value, and two independent parallel threads are used to calculate the horizontal gradient value G x and the vertical gradient value G y . Firstly, the row-column conversion submodule buffers the filtered gray value output by the Gaussian filter module, and when the filtered gray values of the eight pixel points except the center pixel in the row-column conversion submodule are buffered, the values are output in parallel to thread 1 and thread 2. Then, thread 1 first uses six bit operation multipliers in the 3x2 array bit operation multiplier to perform parallel pipeline addition operations on the sign bit and the mantissa of the filtered gray value respectively, obtains six horizontal filter components, and then uses five adders and one register in the six-input addition tree to calculate the sum of the six horizontal filter components in parallel, and obtains G x . Thread 2 first uses six bit operation multipliers in the 2x3 array bit operation multiplier to perform parallel pipeline addition operations on the sign bit and the mantissa of the filtered gray value respectively, obtains six vertical filter components, and then uses five adders and one register in the six-input addition tree to calculate the sum of the six vertical filter components in parallel, and obtains G y .
[0056] Specifically, for the 3x2 array bit operation multiplier, adder 1 receives the outputs of bit operation multiplier 1 and bit operation multiplier 4, adder 2 receives the outputs of bit operation multiplier 2 and bit operation multiplier 5, adder 3 receives the outputs of bit operation multiplier 3 and bit operation multiplier 6, adder 4 receives the outputs of adder 1 and adder 2, register 1 receives the output of adder 3, adder 5 receives the outputs of adder 4 and register 1, and outputs G x ;
[0057] For the 2x3 array bit operation multiplier, adder 6 receives the outputs of bit operation multiplier 7 and bit operation multiplier 10, adder 7 receives the outputs of bit operation multiplier 8 and bit operation multiplier 11, adder 8 receives the outputs of bit operation multiplier 9 and bit operation multiplier 12, adder 9 receives the outputs of adder 7 and adder 8, register 2 receives the output of adder 6, adder 10 receives the outputs of adder 9 and register 2, and outputs G y ;
[0058] The image structure direction calculation module adopts the parallel pipeline architecture designed in Figure 5 to process G x and G y . First, the 3x1 array multiplier uses three floating point multipliers to calculate and G x G y in parallel; second, one subtractor is used to calculate ; meanwhile, one bit operation multiplier is used to calculate t x G y by adding 1 to the exponent of G y = 2G x G y , and the output delay of the bit operation multiplier is adjusted to align the timing of the data; third, one divider is used to calculate t y / t x in a pipeline to obtain the tangent value tan2α of the image structure direction; then, the arctangent value 2α of tan2α is calculated in a pipeline using a pipeline coordinate rotation digital computer; then, one bit operation divider is used to calculate the image structure direction α by subtracting 1 from the exponent of 2α in a pipeline, and the image structure direction matrix of the current frame aerial image is obtained by calculating the image structure direction pixel by pixel.
[0059] Two frame buffers are provided in the feature cache module to perform ping-pong caching on the image structure direction matrices of the current frame and the previous frame aerial image.
[0060] As shown in Figure 6As shown, the region intercepting submodule 1 in the matching point searching module intercepts a rectangular region as a feature template in the image structure direction matrix of the last frame aerial image, and adopts a pipeline architecture to search for the corresponding matching point of the feature template in the image structure direction matrix of the current frame aerial image in a pipeline manner; in the searching process, a pixel point p in the current frame aerial image is first selected, a feature region with the same size as the feature template is demarcated in the image structure direction matrix of the current frame aerial image with p as the top-left corner, the difference between the image structure directions of the corresponding pixel points in the feature region and the feature template is calculated in a pipeline manner through a subtractor, and the cosine similarity of each element in the intercepted rectangular region is calculated through a pipeline coordinate rotation digital computer; then, the sum of the cosine similarities in the rectangular region is calculated in a pipeline manner through an accumulator to obtain the matching similarity of the pixel point p; then, the matching similarity is calculated pixel by pixel, and the maximum value of the matching similarity is searched in a pipeline manner through a comparator, and the pixel point corresponding to the maximum value of the matching similarity is taken as the matching point of the feature template; the matching point searching module intercepts N feature templates, searches for N matching points corresponding to the N feature templates in a pipeline manner through a pipeline architecture, and constructs N matching point pairs according to the N feature templates and the N matching points, and transmits the N matching point pairs to the transformation matrix calculation module.
[0061] The transformation matrix calculation module solves the affine transformation matrix composed of the N matching point pairs according to the least square method, calculates the transformation matrix between the last frame and the current frame aerial image, and realizes the real-time aerial image matching based on the FPGA parallel pipeline.
[0062] The grayscale module converts the RGB luminance values into the single-precision floating-point number format, and also adopts the single-precision floating-point number format in the matching process, so as to improve the calculation accuracy of the matching process; the grayscale, Gaussian filtering, gradient value calculation and image structure direction calculation processes adopt a parallel pipeline architecture, and the parallel processing capability based on the FPGA improves the data throughput and the image matching efficiency of the image matching process.
[0063] The grayscale module adopts a parallel pipeline architecture, and includes the 3x1 array floating-point conversion submodule, the 3x1 array multiplier and the 3-input addition tree. The 3x1 array floating-point conversion submodule completes the parallel conversion of the R luminance value, the G luminance value and the B luminance value through three floating-point conversion units; the 3x1 array multiplier calculates the products of the R luminance value, the G luminance value and the B luminance value and the weighting coefficients in parallel through three floating-point multipliers to obtain the weighted results of the RGB three channels; and the 3-input addition tree calculates the sum of the weighted results in a pipeline manner through two adders to obtain the grayscale value.
[0064] The gradient value calculation module adopts a parallel pipeline architecture, and two parallel threads are arranged inside, thread 1 calculates G x in parallel through the 3*2 array type bit operation multiplier and the 6-input addition tree y ; the 3*2 array type bit operation multiplier and the 2*3 array type bit operation multiplier are both composed of 6 bit operation multipliers, the bit operation multipliers calculate the product of the filtered gray value in the Sobel filter window and the Sobel operator in parallel by performing addition operation on the sign bit and the exponent in the single-precision floating-point number format, and then parallel pipelined addition operation is performed by using 5 adders in the 6-input addition tree, so that G x and G y are calculated in parallel.
[0065] The image structure direction calculation module calculates the image structure direction by using a parallel pipeline architecture, first, a 3*1 array type multiplier composed of 3 floating-point multipliers is used to calculate G x G y and then, the subtractor and the bit operation multiplier are used to calculate and t y = 2G x G y ; then, the divider is used to calculate t y / t x as the tangent value tan2a of the image structure direction; finally, the floating-point coordinate rotation digital computer is used to perform arctangent operation to calculate 2a, and the bit operation divider is used to perform minus 1 operation on the exponent in the single-precision floating-point number format, so that a is calculated in parallel as the image structure direction.
[0066] The feature cache module is provided with two frame buffers, adopts a ping-pong cache strategy, and stores the image structure direction matrix of the aerial image of the current frame and the last frame respectively, so that the image structure direction calculation module and the matching point search module can work in parallel, data conflict is avoided, and the image matching efficiency is improved.
[0067] The matching point search module adopts a pipeline architecture composed of the subtractor, the floating-point coordinate rotation digital computer and the accumulator, calculates the matching similarity between the feature template of the aerial image of the last frame and the aerial image of the current frame in parallel, realizes pipelining of the matching point search process, and thus the data throughput is improved.
[0068] The above merely describes the basic scheme of the embodiment method of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. All the changes falling within the equivalent meaning and range of the claims will be included in the scope of the claims.
Claims
1. A parallel pipelined aerial image real-time matching system based on FPGA, characterized by: It includes FPGA-based grayscale module, Gaussian filter module, gradient value calculation module, image structure direction calculation module, feature cache module, matching point search module and transformation matrix calculation module; The grayscale module is used to read the RGB brightness value in the aerial image pixel by pixel, parallel pipeline processing of the R brightness value, G brightness value and B brightness value in the RGB brightness value, calculate the grayscale value of the current pixel in single-precision floating-point format, and transmit it to the Gaussian filtering module; The Gaussian filter module is used to calculate the grayscale value of the pixel after filtering in parallel and pipeline manner, and then transmit it to the gradient value calculation module; The gradient value calculation module is used to calculate the horizontal gradient value G of the pixel point x and the vertical gradient value G y , and then transmitted to the image structure direction calculation module; The image structure direction calculation module receives the horizontal gradient value G of the pixel point. x and the vertical gradient value G y , perform the following calculations: Taking α as the image structure direction of the current pixel, the image structure direction matrix corresponding to the current aerial image is obtained by pixel-by-pixel calculation and then transmitted to the feature cache module; The feature cache module utilizes two frame buffers to cache the image structure direction matrix of the current frame and the previous frame of the aerial image in a ping-pong manner; The matching point search module randomly selects N rectangular areas in the image structure direction matrix of the previous frame of the aerial image as feature templates, and performs a pipelined element-by-element calculation of matching similarities in the image structure direction matrix of the current frame of the aerial image to obtain N matching points corresponding to the N feature templates; constructs N matching point pairs based on the N feature templates and the N matching points, and transmits them to the transformation matrix calculation module; The transformation matrix calculation module solves the affine transformation matrix formed by the N matching point pairs according to the least squares method, calculates the transformation matrix between the aerial images of the previous frame and the current frame, and realizes parallel pipelined real-time matching of aerial images based on FPGA.
2. The FPGA-based parallel pipeline real-time aerial image matching system according to claim 1, characterized in that: The grayscale module includes a 3×1 array floating point conversion submodule, a 3×1 array multiplier and a 3-input addition tree; The 3×1 array floating-point conversion submodule converts the R brightness value, the G brightness value, and the B brightness value from a fixed-point format to a single-precision floating-point format in parallel through three floating-point conversion units; The 3×1 array multiplier uses three floating-point multipliers to parallelly calculate the products of the R brightness value, the G brightness value, and the B brightness value with the corresponding weighting coefficients to obtain weighted results of the three RGB channels respectively; the three-input addition tree uses two adders and one register to parallelly pipeline calculate the sum of the weighted results as the grayscale value of the current pixel in single-precision floating-point format, and transmits it to the Gaussian filter module.
3. The FPGA-based parallel pipeline real-time aerial image matching system according to claim 1, characterized in that: The Gaussian filter module includes a row-column conversion submodule, a 3×3 array multiplier and a 9-input addition tree; After the row-column conversion submodule caches the grayscale value of each pixel, it makes a judgment pixel by pixel: If the current pixel is in the last two rows or columns of the current frame aerial image, no operation is performed and the next pixel is judged. Otherwise, take the current pixel as the first row and first column element, select pixels to the right and downward, and form a three-row and three-column pixel grayscale value matrix based on the grayscale value of each pixel, and input it into the 3×3 array multiplier; The 3×3 array multiplier calculates the product of each element in the three-row and three-column pixel grayscale value matrix and the corresponding filter coefficient through 9 floating-point multipliers, and obtains the Gaussian filtering results of each element in the three-row and three-column pixel grayscale value matrix respectively; the filter coefficients of the 9 floating-point multipliers conform to the Gaussian filtering coefficients; the 9-input addition tree calculates the sum of the Gaussian filtering results corresponding to the three-row and three-column pixel grayscale value matrix through 8 adders and 3 registers, and uses the sum of the Gaussian filtering results as the grayscale value of the current pixel after filtering and transmits it to the gradient value calculation module.
4. The FPGA-based parallel pipeline real-time aerial image matching system according to claim 1, characterized in that: The gradient value calculation module includes a row-column conversion submodule, a 3×2 array bit operation multiplier, a 2×3 array bit operation multiplier and two 6-input addition trees; The row-column conversion submodule caches the filtered grayscale values of each pixel and then makes a judgment pixel by pixel: If the current pixel is in the last four rows or columns of the current frame aerial image, no operation is performed and the next pixel is judged. Otherwise, take the current pixel as the first row and first column element, select pixels to the right and downward, and form a three-row and three-column pixel filter grayscale value matrix based on the grayscale value of each pixel after filtering; The 3×2 array-type bit operation multiplier performs an addition operation on the sign bit and the exponent in the single-precision floating-point format, and concurrently calculates the product of the filtered grayscale values of the first and third columns of the three-row, three-column pixel point filtered grayscale value matrix and the Sobel operator, and inputs the product into the first 6-input addition tree; The 2×3 array-type bit operation multiplier performs an addition operation on the sign bit and the exponent in the single-precision floating-point format, and concurrently calculates the product of the filtered grayscale values of the first and third row elements in the three-row and three-column pixel point filtered grayscale value matrix and the Sobel operator, and inputs the product into the second 6-input addition tree; The two 6-input addition trees each include 5 adders and 1 register, which respectively calculate the sum of the results of the 3×2 array bit operation multiplier and the 2×3 array bit operation multiplier, and record it as the horizontal gradient value G of the current pixel point. x and the vertical gradient value G y , and then transmitted to the image structure direction calculation module.
5. The FPGA-based parallel pipeline real-time aerial image matching system according to claim 1, characterized in that: The image structure direction calculation module includes a 3×1 array multiplier, a subtractor and a bit operation multiplier, a divider, a pipeline coordinate rotation digital calculation unit and a bit operation divider; The 3×1 array multiplier calculates in parallel through three floating-point multipliers. G x G y and The value is then calculated in parallel through the subtractor and the bit operation multiplier. and t y =2G x G y Then, using the divider, calculate t y / t x As the tangent value tan2α of the image structure direction of the current pixel point; finally, the pipeline coordinate rotation digital calculation unit is used to perform an inverse tangent operation to calculate the value of 2α, and then the bit operation divider is used to subtract the intermediate order code in the single-precision floating-point format, and α is calculated in a pipeline manner as the image structure direction of the current pixel point, and the corresponding image structure direction matrix is obtained and transmitted to the feature cache module.
6. The FPGA-based parallel pipeline real-time aerial image matching system according to claim 1, characterized in that: The matching point search module includes a first region interception submodule, a second region interception submodule, a subtractor, a pipeline coordinate rotation digital calculation unit, an accumulator and a comparator; The first region interception submodule is used to arbitrarily select N rectangular regions in the image structure direction matrix of the previous frame of the aerial image as feature templates; For each feature template, the second region interception submodule is used to determine element by element in the image structure direction matrix of the aerial image of the current frame: In the image structure direction matrix of the aerial image of the current frame, the current element is taken as the first row and first column element. If corresponding elements can be selected to the right and downward to form a rectangular area with the same size as the feature template, the rectangular area corresponding to the current element is recorded as a feature area; otherwise, no operation is performed and the judgment is continued on the next element; For each feature region, a subtractor is used to pipeline-calculate the difference between the feature region and the corresponding element in the feature template to obtain a corresponding difference matrix, and the difference matrix is transmitted to a pipeline coordinate rotation digital calculation unit; The pipeline coordinate rotation digital calculation unit calculates the cosine value of each difference in the difference matrix as the cosine similarity, obtains the corresponding cosine similarity matrix, and transmits the cosine similarity matrix to the accumulator; The accumulator adds all cosine similarities in the cosine similarity matrix, records it as the matching similarity of the current element, and transmits it to the comparator; The comparator compares the maximum value of the matching similarity in a pipeline search, and uses the element corresponding to the maximum value of the matching similarity as the matching point of the feature template; Then, a matching point pair is formed based on the element in the first row and first column of the feature template and the currently obtained matching point; Repeat the above process to obtain N matching points corresponding to the N feature templates and N matching point pairs, and transmit them to the transformation matrix calculation module.