A Fast Matching System for Monocular Aerial Images Based on FPGA

Abstract

The present invention relates to the technical field of UAV aerial photography image processing, and discloses a fast matching system for monocular aerial photography images based on FPGA. Using FPGA as the processing platform, under the coordination of a monocular aerial photography image acquisition unit, a system control unit, a memory, a data communication unit, and an FPGA peripheral circuit, it processes the monocular aerial photography images obtained by the UAV online. The modules in the aerial photography image processing unit use the AXI-Stream bus protocol for data interaction, and each module adopts a parallel pipeline architecture to process the monocular aerial photography images, calculate the feature direction matrix of the monocular aerial photography images and intercept the feature direction template, and complete the matching of adjacent frame images in the monocular aerial photography images by searching for the matching position of the feature direction template of the current frame monocular aerial photography image in the feature direction matrix of the previous frame monocular aerial photography image. The present invention effectively improves the parallelism, data throughput, and operation efficiency of the UAV monocular aerial photography image matching process, and realizes the fast matching of UAV monocular aerial photography images.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV aerial image processing, and particularly to a fast matching system for monocular aerial images based on FPGA. Background Art

[0002] UAV remote sensing is a new type of remote sensing technology that combines remote sensing mapping technology, UAV flight technology, positioning technology, measurement and control technology, and remote sensing application technology. With the rapid development of the scientific and technological society and the continuous deepening of remote sensing informatization construction, the application fields and research scope of UAV remote sensing technology have been gradually expanded and widely used in many fields, such as crop monitoring, smart city, emergency rescue, etc. During the execution of UAV remote sensing tasks, providing real-time and reliable positioning information for the UAV platform is an important guarantee for the successful completion of remote sensing tasks.

[0003] Currently, the Global Positioning System (GPS) is often used in UAV remote sensing technology to provide positioning information for the UAV platform. The GPS positioning technology is widely used, the positioning method is mature, and the device is small in size and easy to integrate. In the case of good GPS satellite signals, high positioning accuracy can be achieved. However, the positioning effectiveness of this method seriously depends on the quality of GPS satellite signals. In a GPS-denied environment where there are interferences or absences in GPS satellite signals, at this time, GPS will fail and cannot effectively provide positioning information for the UAV platform, resulting in UAV out of control and UAV remote sensing task failure. UAV autonomous positioning technology is the key to meeting the positioning requirements of UAV remote sensing tasks in a GPS-denied environment.

[0004] Existing UAV autonomous positioning technologies generally rely on visual image processing technology. By matching adjacent frame images in the monocular aerial images obtained by the UAV, the transformation matrix between adjacent frame images is obtained, and then the pose parameters of the UAV platform are calculated from the transformation matrix to achieve real-time autonomous positioning of the UAV platform. However, the currently widely used UAV monocular aerial image matching system often adopts a Graphics Processing Unit (GPU) platform, which is large in size, high in power consumption, poor in stability, and insufficient in operation parallelism. Summary of the Invention

[0005] To make up for the above deficiencies, the present invention designs a fast matching system for monocular aerial images based on FPGA, which processes the monocular aerial images obtained by the unmanned aerial vehicle (UAV) with FPGA as the processing platform. By searching for the matching position of the feature direction template of the current frame of monocular aerial image in the feature direction matrix of the previous frame of monocular aerial image, the matching of adjacent frame images in the UAV's monocular aerial images is realized. During the matching process of the monocular aerial images, each processing module adopts a parallel pipeline architecture, and the modules use the AXI-Stream bus protocol for data interaction. With the powerful parallel processing ability of the FPGA platform, the parallelism, data throughput and operation efficiency of the UAV's monocular aerial image matching process are effectively improved, realizing the fast matching of the UAV's monocular aerial images. Moreover, with the advantages of low power consumption, small size and high stability of the FPGA, the miniaturization of the system can be achieved and the system reliability can be improved.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A fast matching system for monocular aerial images based on FPGA, comprising a monocular aerial image acquisition unit, a system control unit, an aerial image processing unit, a memory and a data communication unit; the aerial image processing unit includes an aerial camera status monitoring module, an aerial camera driving module, an aerial image receiving module, a memory driving module, a grayscale module, a Gaussian filtering module, a feature direction calculation module, a feature direction matrix intercepting module, a feature direction template caching module, a template matching module and a transformation matrix calculation module deployed on the FPGA, and the monocular aerial image acquisition unit includes a monocular aerial camera, a camera pose adjustment module and an aerial image transmission interface;

[0008] The system control unit reads the camera status parameters in the aerial camera status monitoring module through the data communication unit, sets the camera working parameters and attitude parameters according to the read camera status parameters, and sends them to the aerial camera driving module;

[0009] The aerial camera driving module generates control signals according to the camera working parameters and attitude parameters, and respectively controls the monocular aerial camera and the camera pose adjustment module;

[0010] The monocular aerial camera sends the collected monocular aerial sequence images to the aerial image receiving module through the aerial image transmission interface;

[0011] The aerial image receiving module reads the monocular aerial sequence images and stores them frame by frame in the memory through the memory driving module;

[0012] The grayscale module reads the current frame of monocular aerial photography sequence images in the memory through the memory driving module, converts them into AXI-Stream image data streams, and processes the AXI-Stream image data streams in a pipelined manner to obtain grayscale image data streams, and sends the grayscale image data streams to the Gaussian filtering module;

[0013] The Gaussian filtering module processes the grayscale image data stream in a pipelined manner, performs Gaussian filtering denoising operation, obtains the filtered image data stream, and sends the filtered image data stream to the feature direction calculation module;

[0014] The feature direction calculation module convolves the filtered image data stream with the Sobel operator to obtain a gradient matrix, calculates the tangent value of the feature direction by taking the ratio of the product and the square difference of the gradient matrix, then calculates the arctangent value of the tangent value of the feature direction as the feature direction matrix, and stores the feature direction matrix in the memory through the memory driving module;

[0015] The feature direction matrix truncation module reads the feature direction matrix of the current frame of monocular aerial photography image in the memory through the memory driving module, and truncates the region of interest in the feature direction matrix as the feature direction template and stores it in the feature direction template cache module;

[0016] The template matching module reads the feature direction matrix of the previous frame of monocular aerial photography image in the memory through the memory driving module, calculates the cosine similarity between the feature direction matrix of the previous frame of monocular aerial photography image and the feature direction template pixel by pixel, searches for the pixel position where the maximum value of the cosine similarity is located as the matching position of the feature direction template in the previous frame of monocular aerial photography image, and stores the matching position in the memory through the memory driving module;

[0017] The transformation matrix calculation module reads the matching position of the feature direction template in the previous frame of monocular aerial photography image in the memory through the memory driving module, and then uses the least squares method to solve the transformation matrix between the current frame of monocular aerial photography image and the previous frame of monocular aerial photography image from the position of the feature direction template in the current frame of monocular aerial photography image and the matching position in the previous frame of monocular aerial photography image, and sends the transformation matrix to the system control unit through the data communication unit.

[0018] Furthermore, the aerial photography image transmission interface is adapted to the aerial photography image receiving module and is directly connected through the FPGA pins. The aerial photography image transmission interface includes an Ethernet port, a universal serial bus, a serial interface, a high-definition multimedia interface, a digital video interface, and a video graphics array interface. The aerial photography image receiving module completes the fast decoding of the monocular aerial photography sequence images by processing the data signals incoming from the FPGA pins in a pipelined manner.

[0019] Furthermore, the grayscale module, Gaussian filtering module, and feature direction calculation module all adopt a pipeline architecture. The grayscale module calculates the grayscale values of pixel points in the monocular aerial image in a pipelined manner. The Gaussian filtering module calculates the filtered grayscale values of pixel points in the grayscale image data stream in a pipelined manner. The feature direction calculation module calculates the feature direction matrix in a pipelined manner. The grayscale module and the Gaussian filtering module transmit the grayscale image data stream in a pipelined manner using the AXI-Stream image data stream format. The Gaussian filtering module and the feature direction calculation module transmit the filtered image data stream in a pipelined manner using the AXI-Stream image data stream format. Based on the parallel processing ability of the FPGA, the transmission rates of the grayscale image data stream and the filtered image data stream are matched with the data processing efficiencies of the grayscale module, Gaussian filtering module, and feature direction calculation module.

[0020] Furthermore, the system control unit consists of an embedded computer and control software. The control software is deployed on the embedded computer. The data communication unit consists of a data transmission interface and a data interface drive circuit.

[0021] The data transmission interface is adapted to the embedded computer and includes an Ethernet port, a universal serial bus, a serial interface, a high-speed serial computer expansion bus, a PCI expansion interface for instrument systems, and a PXIe interface.

[0022] Furthermore, the memory contains three data buffers composed of a solid-state electronic storage chip array. The three data buffers are independent of each other. The aerial image processing unit uses the memory drive module to store the monocular aerial image sequence in the first data buffer, store the feature direction matrix in the second data buffer, and store the matching position in the third data buffer. The memory is directly connected to the memory drive module through the FPGA pins. Using the parallel processing ability of the FPGA, the memory drive module realizes data interaction with the three data buffers at the same time.

[0023] The present invention effectively improves the parallelism, data throughput, and operating efficiency of the matching process of the drone's monocular aerial image, realizes the fast matching of the drone's monocular aerial image, and the FPGA has low power consumption, small volume, and high stability, which can miniaturize the system and improve the system reliability. Description of the Drawings

[0024] Figure 1 is the block diagram of the fast matching system for the drone's monocular aerial image based on the FPGA of the present invention.

[0025] Figure 2 is the framework diagram of the aerial image processing unit of the present invention. Detailed Embodiments

[0026] Figure 1 It is a block diagram of a fast matching system for monocular aerial images based on FPGA. The system consists of a monocular aerial image acquisition unit, a system control unit, an aerial image processing unit, a memory, a data communication unit, and an FPGA peripheral circuit. The monocular aerial image acquisition unit is composed of a monocular aerial camera, a camera pose adjustment module, and an aerial image transmission interface. The system control unit is composed of an embedded computer and control software. The data communication unit is composed of a data transmission interface and a data interface drive circuit. The data transmission interface is adapted to the embedded computer and includes an Ethernet port, a universal serial bus, a serial interface, a high-speed serial computer expansion bus, a PCI expansion interface for instrument systems, and a PXIe interface. The aerial image processing unit is a hardware program deployed inside the FPGA. After the system is powered on, the program configuration and reset of the aerial image processing unit inside the FPGA are completed with the help of the FPGA peripheral circuit, and the system is in a standby state. Figure 2 It is Figure 1 a framework diagram of the aerial image processing unit 101 in the system, including an aerial camera status monitoring module, an aerial camera drive module, an aerial image receiving module, a memory drive module, a grayscale conversion module, a Gaussian filtering module, a feature direction calculation module, a feature direction matrix truncation module, a feature direction template cache module, a template matching module, and a transformation matrix calculation module.

[0027] In the standby state of the system, the aerial camera status monitoring module continuously obtains the camera status parameters of the monocular aerial camera. Before the system works, the control software deployed inside the embedded computer reads the camera status parameters in the aerial camera status monitoring module with the help of the data communication unit. During the reading process, the data interface drive circuit drives the data transmission interface to complete the data transmission between the system control unit and the aerial image processing unit. The control software sets the camera working parameters and attitude parameters according to the camera status parameters, and sends the camera working parameters and attitude parameters to the aerial camera drive module through the data communication unit.

[0028] After the system starts to work, the aerial camera drive module generates control signals according to the camera working parameters and attitude parameters, respectively controlling the monocular aerial camera and the camera pose adjustment module;

[0029] The monocular aerial camera sends the collected monocular aerial sequence images to the aerial image receiving module through the aerial image transmission interface;

[0030] The aerial image receiving module reads the monocular aerial sequence images and stores them frame by frame into the memory through the memory drive module;

[0031] The grayscale module reads the current frame of the monocular aerial photography sequence image in the memory through the memory driving module, converts it into an AXI-Stream image data stream, and processes the AXI-Stream image data stream in a pipelined manner to obtain a grayscale image data stream, and sends the grayscale image data stream to the Gaussian filtering module;

[0032] The Gaussian filtering module processes the grayscale image data stream in a pipelined manner, performs Gaussian filtering denoising operation, obtains the filtered image data stream, and sends the filtered image data stream to the feature direction calculation module;

[0033] The feature direction calculation module convolves the filtered image data stream with the Sobel operator to obtain a gradient matrix, calculates the tangent value of the feature direction by taking the ratio of the product and the square difference of the gradient matrix, then calculates the arctangent value of the tangent value of the feature direction as the feature direction matrix, and stores the feature direction matrix in the memory through the memory driving module;

[0034] The feature direction matrix truncation module reads the feature direction matrix of the current frame of the monocular aerial photography image in the memory through the memory driving module, and truncates the region of interest in the feature direction matrix as the feature direction template and stores it in the feature direction template cache module;

[0035] The template matching module reads the feature direction matrix of the previous frame of the monocular aerial photography image in the memory through the memory driving module, calculates the cosine similarity between the feature direction matrix of the previous frame of the monocular aerial photography image and the feature direction template pixel by pixel, searches for the pixel position where the maximum value of the cosine similarity is located as the matching position of the feature direction template in the previous frame of the monocular aerial photography image, and stores the matching position in the memory through the memory driving module;

[0036] The transformation matrix calculation module reads the matching position of the feature direction template in the previous frame of the monocular aerial photography image in the memory through the memory driving module, and then uses the least squares method to solve the transformation matrix between the current frame of the monocular aerial photography image and the previous frame of the monocular aerial photography image from the position of the feature direction template in the current frame of the monocular aerial photography image and the matching position in the previous frame of the monocular aerial photography image, and sends the transformation matrix to the system control unit through the data communication unit.

[0037] The aerial photography image transmission interface is adapted to the aerial photography image receiving module and is directly connected through the FPGA pins. The aerial photography image transmission interface includes an Ethernet port, a universal serial bus, a serial interface, a high-definition multimedia interface, a digital video interface, and a video graphics array interface. The aerial photography image receiving module completes the fast decoding of the monocular aerial photography sequence image by processing the data signals input through the FPGA pins in a pipelined manner.

[0038] The grayscale module, Gaussian filter module and feature direction calculation module all adopt a pipeline architecture. The grayscale module pipeline calculates the grayscale values of pixels in the monocular aerial image, the Gaussian filter module pipeline calculates the filtered grayscale values of pixels in the grayscale image data stream, and the feature direction calculation module pipeline calculates the feature direction matrix; the grayscale image data stream is pipelined using the AXI-Stream image data stream format between the grayscale module and the Gaussian filter module, and the filtered image data stream is pipelined using the AXI-Stream image data stream format between the Gaussian filter module and the feature direction calculation module. Based on the FPGA parallel processing capability, the transmission rates of the grayscale image data stream and the filtered image data stream are matched with the data processing efficiency of the grayscale module, the Gaussian filter module and the feature direction calculation module, effectively avoiding data blocking, improving data throughput, and realizing online processing and fast matching of the monocular aerial image.

[0039] The memory includes three data buffers composed of a solid-state electronic storage chip array, and the three data buffers are independent of each other; the aerial image processing unit uses a memory driver module to store a monocular aerial sequence image into a first data buffer, a feature direction matrix into a second data buffer, and a matching position into a third data buffer. The memory is directly connected to the memory driver module through an FPGA pin. By using the FPGA parallel processing capability, the memory driver module realizes data interaction with the three data buffers at the same time.

[0040] The above is only a basic scheme of the specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be thought of by any person familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. All changes that fall within the equivalent meaning and scope of the claims will be included in the scope of the claims.

Claims

1. A fast matching system for monocular aerial images based on FPGA, characterized in that, It includes a monocular aerial image acquisition unit, a system control unit, an aerial image processing unit, a memory, and a data communication unit; the aerial image processing unit includes an aerial camera status monitoring module, an aerial camera driving module, an aerial image receiving module, a memory driving module, a grayscale module, a Gaussian filtering module, a feature direction calculation module, a feature direction matrix truncation module, a feature direction template cache module, a template matching module, and a transformation matrix calculation module deployed on an FPGA. The monocular aerial image acquisition unit includes a monocular aerial camera, a camera pose adjustment module, and an aerial image transmission interface; The system control unit reads the camera status parameters in the aerial camera status monitoring module through the data communication unit, sets the camera working parameters and attitude parameters according to the read camera status parameters, and sends them to the aerial camera driving module; The aerial camera driving module generates control signals according to the camera working parameters and attitude parameters, and respectively controls the monocular aerial camera and the camera pose adjustment module; The monocular aerial camera sends the collected monocular aerial sequence images to the aerial image receiving module through the aerial image transmission interface; The aerial image receiving module reads the monocular aerial sequence images and stores them frame by frame in the memory through the memory driving module; The grayscale module reads the current frame of the monocular aerial sequence image in the memory through the memory driving module, converts it into an AXI-Stream image data stream, and processes the AXI-Stream image data stream in a pipelined manner to obtain a grayscale image data stream, and sends the grayscale image data stream to the Gaussian filtering module; The Gaussian filtering module processes the grayscale image data stream in a pipelined manner, performs Gaussian filtering denoising operation, obtains the filtered image data stream, and sends the filtered image data stream to the feature direction calculation module; The feature direction calculation module convolves the filtered image data stream with the Sobel operator to obtain a gradient matrix, calculates the tangent value of the feature direction by taking the ratio of the product and the square difference of the gradient matrix, then calculates the arctangent value of the tangent value of the feature direction as the feature direction matrix, and stores the feature direction matrix in the memory through the memory driving module; The feature direction matrix truncation module reads the feature direction matrix of the current frame of the monocular aerial image in the memory through the memory driving module, and truncates the region of interest in the feature direction matrix as the feature direction template and stores it in the feature direction template cache module; The template matching module reads the feature direction matrix of the previous frame of the monocular aerial image in the memory through the memory driving module, calculates the cosine similarity between the feature direction matrix of the previous frame of the monocular aerial image and the feature direction template pixel by pixel, searches for the pixel position where the maximum value of the cosine similarity is located as the matching position of the feature direction template in the previous frame of the monocular aerial image, and stores the matching position in the memory through the memory driving module; The transformation matrix calculation module reads the matching positions of the feature direction templates in the previous-frame monocular aerial image from the memory through the memory driving module, and then uses the least squares method to solve the transformation matrix between the current-frame monocular aerial image and the previous-frame monocular aerial image based on the positions of the feature direction templates in the current-frame monocular aerial image and the matching positions in the previous-frame monocular aerial image, and sends the transformation matrix to the system control unit through the data communication unit.

2. The monocular aerial image rapid matching system based on FPGA according to claim 1, characterized in that, The aerial image transmission interface is adapted to the aerial image receiving module and is directly connected through the FPGA pins. The aerial image transmission interface includes an Ethernet port, a universal serial bus, a serial interface, a high-definition multimedia interface, a digital video interface, and a video graphics array interface. The aerial image receiving module completes the fast decoding of the monocular aerial image sequence by processing the data signals incoming through the FPGA pins in a pipelined manner.

3. A monocular aerial image fast matching system based on FPGA according to claim 1, characterized in that, The grayscale module, the Gaussian filtering module, and the feature direction calculation module all adopt a pipelined architecture. The grayscale module calculates the grayscale values of the pixel points in the monocular aerial image in a pipelined manner. The Gaussian filtering module calculates the filtered grayscale values of the pixel points in the grayscale image data stream in a pipelined manner. The feature direction calculation module calculates the feature direction matrix in a pipelined manner. The grayscale module and the Gaussian filtering module transmit the grayscale image data stream in a pipelined manner using the AXI-Stream image data stream format. The Gaussian filtering module and the feature direction calculation module transmit the filtered image data stream in a pipelined manner using the AXI-Stream image data stream format. Based on the parallel processing ability of the FPGA, the transmission rates of the grayscale image data stream and the filtered image data stream are matched with the data processing efficiencies of the grayscale module, the Gaussian filtering module, and the feature direction calculation module.

4. A monocular aerial image rapid matching system based on FPGA according to claim 1, characterized in that, The system control unit consists of an embedded computer and control software. The control software is deployed on the embedded computer. The data communication unit consists of a data transmission interface and a data interface driving circuit. The data transmission interface is adapted to the embedded computer and includes an Ethernet port, a universal serial bus, a serial interface, a high-speed serial computer expansion bus, a PCI expansion interface for instrument systems, and a PXIe interface.

5. A monocular aerial image fast matching system based on FPGA according to claim 1, characterized in that The memory contains three data buffers composed of a solid-state electronic storage chip array. The three data buffers are independent of each other. The aerial image processing unit uses the memory driving module to store the monocular aerial image sequence in the first data buffer, store the feature direction matrix in the second data buffer, and store the matching positions in the third data buffer. The memory is directly connected to the memory driving module through the FPGA pins. Using the parallel processing ability of the FPGA, the memory driving module realizes data interaction with the three data buffers at the same time.

Images