FPGA-based unmanned aerial vehicle video oil pipeline display method

Abstract

The application provides an unmanned aerial vehicle video oil pipeline display method based on FPGA, solves the problem that specific pipeline information cannot be accurately displayed in real-time video when an unmanned aerial vehicle carries a double photoelectric pod, flies along an oil pipeline, shoots and transmits real-time video, and the technical scheme of the method adopts an FPGA chip as an independent video processing engine and a video decoding module of the whole system architecture, significantly strengthens the image processing problem ability, processes video streams, displays oil pipeline information on the output video of the user end by fusing flight control data and changing video image RGB24 color space data, has high precision in displaying pipeline information, 3D environment reconstruction provides reliable data and visibility, accurately displays specific pipeline information through the video, improves the work efficiency of customers in real-time management of oil pipelines, and effectively controls the occurrence of oil pipeline risks in time. The application is used in the field of unmanned aerial vehicle inspection of oil pipelines.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection of oil pipelines, and more particularly to a UAV video display method based on FPGA for oil pipelines. Background Technology

[0002] With the rapid development of my country's oil and gas industry, the construction of oil and gas pipelines has progressed by leaps and bounds. Oil and gas pipelines are a crucial component of the oil and gas system, and their safe and reliable operation is directly related to the stable development of the national economy and the protection and control of the natural environment. Some oil and gas pipelines, due to long-term exposure to the natural environment, not only bear normal mechanical loads and the internal pressure of oil transportation, but also endure external damage such as rainfall, snowfall, landslides, and man-made drilling. These factors will accelerate the aging of various pipeline components. If not detected and eliminated in time, they may develop into various malfunctions, posing a serious threat to the safety and stability of the oil system. Therefore, pipeline inspection is a fundamental task in ensuring the safety of oil pipeline equipment. By inspecting oil and gas pipelines, we can understand the pipeline's operating status and changes in the surrounding environment, promptly identify equipment defects and safety hazards, and propose specific maintenance suggestions to eliminate defects in a timely manner, prevent malfunctions, and thus ensure the safety of oil and gas pipelines and the stable operation of the oil and gas system.

[0003] Unmanned aerial vehicles (UAVs) are powered, controllable, reusable unmanned aircraft capable of carrying multiple mission devices and performing various tasks. They provide comprehensive and rapid information on target terrain, hydrology, and meteorology. Equipped with high-resolution aerial cameras and mapping pods, they can survey and photograph mission areas, providing visualized video and high-definition photographs, and performing data processing and comparison. They offer regional geographic and meteorological data, enabling timely access to real-time environmental information around oil and gas pipelines, providing more convenient and time-saving methods for pipeline inspection.

[0004] Compared to manual oil pipeline inspections, drone inspections are faster, more efficient, unaffected by terrain or weather, and offer higher safety, while reducing the workload of personnel. Low-altitude drone photogrammetry is more convenient and less costly than fiber optic electronic sensing. Compared to satellites, drones offer advantages such as maneuverability, environmental adaptability, and high imaging resolution, enabling both regular pipeline inspections and emergency investigations. Drone inspections are cost-effective and particularly suitable for monitoring pipeline systems.

[0005] By using drones equipped with dual-light photoelectric pods to fly along oil pipelines, they can capture and transmit real-time inspection videos to determine whether there are leaks or third-party encroachment. However, since most oil pipelines are buried underground, it is impossible to accurately determine the location of the pipelines in the inspection videos. Therefore, it is essential to overlay oil pipeline information onto the real-time inspection videos.

[0006] Because drones capture and save real-time inspection videos, which are then checked by relevant personnel after filming, this method has an excessively long video processing cycle and cannot achieve real-time performance. If there are equipment defects or safety hazards, the risks will be further amplified. Alternatively, drones equipped with dual electro-optical pods can fly along oil pipelines to capture and transmit real-time videos, but the real-time videos cannot accurately display specific pipeline information. Summary of the Invention

[0007] A drone equipped with dual electro-optical pods flies along an oil pipeline, capturing and transmitting real-time video. However, the real-time video cannot accurately display the specific information of the pipeline. This method uses an FPGA as an independent video processing engine to process the video stream. By fusing data and changing the RGB24 color space data of the video image, the oil pipeline information is displayed on the video output to the user. This invention provides a method and device for displaying oil pipeline information from a drone based on FPGA. This method displays pipeline information with high accuracy and provides reliable data for 3D environment reconstruction.

[0008] In one aspect, the present invention provides a method for displaying video oil pipelines from a drone based on FPGA, the method comprising the following steps:

[0009] Step 1: The geodetic coordinates of the drone camera's location are transformed into Gaussian plane coordinates using a forward Gaussian transform. The Gaussian plane, combined with the altitude, forms a three-dimensional space, and the camera center point obtains a three-dimensional coordinate. Based on the focal length, the camera center spatial coordinates, and the angle, the spatial coordinates corresponding to each pixel in the target image are determined. Based on the spatial coordinates of each pixel and the spatial coordinates of the camera center point, the coordinates of the actual position of the pixel in the Gaussian plane are calculated. The Gaussian coordinates of the actual position of the pixel are then transformed using an inverse Gaussian transform to obtain its corresponding geodetic coordinates.

[0010] Step 2: Based on geodetic coordinates The obtained latitude and longitude information is used as input and read into the FPGA processor. It is then queried and matched with the GPS positioning information in the cache. The latitude and longitude are matched to obtain the precise height information of the corresponding pixel. Then, the range and side length of the target image area captured by the real-time video stream are calculated using the four-corner coordinate information. The GPS positioning information includes three information values: longitude, latitude, and height, determined for each point.

[0011] Step 3: Match the oil pipeline information uploaded by the ground station with the target image area to determine the location of the oil pipeline on the target image. By changing the RGB24 color space data of the oil pipeline relative to the image area, the pipeline information is displayed in the real-time video.

[0012] Preferably, the coordinates of the actual position of the pixel in the Gaussian plane are calculated, specifically using known geodetic coordinates. and the longitude of the central meridian Calculate Gaussian plane coordinates The formula is as follows:

[0013]

[0014]

[0015]

[0016] Where B is latitude, The unit is radians. , where is the radius of curvature of the circle.

[0017] , , denoted by the second eccentricity, a is the semi-major axis of the rotating ellipsoid, b is the semi-minor axis, and X is the arc length of the meridian.

[0018] Preferably, the Gaussian coordinates of the actual location of the pixel are obtained by inverse Gaussian transformation to obtain its corresponding geodetic coordinates; the Gaussian plane coordinates are known. and designated central meridian longitude Calculate the geodetic coordinates :

[0019]

[0020]

[0021] in, , , , ,

[0022] Let be the latitude of the base point calculated from the meridian arc length X, and 'a' be the semi-major axis of the rotating ellipsoid.

[0023] Preferably, the side length of the target image region is calculated using the four-corner coordinate information; specifically, this includes calculating the range and side length information of the image region captured by the real-time video stream using the four-corner coordinate information.

[0024] Preferably, the location of the oil pipeline on the target image is determined by matching the oil pipeline information uploaded by the ground station with the target image area. Specifically, this includes: displaying pipeline information in real-time video; uploading the oil pipeline information in the form of a kml file to the FPGA in advance; wherein the kml file is a pipeline file containing information on multiple discrete coordinate points, including the longitude and latitude of each point; during the matching process, overlapping coordinate points are tagged and cached; after the matching is completed, the location of the oil pipeline on the image is determined.

[0025] This invention provides an FPGA-based method for displaying oil pipeline information from a drone video feed. It solves the problem that drones equipped with dual optoelectronic pods fly along oil pipelines, capturing and transmitting real-time video, but the video cannot accurately display specific pipeline information. By using an FPGA chip as an independent video processing engine, the method processes the video stream, fuses data, and modifies the RGB24 color space data of the video image, displaying oil pipeline information on the video output to the user. The displayed pipeline information is highly accurate, and the 3D environment reconstruction provides reliable data and visibility. This precise display of pipeline information improves the efficiency of real-time management of oil pipelines for customers and provides timely warnings to effectively control oil pipeline risks.

[0026] In another aspect, this invention provides an FPGA-based video processing buffer device. The device includes a microcontroller ARM and an FPGA processor. The microcontroller ARM and the FPGA chip are connected via two dual-port RAMs. Data communication between the microcontroller ARM and the FPGA processor is via SPI communication. During a video data write operation, a write address is used to write data from port A1 to memory RAM1. During a read operation, a read address is used to read the logically processed video data from port B1. Similarly, a write address is used to write data from port A2 to memory RAM2, and a read address is used to read the data from port B2. The logically processed video data is then transmitted to the CPU.

[0027] Preferably, the data communication between the microcontroller ARM and the FPGA processor adopts SPI communication, which means that the microcontroller ARM and the FPGA processor are connected by a serial peripheral interface SPI. The serial peripheral interface SPI is a high-speed full-duplex high-speed communication bus, and the master output and slave input mosi and master input and slave input miso of the serial peripheral interface SPI are used to transmit data respectively.

[0028] Preferably, the FPGA processor is configured with a Serial Peripheral Interface (SPI) for connection. The timing design of the SPI is completed using Verilog HDL, and two dual-port RAMs are constructed using IP cores. The video data in the RAMs is transmitted to the FPGA processor via the SPI, and the video data processed by the FPGA processor is transmitted back to the microcontroller ARM.

[0029] Preferably, the video data to be transmitted back to the microcontroller ARM after processing by the FPGA processor refers to the FPGA processor performing video stream processing logic and Kalman filtering on the video data; the processing involves matching the location information of the area captured by the real-time video stream with the known coordinate information of the oil pipeline, and changing the RGB24 color space data of the image where the oil pipeline is located in the video image to realize the display of oil pipeline information in the real-time video.

[0030] Preferably, the FPGA processor controls the A / D conversion and stores the converted video data sequentially into a dual-port RAM. When the video data is stored, it interrupts the ARM microcontroller, which then reads the video data from the dual-port RAM.

[0031] Another aspect of this invention provides an FPGA-based video processing cache device, which solves the problem of excessively long video data processing cycles and the inability to transmit video data in real time during inspection videos. The microcontroller ARM excels in transaction management, while the FPGA chip excels in computation. The FPGA, as an auxiliary function chip and the video decoding module of the entire system architecture, significantly enhances its image processing capabilities and has greater specialization. The FPGA video processing cache device uses two dual-port RAMs to connect the microcontroller ARM and the FPGA chip, transmitting the logically processed video data to the CPU, thus improving the system's data transmission efficiency. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of a method for displaying video oil pipelines from a drone based on FPGA.

[0033] Figure 2 This is a schematic diagram of the hardware architecture and data flow of a drone video oil pipeline display method based on FPGA;

[0034] Figure 3 This is a schematic diagram of the hardware connection between FPGA and ARM, representing a method for displaying oil pipelines from drone video.

[0035] Figure 4 This is a schematic diagram of the FPGA chip working in a drone video oil pipeline display method based on FPGA. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0037] Example 1

[0038] This invention provides an FPGA-based method for displaying video oil pipelines from unmanned aerial vehicles (UAVs), such as... Figure 1 As shown, the method includes the following steps:

[0039] Step 1: The geodetic coordinates of the drone camera's location are transformed into Gaussian plane coordinates using a forward Gaussian transform. The Gaussian plane, combined with the altitude, forms a three-dimensional space, and the camera center point obtains a three-dimensional coordinate. Based on the focal length, the camera center spatial coordinates, and the angle, the spatial coordinates corresponding to each pixel in the target image are determined. Based on the spatial coordinates of each pixel and the spatial coordinates of the camera center point, the coordinates of the actual position of the pixel in the Gaussian plane are calculated. The Gaussian coordinates of the actual position of the pixel are then transformed using an inverse Gaussian transform to obtain its corresponding geodetic coordinates.

[0040] Step 2: Based on geodetic coordinates The obtained latitude and longitude information is used as input and read into the FPGA processor. It is then queried and matched with the GPS positioning information in the cache. The latitude and longitude are matched to obtain the precise height information of the corresponding pixel. Then, the range and side length of the target image area captured by the real-time video stream are calculated using the four-corner coordinate information. The GPS positioning information includes three information values: longitude, latitude, and height, determined for each point.

[0041] Step 3: Match the oil pipeline information uploaded by the ground station with the target image area to determine the location of the oil pipeline on the target image. By changing the RGB24 color space data of the oil pipeline relative to the image area, the pipeline information is displayed in the real-time video.

[0042] In one example, determining the Gaussian plane coordinates of the actual location of a pixel refers to knowing the geodetic coordinates. and the longitude of the central meridian Calculate Gaussian plane coordinates The formula is as follows:

[0043]

[0044]

[0045]

[0046] Where B is latitude, The unit is radians. , where is the radius of curvature of the circle.

[0047] , , denoted by the second eccentricity, a is the semi-major axis of the rotating ellipsoid, b is the semi-minor axis, and X is the arc length of the meridian.

[0048] In one embodiment, the Gaussian coordinates of the actual location of a pixel are transformed by an inverse Gaussian transform to obtain its corresponding geodetic coordinates; the Gaussian plane coordinates are known. and designated central meridian longitude Calculate the geodetic coordinates :

[0049]

[0050]

[0051] in, , , , , The latitude of the base point is calculated by inversely from the arc length of the meridian.

[0052] In one embodiment, based on geodetic coordinates The obtained latitude and longitude information is used as input and read into the FPGA processor. It is then queried and matched with the cached GPS positioning information. By matching two data values, the precise altitude information of the corresponding point can be obtained. The GPS positioning information includes three information values: longitude, latitude, and altitude for each point.

[0053] Among them, the side length of the target image region is calculated using the four-corner coordinate information; the range and side length information of the real-time video stream captured image region are calculated using the four-corner coordinate information, where 'a' is the semi-major axis of the ellipsoid of revolution.

[0054] In one embodiment, the location of the oil pipeline on the target image is determined by matching the oil pipeline information uploaded by the ground station with the target image area. The pipeline information is then displayed in the real-time video by changing the RGB24 color space data of the relative position. The oil pipeline information is pre-uploaded to the FPGA, where the kml file is a pipeline file containing information on multiple discrete coordinate points, including the longitude and latitude of each point. The coordinate point information in the image area is matched, and overlapping coordinate points are tagged and cached during the matching process. After the matching is completed, the location of the oil pipeline on the image is determined. By changing the RGB24 color space data of the relative position coordinate point image in the real-time video stream, the pipeline information can be displayed in the real-time video.

[0055] This invention provides an FPGA-based method for displaying oil pipeline information from a drone video feed. It solves the problem that drones equipped with dual optoelectronic pods fly along oil pipelines, capturing and transmitting real-time video, but the video cannot accurately display specific pipeline information. By using an FPGA chip as an independent video processing engine, the method processes the video stream, fuses data, and modifies the RGB24 color space data of the video image, displaying oil pipeline information on the video output to the user. The displayed pipeline information is highly accurate, and the 3D environment reconstruction provides reliable data and visibility. This precise display of pipeline information improves the efficiency of real-time management of oil pipelines for customers and provides timely warnings to effectively control oil pipeline risks.

[0056] In another aspect of this invention, an FPGA-based video processing buffer is provided. The device includes a microcontroller ARM and an FPGA processor. The microcontroller ARM and the FPGA chip are connected via two dual-port RAMs. Data communication between the microcontroller ARM and the FPGA processor is via SPI communication. During a video data write operation, a write address is used to write data from port A1 to memory RAM1. During a read operation, a read address is used to read data from port B1 after logical processing of the video data. Similarly, a write address is used to write data from port A2 to memory RAM2, and a read address is used to read data from port B2. The logically processed video data is then transmitted to the CPU.

[0057] In one embodiment, the data communication between the microcontroller ARM and the FPGA processor uses SPI communication. This means that the microcontroller ARM and the FPGA processor are connected via a serial peripheral interface (SPI). The SPI is a high-speed, full-duplex communication bus, and data is transmitted through the master output / slave input (mosi) and master input / slave input (miso) of the SPI. Figure 3 As shown.

[0058] In one embodiment, the FPGA processor uses a Serial Peripheral Interface (SPI) for connection. The timing design of the SPI is completed using VerilogHDL, and two dual-port RAMs are constructed using IP cores. The video data in the RAMs is transmitted to the FPGA processor via the SPI, and the video data processed by the FPGA processor is transmitted back to the microcontroller ARM.

[0059] In one embodiment, the video data to be transmitted back to the microcontroller ARM after processing by the FPGA processor refers to the FPGA processor performing video stream processing logic and Kalman filtering on the video data; the processing involves matching the location information of the area captured by the real-time video stream with the known coordinate information of the oil pipeline, and changing the RGB24 color space data of the image where the oil pipeline is located in the video image to realize the display of oil pipeline information in the real-time video.

[0060] like Figure 4 As shown, after the FPGA is powered on and configured, the image acquisition module uses the IIC communication protocol to configure the dual-light optoelectronic pod CMOS image sensor and capture the video stream. The DDR2 control module controls the FIFO buffer, which writes the data to the frame buffer module DDR2 SDRAM. The FPGA video processing module reads the value of the parameter configuration register and reads the image from the FIFO buffer in the frame buffer module for processing. After the video is processed, the display driver module reads the image and sends out the corresponding horizontal and vertical synchronization signals according to the VGA protocol to realize the display of the processed image in real time.

[0061] The FPGA processor controls the A / D conversion and stores the converted video data sequentially into a dual-port RAM. When the video data is stored, it interrupts the ARM microcontroller, which then reads the video data from the dual-port RAM.

[0062] Another aspect of this invention provides an FPGA-based video processing cache device, which solves the problem of excessively long video data processing cycles and the inability to transmit video data in real time during inspection videos. The microcontroller ARM excels in transaction management, while the FPGA chip excels in computation. The FPGA, as an auxiliary function chip and the video decoding module of the entire system architecture, significantly enhances its image processing capabilities and has greater specialization. The FPGA video processing cache device uses two dual-port RAMs to connect the microcontroller ARM and the FPGA chip, transmitting the logically processed video data to the CPU, thus improving the system's data transmission efficiency.

[0063] Example 2

[0064] This invention provides an FPGA-based method for displaying video oil pipelines from unmanned aerial vehicles (UAVs). Figure 2 , Figure 3 As shown, the method includes the following steps:

[0065] Firstly, the system uses a combination of ARM and FPGA as its core. ARM processors are suitable for control applications, so ARM is used as the motherboard to control the flight of the inspection drone. Due to its own characteristics, FPGA devices are suitable for high-speed parallel acquisition and processing applications, and have advantages that ARM or DSP processors cannot match. Therefore, FPGA is used as an independent video processing chip. This combination combines the characteristics of each component, resulting in strong processing capabilities and a wide range of applications.

[0066] ARM Components: Due to its powerful transaction management capabilities and control logic, ARM can handle most of the computer's task logic and data acquisition from peripheral interfaces, such as communication with peripherals, data processing, implementation of the entire control logic, and communication with ground stations. Furthermore, ARM itself features low power consumption, low cost, multiple serial ports, and ease of debugging, meeting the requirements for a main control chip.

[0067] The STM32H7 series 32-bit microcontroller based on the ARM Cortex-M7 core from STMicroelectronics was selected as the main control chip for this design. This product is the industry's highest-performing ARM Cortex-M general-purpose MCU, which combines a powerful dual-core processor, energy-saving features, and enhanced network security.

[0068] FPGA Section: Due to its powerful field-programmable and parallel computing capabilities, the FPGA can serve as a coprocessor, porting the complex parallel computing logic of ARM processors to the FPGA for processing. Furthermore, the internal logic of the FPGA can be added to, deleted from, and modified according to actual needs. In this design, the FPGA will implement video stream processing and the hardware acceleration unit function of Extended Kalman Filter (EKF), greatly enhancing scalability and flexibility. Intel's low-cost, low-power Cyclone IVE series FPGA chip is selected.

[0069] Firstly, the interface design between the ARM and FPGA is addressed. Data exchange between the ARM and FPGA utilizes a dual-port RAM chip to implement a FIFO function (FIFO stands for First In / First Out, meaning first-in, first-out). Due to the rapid development of microelectronics technology, new-generation FIFO chips are becoming increasingly larger in capacity, smaller in size, and cheaper. The core of this system adopts a combination of ARM and FPGA, using two dual-port RAM chips for 24-bit data transmission between the ARM and FPGA. The FPGA controls the A / D conversion and stores the converted data into the dual-port RAM in a specific order. When the data storage is complete, the ARM is interrupted, and the ARM reads the data from the dual-port RAM. This approach reduces costs and increases data transmission throughput compared to a FIFO-based solution.

[0070] In this system, the ARM processor serves as the system control core, responsible for controlling the overall system timing and uploading data to the server for storage via the network. The FPGA handles the A / D converter's mode configuration and data transmission. This combination leverages the advantages of ARM in control and FPGA in data acquisition, offering strong versatility and flexible configuration. The FPGA's design functions in this system are as follows: resetting and configuring the A / D converter's operating mode; an internal address adder controls the writing of data from the A / D converter into the dual-port RAM; and when the dual-port RAM is full (24KB), the address adder is reset.

[0071] By using a dual-port RAM connection between the ARM and FPGA, the ARM and FPGA can work asynchronously, which improves system efficiency.

[0072] The ARM and FPGA use SPI communication for data communication. In the ARM, SPI can be set to DMA mode for reading and input. DMA is a data exchange mode that accesses memory directly without going through the CPU. Using DMA, there is no need to perform an external interrupt every time data is received. Instead, the interrupt service is executed only after enough data is received, which greatly reduces the CPU resource consumption.

[0073] In the FPGA, Verilog HDL was used to design the SPI timing, and two simple dual-port RAM memories were built using IP cores. These RAM memories are used to store data transmitted from the ARM to the FPGA via SPI, and data that the FPGA will process and then send back to the ARM. The ARM and FPGA use a half-duplex communication mode to ensure that each data sent back to the ARM is data that has been processed by the FPGA. The overall data flow diagram is shown below. Figure 2As shown.

[0074] Among them, SPI serial peripheral interface, high-speed full-duplex high-speed communication bus; MOSI master output and slave input; MISO master input and slave input.

[0075] The Cyclone IV E series FPGAs support dual-port RAM configurations. This design uses a simple dual-port RAM with an 8-bit width and 512-bit depth. Read and write operations each have dedicated address ports. Write operations use the write address to write data from port A to memory, and read operations use the read address to read data from port B. A hardware connection diagram between the FPGA and ARM is shown below. Figure 3 As shown.

[0076] This device employs a combination of ARM and FPGA at its core. Two dual-port RAM IDT7205 chips are used for 24-bit data transmission between the ARM and FPGA. The FPGA controls the A / D conversion and stores the converted data into the dual-port RAM in a specific order. Once the data storage is complete, the ARM is interrupted, and the ARM reads the data from the dual-port RAM. Figure 4 As shown.

[0077] Vertical takeoff and landing (VTOL) drones are used for oil pipeline inspection. Before takeoff, the pipeline coordinates are input into the ground station software QGC. The ground station's mission management module performs trajectory planning and generates a preset route. After confirmation by the operator, the route is uploaded to the drone. The operator performs a pre-takeoff check at the ground station. After confirmation, the drone will automatically cruise according to the preset route.

[0078] The vertical take-off drone is equipped with a dual-light electro-optical pod, which takes pictures of the area within 100 meters to the left and right of the oil pipeline along the preset route. The video capture card collects real-time video information and acquires image data in real time. The raw video data (YUY2 format) collected in the video capture card is converted into RGB24 format and saved as a bitmap. The image data is acquired in real time and stored in matrix form.

[0079] Camera calibration involves obtaining the camera's internal parameters, including focal lengths fx and fy, and optical centers Cx and Cy.

[0080] The main controller processes the acquired image, first converting it to grayscale and then separating its color channels into R, G, and B channels to reduce the amount of data processed and eliminate interference. A filtering algorithm is then applied to the pixel coordinates of corresponding points to prevent pixel jumps from affecting the calculation of position information.

[0081] Drone aerial images are all EXIF ​​type images. The internal attributes of the images have a very rich series of values, including shooting date, image center latitude and longitude, focal length, altitude, heading angle, roll angle, pitch angle, etc.

[0082] In summary, the UAV flight controller can obtain 12 raw data points: 1. Electro-optical pod heading angle, 2. Electro-optical pod azimuth angle, 3. Electro-optical pod pitch angle, 4. Electro-optical pod roll angle, 5. Electro-optical pod field of view, 6. UAV longitude, 7. UAV latitude, 8. UAV altitude, 9. Image center longitude, 10. Image center latitude, 11. Image center altitude, and 12. GPS time.

[0083] First, based on the raw data transmitted by the drone, the image area of ​​the real-time video is calculated. To calculate the image area, the longitude and latitude of the four corners are determined.

[0084] Definitions of technical terms used in the method:

[0085] ① Focal length: Focal length refers to the distance from the equivalent optical center of the lens to the camera sensor.

[0086] ② Geodetic coordinates: Coordinates used in geodesy with a reference ellipsoid as the reference surface. The position of a point P on the ground is represented by geodetic longitude L, geodetic latitude B, and geodetic height H. When the point lies on the reference ellipsoid, only geodetic longitude and geodetic latitude are used. Geodetic longitude is the angle between the geodetic meridian plane passing through the point and the initial geodetic meridian plane; geodetic latitude is the angle between the normal line passing through the point and the equatorial plane; and geodetic height is the distance from the point on the ground along the normal line to the reference ellipsoid.

[0087] ③ Gaussian plane coordinate system: This refers to a coordinate system with the intersection of the central meridian and the equator as the origin, the projection of the central meridian as the vertical axis X, and the projection of the equator as the horizontal axis Y, with the projection of the equator as the horizontal axis Y, and the projection of the equator as the horizontal axis Y, and the projection of the equator as the horizontal axis Y, and the projection of the equator as the horizontal axis Y.

[0088] Because the Earth is ellipsoidal, and latitude and longitude are geodetic coordinates that are inconvenient to calculate, we first use the Gaussian projection formula to convert the geodetic plane coordinates into Gaussian plane coordinates. These Gaussian plane coordinates, together with the altitude, form a three-dimensional spatial coordinate system. Based on the camera imaging principle, we use the coordinates of the pixel's center as one spatial coordinate and the camera's center coordinate as another. The intersection of the straight line passing through these two points and the Gaussian plane is the actual location of that pixel.

[0089] Specific steps

[0090] Step 1: The geodetic coordinates of the location of the drone camera are transformed into Gaussian plane coordinates through Gaussian forward transformation. The Gaussian plane plus the altitude constitutes a three-dimensional space, so that each point has a corresponding three-dimensional coordinate.

[0091] Step 2: Determine the spatial coordinates of each pixel in the image based on the focal length, the spatial coordinates of the camera center, and the angle.

[0092] Step 3: Calculate the coordinates of the actual position of the pixel in the Gaussian plane based on the pixel's spatial coordinates and the camera center's spatial coordinates.

[0093] Step 4: Obtain the corresponding geodetic coordinates from the Gaussian coordinates of the actual location obtained in Step 3 through inverse Gaussian transformation.

[0094] Given geodetic coordinates and the longitude of the central meridian Calculate Gaussian plane coordinates The formula is as follows:

[0095]

[0096]

[0097]

[0098] Where B is latitude, The unit is radians. , where is the radius of curvature of the circle.

[0099] , , denoted by the second eccentricity, a is the semi-major axis of the rotating ellipsoid, b is the semi-minor axis, and X is the arc length of the meridian.

[0100] Given Gaussian plane coordinates and designated central meridian longitude Calculate geodetic coordinates :

[0101]

[0102]

[0103] in, , , , , Let be the latitude of the base point calculated from the meridian arc length X, and 'a' be the semi-major axis of the rotating ellipsoid.

[0104] a. The calculated latitude and longitude information is used as input and read into the FPGA. It is then queried and matched with the cached GPS positioning information (including the three information values ​​of longitude, latitude and altitude for each point). By matching two of the data values, the accurate altitude information of the corresponding point can be obtained.

[0105] b. Using the coordinates of the four corners, the range and side length of the image area captured by the real-time video stream are calculated.

[0106] c. Upload oil pipeline information to the FPGA in advance in the form of a kml file, which is a pipeline file containing information on multiple discrete coordinate points, including the longitude and latitude of each point.

[0107] d. The coordinate information of the coordinate points in the above image area is matched. During the matching process, the overlapping coordinate points are tagged and cached. After the matching is completed, the location of the oil pipeline on the image is determined. By changing the RGB24 color space data of the relative position coordinate point image in the real-time video stream, the pipeline information can be displayed in the real-time video.

[0108] This invention provides an FPGA-based method for displaying oil pipeline information from a drone video feed. It solves the problem that drones equipped with dual optoelectronic pods fly along oil pipelines, capturing and transmitting real-time video, but the video cannot accurately display specific pipeline information. By using an FPGA chip as an independent video processing engine, the method processes the video stream, fuses data, and modifies the RGB24 color space data of the video image, displaying oil pipeline information on the video output to the user. The displayed pipeline information is highly accurate, and the 3D environment reconstruction provides reliable data and visibility. This precise display of pipeline information improves the efficiency of real-time management of oil pipelines for customers and provides timely warnings to effectively control oil pipeline risks.

Claims

1. A method for displaying oil pipelines from a drone based on FPGA, characterized in that, The method includes the following steps: Step 1: The geodetic coordinates of the drone camera's location are transformed into Gaussian plane coordinates using a forward Gaussian transform. The Gaussian plane, combined with the altitude, forms a three-dimensional space, and the camera center point obtains a three-dimensional coordinate. Based on the focal length, the camera center spatial coordinates, and the angle, the spatial coordinates corresponding to each pixel in the target image are determined. Based on the spatial coordinates of each pixel and the spatial coordinates of the camera center point, the coordinates of the actual position of the pixel in the Gaussian plane are calculated. The Gaussian coordinates of the actual position of the pixel are then transformed using an inverse Gaussian transform to obtain its corresponding geodetic coordinates. Step 2: Based on geodetic coordinates The obtained latitude and longitude information is used as input and read into the FPGA processor. It is then queried and matched with the GPS positioning information in the cache. The latitude and longitude are matched to obtain the precise height information of the corresponding pixel. Then, the range and side length of the target image area captured by the real-time video stream are calculated using the four-corner coordinate information. The GPS positioning information includes three information values: longitude, latitude, and height, determined for each point. Step 3: Match the oil pipeline information uploaded by the ground station with the target image area to determine the location of the oil pipeline on the target image. By changing the RGB24 color space data of the oil pipeline relative to the image area, the pipeline information is displayed in the real-time video.

2. The method for displaying oil pipelines in UAV video based on FPGA according to claim 1, characterized in that, The coordinates of the actual position of the pixel in the Gaussian plane are obtained, and the coordinates in the Gaussian plane are calculated. The formula is as follows: Where B is latitude, The unit is radians. , where is the radius of curvature of the circle. , , denoted by the second eccentricity, a is the semi-major axis of the rotating ellipsoid, b is the semi-minor axis, and X is the arc length of the meridian.

3. The method for displaying oil pipelines in UAV video based on FPGA according to claim 2, characterized in that, The Gaussian coordinates of the actual location of the pixel are obtained by inverse Gaussian transformation to its corresponding geodetic coordinates; the formula is as follows: in, , , , , The latitude of the base point is calculated by inversely from the arc length of the meridian.

4. The method for displaying oil pipelines in UAV video based on FPGA according to claim 1, characterized in that, The step of matching the oil pipeline information uploaded by the ground station with the target image region to determine the location of the oil pipeline on the target image specifically includes: displaying pipeline information in real-time video; pre-uploading the oil pipeline information to the FPGA in the format of a kml file, wherein the kml file is a pipeline file containing information on multiple discrete coordinate points, including the longitude and latitude of each point; during the matching process, overlapping coordinate points are tagged and cached; after the matching is completed, the location of the oil pipeline on the image is determined.

Images