An FPGA-based unmanned aerial vehicle hyperspectral acquisition system and method

Abstract

Multiple embodiments of this specification relate to the field of remote sensing technology, and more specifically, to an FPGA-based unmanned aerial vehicle (UAV) hyperspectral acquisition system and method. The system includes an FPGA processing module, a hyperspectral camera module, a visible light camera module, an inertial measurement module, a zenithal light module, and a power module, all carried by the UAV. The FPGA processing module receives pulse-per-second signals and positioning data from the UAV, and provides trigger signals to the hyperspectral camera module, the visible light camera module, the inertial measurement module, and the zenithal light module based on the pulse-per-second signals. Based on the trigger signals, an acquisition cycle is generated. During the acquisition cycle, the FPGA processing module performs the following steps: creating a temporary storage area; receiving hyperspectral image data, visible light image data, flight attitude data, and zenithal light data; aligning and caching them according to the time axis; calculating reflectivity and performing geometric correction; generating data packets, persistently storing them, and transmitting them back to a ground control center.

Description

Technical Field

[0001] Multiple embodiments of this specification relate to the field of remote sensing technology, and more particularly to an FPGA-based unmanned aerial vehicle (UAV) hyperspectral acquisition system and method. Background Art

[0002] Hyperspectral imaging technology, with its high spatial resolution and rich spectral information, plays a key role in numerous fields, including precision remote sensing and environmental monitoring. To enable rapid, high-precision monitoring of large target areas, unmanned aerial vehicle (UAV)-mounted hyperspectral data acquisition systems have attracted considerable attention. Their flexibility, ease of deployment, and significant practical advantages are evident. However, existing systems present numerous challenges.

[0003] Currently, most systems have low sensor integration, with hyperspectral cameras, visible light cameras, and other sensors operating independently and lacking coordinated control. In complex monitoring scenarios, data from various sensors cannot be properly aligned or verified, impacting data quality and monitoring accuracy. Furthermore, the sampling rates and data volumes of various sensors vary significantly during drone flight, making data synchronization difficult. Existing synchronization solutions that rely on software algorithms are limited by processing speed and transmission latency, making it difficult to achieve high-precision synchronization. This can cause misalignment between images and other data, hindering subsequent analysis. Therefore, research into new hyperspectral image acquisition technologies is necessary. Summary of the Invention

[0004] Multiple embodiments of this specification describe an FPGA-based unmanned aerial vehicle (UAV)-mounted hyperspectral acquisition system and method.

[0005] In a first aspect, the embodiments of this specification provide an FPGA-based unmanned aerial vehicle (UAV) hyperspectral acquisition system, comprising an FPGA processing module, a hyperspectral camera module, a visible light camera module, an inertial measurement module, a zenithal light module, and a power module carried by the UAV. The FPGA processing module receives a pulse-per-second signal and positioning data provided by the UAV, and provides a trigger signal to the hyperspectral camera module, the visible light camera module, the inertial measurement module, and the zenithal light module based on the pulse-per-second signal. The acquisition cycle is generated based on the trigger signal.

[0006] During the acquisition cycle, the FPGA processing module performs the following steps:

[0007] Create a temporary storage area based on the preset data packet size;

[0008] Receiving the trigger signal to trigger the hyperspectral camera module, visible light camera module, inertial measurement module and zenith light module to collect hyperspectral image data, visible light image data, flight attitude data and zenith light data;

[0009] Aligning the hyperspectral image data, flight attitude data, zenith light data, and positioning data according to the time axis and caching them according to the visible light image data, and writing the visible light image data into the temporary storage area;

[0010] Calculating reflectivity based on the aligned zenith light data and hyperspectral image data, performing geometric correction on the hyperspectral image data based on the flight attitude data and the positioning data to obtain hyperspectral data, and writing the hyperspectral data into the temporary storage area;

[0011] Generate a data packet according to the temporary storage area, store the data packet in a designated persistent storage device and transmit it back to a designated ground control center, clear the cache and reclaim the temporary storage area.

[0012] In a second aspect, the embodiments of this specification provide a method for collecting hyperspectral data from an unmanned aerial vehicle (UAV) based on an FPGA, comprising the following steps:

[0013] Receive the pulse-per-second signal and positioning data provided by the drone, provide a trigger signal to the hyperspectral camera module, visible light camera module, inertial measurement module and zenith light module according to the pulse-per-second signal, and generate an acquisition cycle according to the trigger signal.

[0014] During the collection cycle, the following steps are performed:

[0015] Create a temporary storage area based on the preset data packet size;

[0016] Receiving the trigger signal to trigger the hyperspectral camera module, visible light camera module, inertial measurement module and zenith light module to collect hyperspectral image data, visible light image data, flight attitude data and zenith light data;

[0017] Aligning the hyperspectral image data, flight attitude data, zenith light data, and positioning data according to the time axis and caching them according to the visible light image data, and writing the visible light image data into the temporary storage area;

[0018] Calculating reflectivity based on the aligned zenith light data and hyperspectral image data, performing geometric correction on the hyperspectral image data based on the flight attitude data and the positioning data to obtain hyperspectral data, and writing the hyperspectral data into the temporary storage area;

[0019] Generate a data packet according to the temporary storage area, store the data packet in a designated persistent storage device and transmit it back to a designated ground control center, clear the cache and reclaim the temporary storage area.

[0020] In a third aspect, embodiments of this specification provide an electronic device, including a processor and a memory;

[0021] The processor is connected to the memory;

[0022] The memory is used to store executable program code;

[0023] The processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to execute the method described in any one of the above aspects.

[0024] In a fourth aspect, an embodiment of this specification provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in any one of the above aspects is implemented.

[0025] In a fifth aspect, embodiments of this specification provide a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0026] The beneficial effects of the technical solutions provided by some embodiments of this specification include at least:

[0027] In multiple embodiments of this specification, the provided hyperspectral acquisition system and method receives the pulse-per-second signal and positioning data provided by the drone through the FPGA processing module, and provides a trigger signal to each sensor, thereby achieving high-precision data synchronization. This overcomes the data synchronization difficulties caused by differences in sensor sampling rates in existing systems, and ensures accurate alignment of images and other data. Data acquisition and processing are performed on the FPGA, and after data acquisition is completed, hyperspectral data processing is completed synchronously, which can achieve fast hyperspectral data acquisition without the need for ground processing. Hyperspectral image distortion caused by changes in the drone's flight attitude is corrected, thereby improving the spatial accuracy and data quality of the image. By analyzing the inter-frame difference values ​​to distinguish between key frames and non-key frames, and selecting a suitable interpolation algorithm or simplifying the processing flow accordingly, the relationship between system performance and resource consumption is effectively balanced, and the system burden is reduced while ensuring data processing quality.

[0028] Other features and advantages of the various embodiments of this specification will be further disclosed in the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 This is a schematic diagram of the structure of the drone-mounted hyperspectral acquisition system provided in the embodiments of this specification.

[0031] Figure 2 This is a schematic diagram of the signal relationship of the drone-mounted hyperspectral acquisition system provided in the embodiments of this specification.

[0032] Figure 3 This is a schematic diagram of the data collection path provided in the embodiments of this specification.

[0033] Figure 4 This is a flow chart of the method for calculating reflectivity provided in the embodiments of this specification.

[0034] Figure 5 This is a flow chart of the geometric correction method provided in the embodiments of this specification.

[0035] Figure 6 This is a schematic diagram of the collection cycle provided in the embodiments of this specification.

[0036] Figure 7 This is a flow chart of the drone-mounted hyperspectral acquisition method provided in the embodiments of this specification. DETAILED DESCRIPTION

[0037] The following is an explanation and description of the technical solutions of the embodiments of this specification in conjunction with the drawings of the embodiments of this specification. However, the following embodiments are only preferred embodiments of this specification and are not exhaustive. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without making any creative work are all within the scope of protection of this specification.

[0038] Throughout this specification, the claims, and the accompanying drawings, the terms "first," "second," "third," and the like are used to distinguish between different items, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may include other steps or elements inherent to the process, method, product, or apparatus.

[0039] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, they should not be understood as limitations on this specification.

[0040] The data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection of relevant data complies with the relevant laws, regulations and standards of relevant countries and regions.

[0041] Before introducing the technical solution recorded in this specification, the application scenarios of the technical solution and related technologies are introduced.

[0042] Hyperspectral images are image data containing continuous and dense spectral information, with each pixel covering the spectral response of dozens to hundreds of narrow wavelength bands. Compared to multispectral images, hyperspectral images have higher spectral resolution and can capture subtle spectral characteristics of targets. They are widely used in environmental monitoring, precision agriculture, geological exploration, military reconnaissance, and other fields. Hyperspectral images can capture changes in reflectivity of objects within a very narrow wavelength range, making it more accurate to identify and differentiate different materials. Hyperspectral images are typically stored in the form of a three-dimensional data cube, with two dimensions representing spatial coordinates (i.e., location within the image) and the third dimension representing the spectral band. This structure allows for detailed spectral analysis of each pixel. While providing a wealth of information, hyperspectral data processing and analysis is relatively complex. Advanced algorithms and techniques are required to process the massive amount of data and extract useful information.

[0043] Hyperspectral data is large in volume, high in dimension, and has strong noise. The drone 20 is also affected by light and atmospheric disturbances during flight, resulting in data distortion errors. Existing hyperspectral data acquisition systems only perform simple processing and are unable to eliminate the impact, resulting in cumbersome subsequent data processing, feature extraction, and other tasks. This increases operation and maintenance costs and processing cycles, affecting real-time decision-making. In complex flight environments, software synchronization methods cannot meet high-precision and real-time requirements due to system calculations and network transmission delays, making them difficult to adapt to complex tasks. In addition, when performing missions, the drone 20 faces environmental challenges such as temperature, air pressure, and extreme weather, which threaten flight safety and mission continuity, and also affect image acquisition and data processing accuracy.

[0044] A Field-Programmable Gate Array (FPGA) is an integrated circuit that can be configured by the user through software tools to implement specific logic functions. The core advantage of FPGAs lies in their high flexibility and parallel processing capabilities, making them ideal for high-speed data processing, real-time signal processing, hardware acceleration, and prototyping. Leveraging the FPGA's highly parallel processing capabilities, rapid acquisition and preliminary processing of hyperspectral image data are achieved. Combined with attitude information provided by the inertial measurement unit (IMU), hyperspectral image distortion caused by the drone's 20 flight attitude changes is corrected to ensure spatial accuracy. Simultaneously collected zenithal light data is used to accurately calculate the reflectivity of ground objects, mitigating the impact of varying lighting conditions. At the ground control center (10), machine learning algorithms analyze the hyperspectral images to automatically identify and classify ground objects or phenomena.

[0045] Please see the attached Figure 1 Figure 2 is a schematic diagram of the hyperspectral data acquisition system onboard a drone 20 used in this application. The system includes an FPGA processing module 31, a hyperspectral camera module 34, a visible light camera module 35, an inertial measurement module 33, a zenithal light module 32, and a power module 36, all carried by the drone 20. The drone 20 has a wireless communication connection with the ground control center 10. The red lines in the figure represent the power supply relationship. The power module 36 is connected to the drone 20's battery, and the drone 20 provides sufficient power. The power module 36 also powers the remaining modules. Voltage regulation and other protection features in the power module 36 are implemented using technologies known in the art. Data exchange is established between the drone 20 and the FPGA processing module. Thus, the FPGA processing module 31 can communicate with the ground control center 10 via the drone 20. The blue lines in the figure represent data transmission relationships. The FPGA processing module 31 has a bidirectional data exchange connection with the zenithal light module 32, the inertial measurement module 33, the hyperspectral camera module 34, and the visible light camera module 35. The black lines in the figure represent control relationships. The zenith light module 32 , the inertial measurement module 33 , the hyperspectral camera module 34 , and the visible light camera module 35 are all controlled by the FPGA processing module 31 .

[0046] The FPGA processing module 31 receives the pulse-per-second signal and positioning data provided by the drone 20, multiplies or divides the pulse-per-second signal according to a preset frequency, and provides a trigger signal to the hyperspectral camera module 34, the visible light camera module 35, the inertial measurement module 33, and the zenith light module 32. Figure 2 The drone 20 provides a 1 Hz PPS pulse-per-second signal as a reference signal to the FPGA processing module 31. After frequency division or multiplication by the FPGA processing module 31, 50 Hz and 1 / 3 Hz trigger signals are generated. The 50 Hz trigger signal is sent to the zenith light module 32, the inertial measurement module 33, and the hyperspectral camera module 34, and the 1 / 3 Hz trigger signal is sent to the visible light camera module 35. The zenith light module 32, the inertial measurement module 33, the hyperspectral camera module 34, and the visible light camera module 35 collect data based on the received trigger signals and return their respective collected data to the FPGA processing module 31. The hyperspectral camera module 34 collects hyperspectral data based on the trigger signal, the visible light camera module 35 collects visible light image data based on the trigger signal, the inertial measurement module 33 collects flight attitude data based on the trigger signal, and the zenith light module 32 collects zenith light data based on the trigger signal.

[0047] The FPGA processing module 31 then aligns the hyperspectral data, visible light image data, flight attitude data, zenithal light data, and positioning data along the time axis. It calculates reflectivity based on the aligned zenithal light and hyperspectral data, and geometrically corrects the hyperspectral data based on the flight attitude and positioning data. Based on a preset reporting cycle, the FPGA processing module 31 stores the visible light image data, flight attitude data, zenithal light data, and corrected hyperspectral data in a designated persistent storage device and transmits them to the designated ground control center 10.

[0048] During a collection cycle, the FPGA processing module 31 stores the corrected hyperspectral data, visible light image data, flight attitude data, zenithal light data, and positioning data, aligned along the time axis. When the reporting cycle arrives, the stored corrected hyperspectral data, visible light image data, flight attitude data, zenithal light data, and positioning data are transmitted via the drone 20 to the ground control center 10. Alternatively, after the drone 20 lands, the data can be directly provided to the ground control center 10 by removing the storage device.

[0049] On the other hand, in another embodiment, to prevent data transmission by the FPGA processing module 31 from consuming excessive processing resources of the FPGA processing module 31, the FPGA processing module 31 caches the visible light image data, flight attitude data, zenithal light data, and corrected hyperspectral data in a designated temporary memory. The data cached in the temporary memory is then transmitted to a designated persistent storage device according to a preset period. The data cached in the temporary memory is then transmitted to a designated ground control center 10 via the drone 20. During this process, a corresponding buffer is allocated in the temporary memory based on the size of the visible light image data, flight attitude data, and zenithal light data according to a preset reporting period. A corresponding buffer is allocated in the temporary memory based on a preset multiple of the size of the hyperspectral data. First, the size of the visible light image data, flight attitude data, and zenithal light data in a single acquisition cycle is obtained. Then, the number of acquisition cycles included in the preset period is calculated and multiplied to obtain the size of the buffer to be allocated for the visible light image data, flight attitude data, and zenithal light data.

[0050] In the same way, the size that hyperspectral data directly needs to occupy can be calculated. However, hyperspectral data not only requires caching, but also needs to be calculated and eventually updated. Therefore, as a recommendation, a buffer space three times the size directly occupied by the hyperspectral data distributor should be used. Alternatively, the implementer can set the required buffer size. Temporary memory has a higher access speed, so it takes up less time for the FPGA processing module 31. Then, other modules that interact with temporary memory can retrieve data from the temporary memory on their own, without the intervention of the FPGA processing module 31. After the data transmission is completed, the FPGA processing module 31 clears the data in the corresponding buffer.

[0051] Please see the attached Figure 3 , which is a schematic diagram of the data path for each data collection.

[0052] The data path of zenith light data is as follows: zenith light related signals are collected using a CMOS sensor, the analog signals are converted into digital signals by an ADC, and then enter the FIFO storage device for buffering, and then the zenith light data is output.

[0053] Data path of hyperspectral data: The corresponding CMOS also collects the hyperspectral related signals, directly enters the FIFO storage device cache, and then performs LVDS decoding to obtain the original data, that is, the hyperspectral data.

[0054] Data path of visible light data: CMOS collects visible light-related signals, first enters the FIFO storage device cache, and then performs MIPI decoding to extract valid data, thereby obtaining visible light data.

[0055] The data path for flight attitude data (i.e., IMU data): The drone 20's positioning data is transmitted to the FPGA processing module 31. The inertial measurement module 33 stores the collected drone 20 flight attitude data in a FIFO storage device buffer. This data is then combined with the drone 20's positioning data obtained by the FPGA processing module 31 (this positioning data requires interpolation to obtain time-aligned positioning data). The resulting flight attitude and positioning data are then packaged. The FPGA processing module 31 calculates reflectivity based on the aligned zenithal light data and hyperspectral data, and performs geometric correction on the hyperspectral data based on the flight attitude and positioning data.

[0056] Please see the attached Figure 4 When calculating the reflectance based on the aligned zenith light data and hyperspectral data, the FPGA processing module 31 performs the following steps:

[0057] Step S101) reading the correction coefficient η obtained by measurement under laboratory conditions;

[0058] Step S102) The reflectivity is calculated using the following formula:

[0059] d=DN4 / (DN3×η)

[0060] Among them, d is the reflectance, DN4 is the hyperspectral data, and DN3 is the corresponding zenith light data.

[0061] The method for obtaining the correction coefficient η by measuring under laboratory conditions includes: measuring zenith light data DN1 and hyperspectral measurement data DN2 of a white board with known reflectivity d1 under the same light source in the laboratory, and the correction coefficient η=DN2 / (DN1×d1).

[0062] Please see the attached Figure 5 When geometrically correcting the hyperspectral data according to the flight attitude data, the FPGA processing module 31 performs the following steps:

[0063] For each frame of the hyperspectral data in the correction period and the overlap period, obtaining POS data of the frame, wherein the POS data is obtained according to the positioning data and the flight attitude data, and dividing the frame into a key frame and a non-key frame according to the hyperspectral data and the POS data;

[0064] When it is a key frame, geometric correction is performed on the frame according to the flight attitude data, and geometric correction parameters are stored;

[0065] When it is a non-key frame, calculating an intra-frame difference value of the hyperspectral data of the non-key frame;

[0066] When the intra-frame difference value is greater than a preset reference threshold, geometrically correcting the frame according to the flight attitude data;

[0067] When the intra-frame difference value is not greater than a preset reference threshold, geometric correction is performed using geometric correction parameters of the latest key frame.

[0068] When using the geometric correction parameters of the most recent keyframe for geometric correction, linear interpolation is used to correct the parameters. This includes reading the POS data and geometric correction parameters of the two most recent keyframes, along with the POS data of the current frame. Based on the POS data and geometric correction parameters of the two keyframes and the POS data of the current frame, linear interpolation is used to generate the geometric correction parameters for the current frame.

[0069] The method of dividing frames into key frames and non-key frames according to the hyperspectral data and POS data includes:

[0070] Calculating an inter-frame difference value of the hyperspectral data between the frame and a previous frame;

[0071] When the inter-frame difference value is greater than a preset inter-frame threshold, the frame is determined to be a key frame;

[0072] When the inter-frame difference value is not greater than a preset inter-frame threshold, the frame is determined to be a non-key frame.

[0073] If the calculated inter-frame difference value is greater than the preset inter-frame threshold, then the frame is considered to contain sufficient new information or changes, and is therefore marked as a keyframe. Conversely, if the inter-frame difference value is not greater than the preset inter-frame threshold, it means that the frame has not changed significantly compared to the previous frame, and is therefore marked as a non-keyframe. It is possible to effectively distinguish which frames contain important information (keyframes) and require detailed processing and analysis; and which frames can be regarded as redundant or secondary information (non-keyframes), which may require less processing resources or use previous keyframe information for approximate processing.

[0074] The method of geometrically correcting the frame according to the flight attitude data includes:

[0075] Step S201) reads the hyperspectral data according to a preset calibration period and a coincidence period, wherein the calibration period is consistent with the reporting period of the visible light image data. Figure 6 The correction period is set to 3 seconds and the overlap period is set to 1 second. Setting the overlap period eliminates the splicing artifacts of the 3-second correction period. Hyperspectral data, flight attitude data, zenithal light data, and positioning data are sampled at 50 Hz and aligned based on the trigger signal of the visible light image data. The trigger signal for the visible light image data is 1 / 3 Hz, occurring every 3 seconds. Alignment is performed based on the rising edge of this 3-second trigger signal to determine the start time of the 50 Hz trigger signal for the hyperspectral data, flight attitude data, zenithal light data, and positioning data. A 3-second acquisition cycle forms a complete data packet. Data alignment, reflectivity calculation, and geometric correction are performed in a 3-second cycle.

[0076] Step S202) obtaining POS data of the frame, wherein the POS data is obtained based on the positioning data and the flight attitude data, and the object plane position of each pixel is obtained based on the POS data;

[0077] Step S203) Obtain the corresponding image space position according to the object plane position, and obtain geometric correction parameters.

[0078] The method of obtaining the object plane position of each pixel according to the POS data includes:

[0079] For each frame of the hyperspectral data in the correction period and the overlap period, obtain POS data of the frame, wherein the POS data includes (xt, yt, zt, θt, φt, ψt), where xt, yt, zt represent the longitude, latitude, and altitude of the position, respectively, and θt, φt, ψt represent the pitch angle, roll angle, and yaw angle of the flight attitude of the UAV 20, respectively;

[0080] Calculate the rotation matrix R according to the flight attitude (θt, φt, ψt);

[0081] For each pixel (u,v) in the image, calculate its corresponding object plane coordinates (X,Y,Z) using the collinearity equation.

[0082] The collinear equation is:

[0083]

[0084] Where (X, Y, Z) are the coordinates of the target point on the object plane, (Xc, Yc, Zc) is the center position of the camera (obtained by superimposing the positioning data in the POS data and the relative position deviation between the positioning point and the camera center position), Rij is the element of the rotation matrix, which is calculated from the attitude angles (θt, ϕt, ψt), and (u, v) is the coordinate of the pixel in the image.

[0085] The calculation process of the rotation matrix is:

[0086] The rotation about each individual axis can be represented by a rotation matrix:

[0087] Rotation around the Z-axis (i.e., yaw angle ψ)

[0088]

[0089] Rotation around the Y-axis (pitch angle θ)

[0090]

[0091] Rotation around the X-axis (roll angle ϕ)

[0092]

[0093] Multiply the three basic rotation matrices in ZYX order to get the final rotation matrix:

[0094]

[0095] Expanded into matrix form:

[0096]

[0097] Thus, the value of Rij can be obtained, i,j∈[1,3].

[0098] When the image space position is empty, the pixel is filled into the image space position. When the image space position is not empty, the pixel corresponding to the pixel with a smaller weighted sum of the pitch angle and roll angle is filled in. First, a new blank image space is created, and the corrected pixel is filled into the corresponding image space position. The image space is actually a two-dimensional image. Hyperspectral data includes a two-dimensional image and more dimensions for recording reflectivity. The reflectivity is associated with a pixel in the two-dimensional image. When the pixel value corresponding to the image space is empty, the corrected pixel can be directly filled in. When the pixel in the image space is not empty, it means that the object plane position corresponding to the pixel position has been photographed at other times due to the swing of the drone 20. That is, repeated shooting occurs. At this time, the pixel with a smaller weighted sum of the pitch angle and roll angle is selected to fill in, that is, the pixel with the drone 20 closest to the ideal flight posture is used to fill in.

[0099] When the image space position exceeds the boundaries of the current frame, the pixel is cached until the next calibration cycle. If the image space position exceeds the boundaries of the current frame, it indicates that the captured object plane is at a location corresponding to another frame, so it is cached until the next calibration cycle. If the captured object plane is at a location corresponding to a previous frame, the pixel is eventually discarded.

[0100] At the end of the calibration cycle, the image corresponding to that calibration cycle is checked for any empty pixels. If so, the hyperspectral data from the calibration and overlap periods are interpolated to generate the pixel value at that location. Empty pixels indicate missed capture, so interpolation is used to compensate.

[0101] The method of interpolating and generating the pixel value of the empty pixel using the hyperspectral data within the correction period and the overlap period includes:

[0102] At least three interpolation algorithms are preset, which are divided into high-precision interpolation algorithm, medium-precision interpolation algorithm and low-precision interpolation algorithm according to the accuracy of the interpolation algorithm;

[0103] When it is a key frame, a high-precision interpolation algorithm is pre-selected to generate the pixel value of the empty pixel; when it is a non-key frame, a low-precision interpolation algorithm is pre-selected to generate the pixel value of the empty pixel;

[0104] Obtaining the difference between the time when the interpolation of the previous frame is completed and the end time of the acquisition period, and obtaining the system load according to the difference;

[0105] When the system load is higher than a preset reference load, the accuracy of the interpolation algorithm used is reduced on a preselected basis.

[0106] Exemplarily, the nearest neighbor interpolation algorithm, bilinear interpolation algorithm and bicubic interpolation algorithm are used. The accuracy of the bicubic interpolation algorithm is high precision, which can achieve interpolation of areas with clear edges and smooth transitions. The bilinear interpolation algorithm is a medium-precision interpolation algorithm, which is suitable for ordinary image areas and areas with high real-time requirements. The nearest neighbor interpolation algorithm is a low-precision interpolation algorithm, which is suitable for periods of tight FPGA resources, areas with smooth images, and periods of temporary overload. The interpolation algorithm can use a modular IP core to reduce the difficulty of FPGA programming.

[0107] On the other hand, this manual provides a 20-carry hyperspectral acquisition method for UAVs based on FPGA. Figure 7 , including the steps of:

[0108] Step S301) receiving the pulse-per-second signal and positioning data provided by the drone 20, multiplying or dividing the pulse-per-second signal according to a preset frequency, and providing a trigger signal to the hyperspectral camera module 34, the visible light camera module 35, the inertial measurement module 33, and the zenith light module 32;

[0109] Step S302) aligning the hyperspectral data, visible light image data, flight attitude data, zenith light data and positioning data along the time axis;

[0110] Step S303) Calculating reflectance based on the aligned zenith light data and hyperspectral data;

[0111] Step S304) performing geometric correction on the hyperspectral data according to the flight attitude data and positioning data;

[0112] Step S305 ) According to a preset reporting cycle, the visible light image data, flight attitude data, zenith light data and the corrected hyperspectral data are stored in a designated persistent storage device and transmitted to a designated ground control center 10 .

[0113] The method for geometrically correcting the hyperspectral data according to the flight attitude data includes:

[0114] Reading the hyperspectral data according to a preset correction period and a coincidence period, wherein the correction period matches a reporting period of the visible light image data;

[0115] For each frame of the hyperspectral data in the correction period and the overlap period, obtaining POS data of the frame, obtaining the object plane position of each pixel according to the POS data, and obtaining the corresponding image space position according to the object plane position;

[0116] When the image space position is empty, fill the image space position with the pixel; when the image space position is not empty, fill the position with a pixel corresponding to a smaller weighted sum of the pitch angle and the roll angle;

[0117] When the image space position exceeds the boundary of the current frame, caching the pixel until the next correction cycle;

[0118] At the end of the calibration period, it is determined whether the image corresponding to the calibration period has any empty pixels. If an empty pixel exists, the hyperspectral data within the calibration period and the overlap period are used to interpolate and generate the pixel value at the position.

[0119] On the other hand, the method for obtaining the object plane position of each pixel according to the POS data includes:

[0120] For each frame of the hyperspectral data in the correction period and the overlap period, obtain POS data of the frame, wherein the POS data includes (xt, yt, zt, θt, φt, ψt), where xt, yt, zt represent the longitude, latitude, and altitude of the position, respectively, and θt, φt, ψt represent the pitch angle, roll angle, and yaw angle of the flight attitude of the UAV 20, respectively;

[0121] Calculate the rotation matrix R according to the flight attitude (θt, φt, ψt);

[0122] For each pixel (u,v) in the image, calculate its corresponding object plane coordinates (X,Y,Z) using the collinearity equation.

[0123] On the other hand, according to a preset reporting cycle, the FPGA processing module 31 stores the visible light image data, flight attitude data, zenith light data and the corrected hyperspectral data in a designated persistent storage device and transmits them to the designated ground control center 10. The method includes:

[0124] storing the visible light image data, flight attitude data, zenith light data, and the corrected hyperspectral data in a designated high-speed cache according to the trigger signal;

[0125] When the preset reporting period is reached, the visible light image data, flight attitude data, zenith light data and the corrected hyperspectral data are stored from the designated cache to a designated persistent storage device;

[0126] Then, the visible light image data, flight attitude data, zenith light data and the corrected hyperspectral data are transmitted from the designated cache to the designated ground control center 10 via the UAV 20;

[0127] The visible light image data, flight attitude data, zenith light data and corrected hyperspectral data corresponding to the reporting period are cleared from a designated high-speed cache.

[0128] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Without departing from the design spirit of this specification, various modifications and improvements made to the technical solutions of this specification by ordinary technicians in this field should fall within the scope of protection determined by the claims of this specification.

Claims

1. An FPGA-based unmanned aerial vehicle hyperspectral acquisition system, characterized in that: The system includes an FPGA processing module, a hyperspectral camera module, a visible light camera module, an inertial measurement module, a zenithal light module, and a power supply module carried by the drone. The FPGA processing module receives the pulse-per-second signal and positioning data provided by the drone, and provides a trigger signal to the hyperspectral camera module, the visible light camera module, the inertial measurement module, and the zenithal light module according to the pulse-per-second signal. The acquisition cycle is generated according to the trigger signal. During the acquisition cycle, the FPGA processing module performs the following steps: Create a temporary storage area based on the preset data packet size; Receiving the trigger signal to trigger the hyperspectral camera module, visible light camera module, inertial measurement module and zenith light module to collect hyperspectral image data, visible light image data, flight attitude data and zenith light data; Aligning the hyperspectral image data, flight attitude data, zenith light data, and positioning data according to the time axis and caching them according to the visible light image data, and writing the visible light image data into the temporary storage area; Calculate reflectance based on aligned zenith light data and hyperspectral image data, Performing geometric correction on the hyperspectral image data according to the flight attitude data and the positioning data to obtain hyperspectral data, including obtaining POS data of each frame of the hyperspectral data in the correction period and the overlap period, dividing the frame into a key frame and a non-key frame according to the hyperspectral data and the POS data, and calculating an inter-frame difference value of the hyperspectral data of the frame and the previous frame; When the inter-frame difference value is greater than a preset inter-frame threshold, the frame is judged to be a key frame; when the inter-frame difference value is not greater than the preset inter-frame threshold, the frame is judged to be a non-key frame; when it is a key frame, the frame is geometrically corrected according to the flight attitude data, and the geometric correction parameters are stored; when it is a non-key frame, the intra-frame difference value of the hyperspectral data of the non-key frame is calculated; when the intra-frame difference value is greater than a preset reference threshold, the frame is geometrically corrected according to the flight attitude data; when the intra-frame difference value is not greater than the preset reference threshold, the frame is geometrically corrected using the geometric correction parameters of the nearest key frame; Writing the hyperspectral data into the temporary storage area; Generate a data packet according to the temporary storage area, store the data packet in a designated persistent storage device and transmit it back to a designated ground control center, clear the cache and reclaim the temporary storage area.

2. The FPGA-based unmanned aerial vehicle hyperspectral acquisition system according to claim 1, characterized in that: When calculating reflectance based on the aligned zenith light data and hyperspectral data, the FPGA processing module performs the following steps: Read the correction factor obtained by measurement under laboratory conditions ; The reflectivity is calculated by the following formula: <h2 style=";text-align:left;direction:ltr">d=DN4 / (DN3×<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"> ) Among them, d is the reflectance, DN4 is the hyperspectral data, and DN3 is the corresponding zenith light data; Among them, the correction factor obtained by measurement under laboratory conditions is The methods include: The zenith light data DN1 collected by the zenith light module and the hyperspectral measurement data DN2 of a whiteboard with a known reflectivity of d1 collected by the hyperspectral camera module are measured under the same light source in the laboratory. The correction coefficient =DN2 / (DN1×d1).

3. The FPGA-based unmanned aerial vehicle hyperspectral acquisition system according to claim 1 or 2, characterized in that: When geometric correction is performed on the hyperspectral data according to the flight attitude data, the FPGA processing module performs the following steps: The method for geometrically correcting the frame according to the flight attitude data comprises: Reading the hyperspectral data according to a preset correction period and a coincidence period, wherein the correction period is consistent with a reporting period of the visible light image data; Obtaining an object plane position of each pixel according to the POS data, wherein the POS data is obtained according to the positioning data and the flight attitude data; Obtaining a corresponding image space position according to the object plane position and obtaining geometric correction parameters; wherein, when obtaining the corresponding image space position according to the object plane position, when the image space position is empty, the pixel is filled into the image space position; and when the image space position is not empty, a pixel corresponding to a pixel having a smaller weighted sum of a pitch angle and a roll angle is filled into the position; When the image space position exceeds the boundary of the current frame, caching the pixel until the next correction cycle; At the end of the calibration period, it is determined whether the image corresponding to the calibration period has any empty pixels. If an empty pixel exists, the hyperspectral data within the calibration period and the overlap period are used to interpolate and generate the pixel value of the empty pixel.

4. The FPGA-based unmanned aerial vehicle hyperspectral acquisition system according to claim 3, characterized in that: The method for obtaining the object plane position of each pixel according to the POS data includes: The POS data includes (xt, yt, zt, θt, φt, ψt), where xt, yt, zt represent the longitude, latitude and altitude of the position respectively, and θt, φt, ψt represent the pitch angle, roll angle and yaw angle of the UAV flight attitude respectively; Calculate the rotation matrix R according to the flight attitude (θt, φt, ψt); For each pixel (u,v) in the image, calculate its corresponding object plane coordinates (X,Y,Z) using the collinearity equation.

5. The FPGA-based unmanned aerial vehicle hyperspectral acquisition system according to claim 4, characterized in that: The method for interpolating and generating a pixel value at the position using the hyperspectral data within the correction period and the overlap period includes: Obtaining POS data of the frame where the empty pixel is located, and obtaining the object plane position of the empty pixel, which is recorded as the object plane empty position; Obtain pixel values ​​corresponding to positions around the empty position on the object plane, and record them as surrounding pixel values; The pixel value of the empty pixel is obtained by interpolation based on the surrounding pixel values.

6. The FPGA-based unmanned aerial vehicle hyperspectral acquisition system according to claim 3, characterized in that: The method of interpolating and generating the pixel value of the empty pixel using the hyperspectral data within the correction period and the overlap period includes: At least three interpolation algorithms are preset, which are divided into high-precision interpolation algorithm, medium-precision interpolation algorithm and low-precision interpolation algorithm according to the accuracy of the interpolation algorithm; When it is a key frame, a high-precision interpolation algorithm is pre-selected to generate the pixel value of the empty pixel; when it is a non-key frame, a low-precision interpolation algorithm is pre-selected to generate the pixel value of the empty pixel; Obtaining the difference between the time when the interpolation of the previous frame is completed and the end time of the acquisition period, and obtaining the system load according to the difference; When the system load is higher than a preset reference load, the accuracy of the interpolation algorithm used is reduced on a preselected basis.

7. A method for collecting hyperspectral data on an unmanned aerial vehicle based on FPGA, characterized in that: Including steps: Receive the pulse-per-second signal and positioning data provided by the drone, provide a trigger signal to the hyperspectral camera module, visible light camera module, inertial measurement module and zenith light module according to the pulse-per-second signal, and generate an acquisition cycle according to the trigger signal. During the collection cycle, the following steps are performed: Create a temporary storage area based on the preset data packet size; Receiving the trigger signal to trigger the hyperspectral camera module, visible light camera module, inertial measurement module and zenith light module to collect hyperspectral image data, visible light image data, flight attitude data and zenith light data; Aligning the hyperspectral image data, flight attitude data, zenith light data, and positioning data according to the time axis and caching them according to the visible light image data, and writing the visible light image data into the temporary storage area; Calculating reflectivity based on the aligned zenithal light data and hyperspectral image data, performing geometric correction on the hyperspectral image data based on the flight attitude data and the positioning data to obtain hyperspectral data, including obtaining POS data of each frame of the hyperspectral data in the correction period and the overlap period, dividing the frame into a key frame and a non-key frame based on the hyperspectral data and the POS data, and calculating an inter-frame difference value of the hyperspectral data of the frame and the previous frame; When the inter-frame difference value is greater than a preset inter-frame threshold, the frame is judged to be a key frame; when the inter-frame difference value is not greater than the preset inter-frame threshold, the frame is judged to be a non-key frame; when it is a key frame, the frame is geometrically corrected according to the flight attitude data, and the geometric correction parameters are stored; when it is a non-key frame, the intra-frame difference value of the hyperspectral data of the non-key frame is calculated; when the intra-frame difference value is greater than a preset reference threshold, the frame is geometrically corrected according to the flight attitude data; when the intra-frame difference value is not greater than the preset reference threshold, the frame is geometrically corrected using the geometric correction parameters of the nearest key frame; Writing the hyperspectral data into the temporary storage area; Generate a data packet according to the temporary storage area, store the data packet in a designated persistent storage device and transmit it back to a designated ground control center, clear the cache and reclaim the temporary storage area.

8. An electronic device, characterized in that: including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to execute the method according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to claim 7 is implemented.

Images