FPGA-based unmanned aerial vehicle-mounted hyperspectral acquisition system and method

Abstract

The embodiment of the invention relates to the technical field of remote sensing, in particular to an unmanned aerial vehicle-mounted hyperspectral acquisition system and method based on an FPGA (Field Programmable Gate Array). The system comprises an FPGA (Field Programmable Gate Array) processing module carried by an unmanned aerial vehicle, a hyperspectral camera module, a visible light camera module, an inertia measurement module, a zenith light module and a power supply module, a trigger signal is provided for a hyperspectral camera module, a visible light camera module, an inertia measurement module and a zenith light module according to the second pulse signal, an acquisition period is generated according to the trigger signal, and in the acquisition period, the FPGA processing module executes the following steps: opening up a temporary storage area; receiving hyperspectral image data, visible light image data, flight attitude data and zenith light data; aligning and caching according to a time axis; calculating the reflectivity, and carrying out geometric correction; and generating a data packet, persistently storing the data packet and returning the data packet to the ground control center.

Description

Technical Field

[0001] Multiple embodiments of this specification relate to the field of remote sensing technology, and specifically to an unmanned aerial vehicle (UAV)-borne hyperspectral acquisition system and method based on a field-programmable gate array (FPGA). Background Art

[0002] Hyperspectral imaging technology plays a key role in many fields such as precision remote sensing and environmental monitoring with high spatial resolution and rich spectral information. To achieve rapid and high-precision monitoring of large-area target regions, UAV-borne hyperspectral data acquisition systems have attracted much attention. They are flexible, mobile, and convenient to deploy, with obvious advantages in practical applications. However, there are many problems in existing systems.

[0003] Currently, most systems have low sensor integration. Hyperspectral cameras, visible light cameras, etc. work independently, lacking coordinated control. In complex monitoring scenarios, the data of each sensor cannot be correctly aligned and cannot complement and confirm each other, affecting data quality and monitoring accuracy. At the same time, when the UAV is flying, there are large differences in the sampling rates and data volumes of each sensor, making data synchronization difficult. Existing synchronization schemes relying on software algorithms are affected by processing speed and transmission delay and are difficult to meet the requirements of high-precision synchronization, resulting in misalignment of images and other data and hindering subsequent analysis. Therefore, new hyperspectral image acquisition technologies need to be studied. Summary of the Invention

[0004] Multiple embodiments of this specification describe an unmanned aerial vehicle (UAV)-borne hyperspectral acquisition system and method based on a field-programmable gate array (FPGA).

[0005] In a first aspect, an embodiment of this specification provides an unmanned aerial vehicle (UAV)-borne hyperspectral acquisition system based on a field-programmable gate array (FPGA), including an FPGA processing module, a hyperspectral camera module, a visible light camera module, an inertial measurement module, a zenith light module, and a power supply module carried by the UAV. The FPGA processing module receives a pulse-per-second signal and positioning data provided by the UAV, provides trigger signals to the hyperspectral camera module, the visible light camera module, the inertial measurement module, and the zenith light module according to the pulse-per-second signal, and generates an acquisition cycle according to the trigger signals. During the acquisition cycle, the FPGA processing module performs the following steps: Open a temporary storage area according to a preset data packet size; Receive the high-spectral image data, visible light image data, flight attitude data, and zenith light data collected by triggering the hyperspectral camera module, the visible light camera module, the inertial measurement module, and the zenith light module by the trigger signal; Align and cache the high-spectral image data, flight attitude data, zenith light data, and positioning data along the time axis according to the visible light image data, and write the visible light image data into the temporary storage area; Calculate the reflectance based on the aligned zenith light data and hyperspectral image data, perform geometric correction on the hyperspectral image data according to the flight attitude data and positioning data to obtain hyperspectral data, and write the hyperspectral data into the temporary storage area; Generate a data packet according to the temporary storage area, store the data packet in a specified persistent storage device and transmit it back to a specified ground control center, clear the cache and recycle the temporary storage area.

[0006] In a second aspect, an embodiment of this specification provides a method for collecting airborne hyperspectral data based on an FPGA, including the steps of: Receive the second pulse signal and positioning data provided by the unmanned aerial vehicle, provide trigger signals to the hyperspectral camera module, visible light camera module, inertial measurement module, and zenith light module according to the second pulse signal, and generate an acquisition period according to the trigger signals, During the acquisition period, perform the following steps: Allocate a temporary storage area according to a preset data packet size; Receive the trigger signals to trigger the hyperspectral camera module, visible light camera module, inertial measurement module, and zenith light module to collect hyperspectral image data, visible light image data, flight attitude data, and zenith light data; Align and cache the hyperspectral image data, flight attitude data, zenith light data, and positioning data according to the visible light image data along the time axis, and write the visible light image data into the temporary storage area; Calculate the reflectance based on the aligned zenith light data and hyperspectral image data, perform geometric correction on the hyperspectral image data according to the flight attitude data and positioning data to obtain hyperspectral data, and write the hyperspectral data into the temporary storage area; Generate a data packet according to the temporary storage area, store the data packet in a specified persistent storage device and transmit it back to a specified ground control center, clear the cache and recycle the temporary storage area.

[0007] In a third aspect, an embodiment of this specification provides an electronic device, including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory to execute the method described in any of the above aspects.

[0008] In a fourth aspect, an embodiment of this specification provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any of the above aspects is implemented.

[0009] In a fifth aspect, embodiments of the present specification provide a computer program product, including a computer program, which, when executed by a processor, implements the method described in any of the above aspects.

[0010] The beneficial effects brought by the technical solutions provided in some embodiments of the present specification at least include: In multiple embodiments of the present specification, the provided hyperspectral acquisition system and method receive the pulse per second signal and positioning data provided by the UAV through the FPGA processing module, and provide trigger signals to each sensor, achieving high-precision data synchronization. This overcomes the problem of difficult data synchronization caused by differences in sensor sampling rates in existing systems, ensuring the precise alignment of images and other data. Data is collected and processed on the FPGA. After data collection is completed, hyperspectral data processing is completed synchronously, enabling fast hyperspectral data acquisition without the need for further processing on the ground. Correction is performed on the distortion of hyperspectral images caused by changes in the UAV flight attitude, improving the spatial accuracy and data quality of the images. By analyzing the inter-frame difference values to distinguish key frames from non-key frames, and accordingly selecting appropriate interpolation algorithms or simplifying the processing flow, the relationship between system performance and resource consumption is effectively balanced, reducing the system burden while ensuring the quality of data processing.

[0011] Other features and advantages of multiple embodiments of the present specification will be further revealed in the following specific embodiments and drawings. Description of the Drawings

[0012] To more clearly illustrate the technical solutions in the embodiments of the present specification, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present specification. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0013] Figure 1 It is a schematic structural diagram of an airborne hyperspectral acquisition system provided by an embodiment of the present specification.

[0014] Figure 2 It is a schematic diagram of the signal relationship of an airborne hyperspectral acquisition system provided by an embodiment of the present specification.

[0015] Figure 3 It is a schematic diagram of the data path for data acquisition provided by an embodiment of the present specification.

[0016] Figure 4 It is a schematic diagram of the process flow for calculating the reflectivity provided by an embodiment of the present specification.

[0017] Figure 5 It is a schematic diagram of the process flow for geometric correction provided by an embodiment of the present specification.

[0018] Figure 6 This is a schematic diagram of the acquisition period provided by the embodiments of this specification.

[0019] Figure 7 This is a schematic diagram of the process flow of the airborne hyperspectral acquisition method provided by the embodiments of this specification. Specific implementation manners

[0020] The technical solutions of the embodiments of this specification will be explained and described below with reference to the accompanying drawings of the embodiments of this specification. However, the following embodiments are only the preferred embodiments of this specification and not all of them. Based on the embodiments in the implementation manners, other embodiments obtained by those skilled in the art without creative efforts all fall within the protection scope of this specification.

[0021] Terms such as "first", "second", "third", etc. in the specification, claims and the above-mentioned drawings of this specification are used to distinguish different objects rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0022] In the following description, terms indicating orientation or positional relationship such as "inner", "outer", "upper", "lower", "left", "right", etc. are only for the convenience of describing the embodiments and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of this specification.

[0023] The data involved in this application are all information and data authorized by users or fully authorized by all parties, and the acquisition of relevant data complies with the relevant laws, regulations and standards of relevant countries and regions.

[0024] Before introducing the technical solutions described in this specification, the application scenarios of the technical solutions and related technologies will be introduced.

[0025] A hyperspectral image is a type of image data that contains continuous and dense spectral information, where each pixel can cover the spectral responses of dozens to hundreds of narrow bands. Compared with multispectral images, hyperspectral images have higher spectral resolution, can capture the subtle spectral features of targets, and are widely used in environmental monitoring, precision agriculture, geological exploration, military reconnaissance and other fields. Hyperspectral images can capture the reflectance changes of ground objects within a very narrow wavelength range, making it more accurate to identify and distinguish different materials. Hyperspectral images usually exist in the form of a three-dimensional data cube, where two dimensions represent spatial coordinates (i.e., positions in the image), and the third dimension represents spectral bands. This structure allows for detailed spectral analysis of each pixel. Although it provides a large amount of information, the processing and analysis of hyperspectral data are relatively complex. Advanced algorithms and technologies are needed to process the massive data and extract useful information.

[0026] Hyperspectral data is large in volume, high in dimension, and has strong noise. The drone 20 is also interfered by light and atmospheric disturbances during flight, and the data has distortion errors. Existing hyperspectral data acquisition systems only perform simple processing and cannot eliminate the impacts, resulting in cumbersome subsequent data processing, feature extraction, etc. It increases the operation and maintenance costs and processing cycle, and affects real-time decision-making. The software synchronization method cannot meet the high-precision and real-time requirements and is difficult to adapt to complex tasks due to system operation, network transmission delays, etc. in a complex flight environment. In addition, the drone 20 faces environmental challenges such as temperature, air pressure, and extreme weather when performing tasks, threatening flight safety and mission continuity, and also affecting the accuracy of image acquisition and data processing.

[0027] FPGA (Field-Programmable Gate Array) is an integrated circuit that can be configured by users through software tools to achieve specific logic functions. The core advantages of FPGA lie in its high flexibility and parallel processing capabilities, making it very suitable for fields such as high-speed data processing, real-time signal processing, hardware acceleration, and prototyping. Utilize the high parallel processing capabilities of FPGA to achieve fast acquisition and preliminary processing of hyperspectral image data. Combine the attitude information provided by the inertial measurement module 33 (IMU) to correct the distortion of hyperspectral images caused by the attitude changes during the flight of the drone 20, ensuring the spatial accuracy of the images. Calculate the accurate reflectance of ground objects through the synchronously acquired zenith light data to reduce the impacts brought by changes in lighting conditions. Analyze the hyperspectral images with machine learning algorithms at the ground control center 10 to achieve automatic identification and classification of ground objects or phenomena.

[0028] Please refer to the appendix Figure 1, which is a schematic structural diagram of the hyperspectral acquisition system carried by the drone 20 used in this application. It includes an FPGA processing module 31, a hyperspectral camera module 34, a visible light camera module 35, an inertial measurement module 33, a zenith light module 32, and a power module 36 carried by the drone 20. The drone 20 has a wireless communication connection with the ground control center 10. The red lines in the figure represent the power supply relationship. The power module 36 is connected to the battery of the drone 20, and the drone 20 supplies sufficient power. The power module 36 supplies power to the other modules. The voltage stabilization and other protections in the power module 36 are implemented by the technologies already disclosed in the art. The drone 20 has data interaction with the FPAG processing module. Thus, the FPGA processing module 31 can achieve a communication connection with the ground control center 10 through the relay of the drone 20. The blue lines in the figure represent the data transmission relationship, and the FPGA processing module 31 has two-way data interaction connections with the zenith light module 32, the inertial measurement module 33, the hyperspectral camera module 34, and the visible light camera module 35. The black lines in the figure represent the control relationship. The zenith light module 32, the inertial measurement module 33, the hyperspectral camera module 34, and the visible light camera module 35 are all controlled by the FPGA processing module 31.

[0029] The FPGA processing module 31 receives the second pulse signal and positioning data provided by the drone 20, and after frequency multiplying or dividing the second pulse signal according to a preset, provides trigger signals to the hyperspectral camera module 34, the visible light camera module 35, the inertial measurement module 33, and the zenith light module 32. Please refer to the appendix Figure 2 , the drone 20 provides a 1Hz PPS second pulse signal to the FPGA processing module 31 as a reference signal. After frequency dividing or multiplying by the FPGA processing module 31, trigger signals of 50Hz and 1 / 3Hz are generated. The 50Hz trigger signal is sent to the zenith light module 32, the inertial measurement module 33, and the hyperspectral camera module 34, and the 1 / 3Hz trigger signal is sent to the visible light camera module 35. The zenith light module 32, the inertial measurement module 33, the hyperspectral camera module 34, and the visible light camera module 35 collect data according to the received trigger signals and return their respective collected data to the FPGA processing module 31. Among them, the hyperspectral camera module 34 collects hyperspectral data according to the trigger signal, the visible light camera module 35 collects visible light image data according to the trigger signal, the inertial measurement module 33 collects flight attitude data according to the trigger signal, and the zenith light module 32 collects zenith light data according to the trigger signal.

[0030] Subsequently, the FPGA processing module 31 aligns the hyperspectral data, visible light image data, flight attitude data, zenith light data, and positioning data along the time axis, calculates the reflectance based on the aligned zenith light data and hyperspectral data, and performs geometric correction on the hyperspectral data according to the flight attitude data and positioning data. According to the preset reporting period, the FPGA processing module 31 stores the visible light image data, flight attitude data, zenith light data, and the corrected hyperspectral data in a specified persistent storage device and transmits them to the specified ground control center 10.

[0031] Within one acquisition cycle, the FPGA processing module 31 aligns and stores the corrected hyperspectral data, visible light image data, flight attitude data, zenith light data, and positioning data along the time axis. When the reporting period arrives, the stored corrected hyperspectral data, visible light image data, flight attitude data, zenith light data, and positioning data are sent to the ground control center 10 via the UAV 20. Alternatively, when the UAV 20 lands, the data is provided to the ground control center 10 by directly removing the storage device.

[0032] On the other hand, in another embodiment, to avoid the FPGA processing module 31 occupying too much processing resources for data transmission, the FPGA processing module 31 caches the visible light image data, flight attitude data, zenith light data, and the corrected hyperspectral data in a specified temporary memory, and transfers the data cached in the temporary memory to the specified persistent storage device according to a preset period, and transmits the data cached in the temporary memory to the specified ground control center 10 via the UAV 20. During the process, according to the preset reporting period, corresponding buffers are allocated in the temporary memory according to the sizes of the visible light image data, flight attitude data, and zenith light data; according to a preset multiple of the size of the hyperspectral data, a corresponding buffer for the hyperspectral data is allocated in the temporary memory. First, obtain the sizes of the visible light image data, flight attitude data, and zenith light data in one acquisition cycle. Then calculate how many acquisition cycles are included in the preset period and multiply them to obtain the sizes of the buffers that should be allocated to the visible light image data, flight attitude data, and zenith light data.

[0033] In the same way, the size directly occupied by the hyperspectral data can be calculated. However, the hyperspectral data not only needs to be cached, but also needs to be calculated and finally updated. Therefore, as a recommendation, a buffer space three times the size directly occupied by the hyperspectral data allocator should be allocated. Or, the implementer can set the size of the required buffer by themselves. The temporary memory has a high access speed, so it takes less time for the FPGA processing module 31. Then, other modules that interact with the temporary memory can obtain data from the temporary memory by themselves, and at this time, the intervention of the FPGA processing module 31 is not required. After the data transmission is completed, the FPGA processing module 31 can clear the data in the corresponding buffer.

[0034] Please refer to Appendix Figure 3 , which shows the data paths of the data collected by each sensor.

[0035] The data path of the zenith light data is as follows: The CMOS sensor is used to collect the zenith light related signals, which are converted from analog signals to digital signals by the ADC, and then enter the FIFO storage device for caching, and then the zenith light data is output.

[0036] The data path of the hyperspectral data: Similarly, the corresponding CMOS collects the hyperspectral related signals, directly enters the FIFO storage device for caching, and then performs LVDS decoding to obtain the original data, that is, the hyperspectral data.

[0037] The data path of the visible light data: The CMOS collects the visible light related signals, first enters the FIFO storage device for caching, and then performs MIPI decoding to extract the valid data, so as to obtain the visible light data.

[0038] The data path of the flight attitude data (i.e., IMU data): The positioning data of the unmanned aerial vehicle 20 is transmitted to the FPGA processing module 31, and the inertial measurement module 33 stores the flight attitude data of the unmanned aerial vehicle 20 collected into the FIFO storage device for caching, and integrates and packages it with the positioning data of the unmanned aerial vehicle 20 obtained in the FPGA processing module 31 (the positioning data needs to be interpolated to obtain the positioning data aligned with the flight attitude data) to obtain the flight attitude data and the positioning data. The FPGA processing module 31 calculates the reflectivity according to the aligned zenith light data and hyperspectral data, and performs geometric correction on the hyperspectral data according to the flight attitude data and the positioning data.

[0039] Please refer to Appendix Figure 4 , when calculating the reflectivity according to the aligned zenith light data and hyperspectral data, the FPGA processing module 31 performs the following steps: Step S101) Read the calibration coefficient η obtained by measurement under laboratory conditions; Step S102) Calculate the reflectivity through the following formula: d = DN4 / (DN3 × η) Among them, d is the reflectivity, DN4 is the hyperspectral data, and DN3 is the corresponding zenith light data.

[0040] Among them, the method for obtaining the correction coefficient η measured under laboratory conditions includes: measuring the zenith light data DN1 and the hyperspectral measurement data DN2 of a whiteboard with a known reflectivity d1 under the same light source in the laboratory, and the correction coefficient η = DN2 / (DN1 × d1).

[0041] Please refer to the appendix Figure 5 When performing geometric correction on the hyperspectral data according to the flight attitude data, the FPGA processing module 31 performs the following steps: For each frame of the hyperspectral data in the correction period and the coincidence period, obtain the POS data of this frame, which is obtained according to the positioning data and the flight attitude data, and divide the frame into a key frame and a non-key frame according to the hyperspectral data and the POS data; When it is a key frame, perform geometric correction on the frame according to the flight attitude data and store the geometric correction parameters; When it is a non-key frame, calculate the intra-frame difference value of the hyperspectral data of the non-key frame; When the intra-frame difference value is greater than a preset reference threshold, perform geometric correction on the frame according to the flight attitude data; When the intra-frame difference value is not greater than the preset reference threshold, perform geometric correction using the geometric correction parameters of the nearest key frame.

[0042] When performing geometric correction using the geometric correction parameters of the nearest key frame, linear interpolation is used to correct the geometric correction parameters. Specifically, it includes: reading the POS data and geometric correction parameters of the two nearest key frames, and the POS data of the current frame. According to the POS data and geometric correction parameters of the two key frames, and the POS data of the current frame, linear interpolation is used to generate the geometric correction parameters of the current frame.

[0043] Among them, the method for dividing the frame into a key frame and a non-key frame according to the hyperspectral data and the POS data includes: Calculate the inter-frame difference value between the frame and the hyperspectral data of the previous frame; When the inter-frame difference value is greater than a preset inter-frame threshold, determine that this frame is a key frame; When the inter-frame difference value is not greater than the preset inter-frame threshold, determine that this frame is a non-key frame.

[0044] If the calculated inter-frame difference value is greater than the preset inter-frame threshold, then it is considered that the frame contains sufficient new information or changes, so it is marked as a key frame. Conversely, if the inter-frame difference value is not greater than the preset inter-frame threshold, it means that there is no significant change in this frame compared with the previous frame, so it is marked as a non-key frame. To effectively distinguish which frames contain important information (key frames), detailed processing and analysis are required; while for which frames can be regarded as redundant or secondary information (non-key frames), perhaps only less processing resources are needed or the information of the previous key frame can be used for approximate processing.

[0045] Among them, the method for geometrically correcting the frame according to the flight attitude data includes: Step S201) Read the hyperspectral data according to a preset correction period and coincidence period, and the correction period is consistent with the reporting period of the visible light image data. Exemplarily, please refer to the appendix Figure 6 , set the correction period to 3s and the coincidence period to 1s. Setting the coincidence period can eliminate the stitching traces of the 3-second correction period. The hyperspectral data, flight attitude data, zenith light data and positioning data are sampled at 50Hz and aligned according to the trigger signal of the visible light image data. The trigger signal of the visible light image data is 1 / 3Hz, that is, it appears once every 3s. Align according to the rising edge of the trigger signal that appears once every 3s to determine the starting moment of the 50Hz trigger signal of the hyperspectral data, flight attitude data, zenith light data and positioning data. 3s is a collection period, forming a complete data packet. And data alignment, reflectivity calculation and geometric correction are carried out at a period of 3s.

[0046] Step S202) Obtain the POS data of this frame, and the POS data is obtained according to the positioning data and flight attitude data. Obtain the object plane position of each pixel according to the POS data; Step S203) Obtain the corresponding image space position according to the object plane position and obtain geometric correction parameters.

[0047] Among them, the method for obtaining the object plane position of each pixel according to the POS data includes: For each frame of the hyperspectral data in the correction period and coincidence period, obtain the POS data of this frame, and the POS data includes (xt, yt, zt, θt, φt, ψt), where xt, yt, zt respectively represent the longitude, latitude and altitude of the position, and θt, φt, ψt respectively represent the pitch angle, roll angle and yaw angle of the flight attitude of the UAV 20; Calculate the rotation matrix R according to the flight attitude (θt, φt, ψt); For each pixel (u, v) in the image, the corresponding object plane coordinates (X, Y, Z) are calculated using the collinearity equation.

[0048] The collinearity equation is:

[0049] Where (X, Y, Z) are the coordinates of the target point on the object plane, (Xc, Yc, Zc) is the center position of the camera (obtained by superimposing the positioning data in the POS data and the relative deviation of the positioning point from the camera center position), Rij are the elements of the rotation matrix, calculated from the attitude angles (θt, ϕt, ψt), and (u, v) are the coordinates of the pixel in the image.

[0050] The calculation process of the rotation matrix is: The rotation of each individual axis can be represented by a rotation matrix: Rotation about the Z-axis (i.e., yaw angle ψ)

[0051] Rotation about the Y-axis (pitch angle θ)

[0052] Rotation about the X-axis (roll angle ϕ)

[0053] Multiply the three basic rotation matrices in the order of ZYX to obtain the final rotation matrix:

[0054] Expanded in matrix form as:

[0055] Thus, the values of Rij can be obtained, where i, j ∈ [1, 3].

[0056] When the image space position is empty, fill the pixel into the image space position. When the image space position is not empty, fill in the pixel with a smaller weighted sum of the pitch angle and roll angle corresponding to the pixel. First, create a blank image space and fill the corrected pixel into the corresponding position in the image space. The image space is actually a two-dimensional image. The hyperspectral data includes a two-dimensional image and more dimensions for recording the reflectance. The reflectance is associated with a pixel in the two-dimensional image. When the pixel value corresponding to the image space is empty, directly fill in the corrected pixel. When the pixel in the image space is not empty, it means that the object plane position corresponding to this pixel has been photographed at other times due to the swing of the drone 20. That is, there is a situation of duplicate shooting. At this time, select the pixel with a smaller weighted sum of the pitch angle and roll angle and fill it in, that is, use the pixel when the drone 20 is in the closest ideal flight attitude.

[0057] When the image space position exceeds the boundary of the current frame, cache the pixel to the next correction period. When it exceeds the boundary of the current frame, it means that the photographed object plane is the position corresponding to other frames, so it is cached for use in the next correction period. When the position of the photographed object plane is the position corresponding to the previous frame, the pixel will eventually be discarded.

[0058] At the end of the calibration period, determine whether there are empty pixels in the image corresponding to this correction period. If there are empty pixels, use the hyperspectral data within the correction period and the coincidence period to interpolate and generate the pixel value at this position. When there are empty pixels, it means that there is a situation of missed shooting, so the interpolation algorithm is used to make up for it.

[0059] The method of using the hyperspectral data within the correction period and the coincidence period to interpolate and generate the pixel value of the empty pixel includes: Preset at least 3 interpolation algorithms, and divide them into high-precision interpolation algorithms, medium-precision interpolation algorithms, and low-precision interpolation algorithms according to the accuracy of the interpolation algorithms; When it is a key frame, preferably use a high-precision interpolation algorithm to generate the pixel value of the empty pixel. When it is a non-key frame, preferably use a low-precision interpolation algorithm to generate the pixel value of the empty pixel; Obtain the difference between the time when the previous frame completed interpolation and the cut-off time of the acquisition period, and obtain the system load according to the difference; When the system load is higher than the preset reference load, reduce the accuracy of the interpolation algorithm used on the basis of the preselection.

[0060] Exemplarily, the nearest neighbor interpolation algorithm, bilinear interpolation algorithm, and bicubic interpolation algorithm are used. The bicubic interpolation algorithm has high precision and can achieve interpolation for regions with clear edges and regions that require smooth transitions. The bilinear interpolation algorithm is a medium-precision interpolation algorithm and is applicable to ordinary image regions and regions with high real-time requirements. The nearest neighbor interpolation algorithm is a low-precision interpolation algorithm and is applicable to periods when FPGA resources are scarce, regions with image smoothing, and periods of temporary overload. The interpolation algorithm can use the modular IP core method to reduce the difficulty of FPGA programming. On the other hand, this specification provides a method for high-spectral acquisition carried by an unmanned aerial vehicle 20 based on FPGA. Please refer to the appendix Figure 7 , including the steps: Step S301) Receive the second pulse signal and positioning data provided by the unmanned aerial vehicle 20, and after frequency multiplying or dividing the second pulse signal according to a preset ratio, provide trigger signals to the hyperspectral camera module 34, visible light camera module 35, inertial measurement module 33, and zenith light module 32; Step S302) Align the hyperspectral data, visible light image data, flight attitude data, zenith light data, and positioning data along the time axis; Step S303) Calculate the reflectivity according to the aligned zenith light data and hyperspectral data; Step S304) Geometrically correct the hyperspectral data according to the flight attitude data and positioning data; Step S305) According to the preset reporting period, store the visible light image data, flight attitude data, zenith light data, and the corrected hyperspectral data in a specified persistent storage device, and transmit them to a specified ground control center 10.

[0061] Among them, the method for geometrically correcting the hyperspectral data according to the flight attitude data includes: According to the preset correction period and coincidence period, the correction period is consistent with the reporting period of the visible light image data, and read the hyperspectral data; For each frame of the hyperspectral data in the correction period and coincidence period, obtain the POS data of this frame, obtain the object plane position of each pixel according to the POS data, and obtain the corresponding image space position according to the object plane position; When the image space position is empty, fill the pixel into the image space position. When the image space position is not empty, fill the pixel with a smaller weighted sum of the pitch angle and roll angle; When the image space position exceeds the boundary of the current frame, cache the pixel to the next correction period; At the end of the calibration period, determine whether there are empty pixels in the image corresponding to this calibration period. If there are empty pixels, use the hyperspectral data within the calibration period and the coincidence period to interpolate and generate the pixel value at this position.

[0062] On the other hand, the method for obtaining the object plane position of each pixel based on the POS data includes: For each frame of the hyperspectral data in the calibration period and the coincidence period, obtain the POS data of this frame. The POS data includes (xt, yt, zt, θt, φt, ψt), where xt, yt, and zt respectively represent the longitude, latitude, and altitude of the position, and θt, φt, and ψt respectively represent the pitch angle, roll angle, and yaw angle of the flight attitude of the UAV 20; Calculate the rotation matrix R according to the flight attitude (θt, φt, ψt); For each pixel (u, v) in the image, use the collinearity equation to calculate its corresponding object plane coordinates (X, Y, Z).

[0063] On the other hand, according to the preset reporting period, the method for the FPGA processing module 31 to store the visible light image data, flight attitude data, zenith light data, and the calibrated hyperspectral data into a specified persistent storage device and transmit them to a specified ground control center 10 includes: According to the trigger signal, store the visible light image data, flight attitude data, zenith light data, and the calibrated hyperspectral data into a specified cache; When the preset reporting period is reached, store the visible light image data, flight attitude data, zenith light data, and the calibrated hyperspectral data from the specified cache into a specified persistent storage device; Then, transmit the visible light image data, flight attitude data, zenith light data, and the calibrated hyperspectral data from the specified cache to the specified ground control center 10 via the UAV 20; Clear the visible light image data, flight attitude data, zenith light data, and the calibrated hyperspectral data corresponding to the reporting period from the specified cache.

[0064] The above-described embodiments are only described in a preferred embodiment manner of this specification, and do not limit the scope of this specification. Without departing from the design spirit of this specification, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of this specification shall fall within the protection scope determined by the claims of this specification.

Claims

1. A UAV-borne hyperspectral acquisition system based on FPGA, characterized in that it includes an FPGA processing module, a hyperspectral camera module, a visible light camera module, an inertial measurement module, a zenith light module and a power supply module carried by a UAV. The FPGA processing module receives the second pulse signal and positioning data provided by the UAV, and provides trigger signals to the hyperspectral camera module, the visible light camera module, the inertial measurement module and the zenith light module according to the second pulse signal, and generates an acquisition period according to the trigger signals. During the acquisition period, the FPGA processing module performs the following steps: Open a temporary storage area according to the preset data packet size; Receive the trigger signals to trigger the hyperspectral camera module, the visible light camera module, the inertial measurement module and the zenith light module to collect hyperspectral image data, visible light image data, flight attitude data and zenith light data; Align and cache the hyperspectral image data, flight attitude data, zenith light data and positioning data along the time axis according to the visible light image data, and write the visible light image data into the temporary storage area; Calculate the reflectance according to the aligned zenith light data and hyperspectral image data, perform geometric correction on the hyperspectral image data according to the flight attitude data and positioning data to obtain hyperspectral data, and write the hyperspectral data into the temporary storage area; Generate a data packet according to the temporary storage area, store the data packet in a specified persistent storage device and send it back to a specified ground control center, clear the cache and recycle the temporary storage area.

2. The UAV-borne hyperspectral acquisition system based on FPGA according to claim 1, characterized in that when calculating the reflectance according to the aligned zenith light data and hyperspectral data, the FPGA processing module performs the following steps: Read the correction coefficient η measured under laboratory conditions; Calculate the reflectance through the following formula: d = DN4 / (DN3 × η) where d is the reflectance, DN4 is the hyperspectral data, and DN3 is the corresponding zenith light data; where the method for obtaining the correction coefficient η measured under laboratory conditions includes: Measure the zenith light data DN1 collected by the zenith light module and the hyperspectral measurement data DN2 of a whiteboard with a known reflectance d1 collected by the hyperspectral camera module under the same light source in the laboratory, and the correction coefficient η = DN2 / (DN1 × d1).

3. The UAV-borne hyperspectral acquisition system based on FPGA according to claim 1 or 2, characterized in that when performing geometric correction on the hyperspectral data according to the flight attitude data, the FPGA processing module performs the following steps: For each frame of the hyperspectral data in the correction period and the coincidence period, obtain the POS data of this frame. The POS data is obtained according to the positioning data and flight attitude data. Divide the frame into key frames and non-key frames according to the hyperspectral data and POS data; When it is a key frame, perform geometric correction on the frame according to the flight attitude data and store the geometric correction parameters; When it is a non-key frame, calculate the intra-frame difference value of the hyperspectral data of the non-key frame; When the intra-frame difference value is greater than a preset reference threshold, geometric correction is performed on the frame according to the flight attitude data; When the intra-frame difference value is not greater than the preset reference threshold, geometric correction is performed using the geometric correction parameters of the nearest key frame; The method for classifying frames into key frames and non-key frames according to the hyperspectral data and POS data includes: Calculating the inter-frame difference value between the hyperspectral data of the frame and the previous frame; When the inter-frame difference value is greater than the preset inter-frame threshold, determining that the frame is a key frame; When the inter-frame difference value is not greater than the preset inter-frame threshold, determining that the frame is a non-key frame; The method for performing geometric correction on the frame according to the flight attitude data includes: Reading the hyperspectral data according to a preset correction period and coincidence period, and the correction period conforms to the reporting period of the visible light image data; Obtaining the POS data of the frame, where the POS data is obtained according to the positioning data and flight attitude data, and obtaining the object plane position of each pixel according to the POS data; Obtaining the corresponding image space position according to the object plane position and obtaining geometric correction parameters; Among them, when obtaining the corresponding image space position according to the object plane position, when the image space position is empty, filling the pixel into the image space position, and when the image space position is not empty, filling the pixel with a smaller weighted sum of the pitch angle and roll angle corresponding to the pixel; When the image space position exceeds the boundary of the current frame, caching the pixel to the next correction period; At the end of the calibration period, determining whether there are empty pixels in the image corresponding to the calibration period. If there are empty pixels, interpolating the pixel values of the empty pixels using the hyperspectral data within the calibration period and the coincidence period.

4. An airborne hyperspectral acquisition system based on FPGA according to claim 3, wherein The method for obtaining the object plane position of each pixel according to the POS data includes: For each frame of the hyperspectral data in the correction period and the coincidence period, obtaining the POS data of the frame, where the POS data includes (xt, yt, zt, θt, φt, ψt), where xt, yt, zt respectively represent the longitude, latitude and altitude of the position, and θt, φt, ψt respectively represent the pitch angle, roll angle and yaw angle of the flight attitude of the unmanned aerial vehicle; Calculating the rotation matrix R according to the flight attitude (θt, φt, ψt); For each pixel (u, v) in the image, calculating its corresponding object plane coordinates (X, Y, Z) using the collinearity equation.

5. An airborne hyperspectral acquisition system based on FPGA according to claim 4, wherein The method for interpolating the pixel values at the position using the hyperspectral data within the calibration period and the coincidence period includes: Obtaining the POS data of the frame where the empty pixel is located, obtaining the object plane position of the empty pixel, denoted as the object plane empty position; Obtaining the pixel values corresponding to the positions around the object plane empty position, denoted as the surrounding pixel values; Interpolating the pixel value of the empty pixel according to the surrounding pixel values.

6. A UAV-borne hyperspectral acquisition system based on FPGA, characterized in that The method of using the hyperspectral data within the calibration period and coincidence period to interpolate and generate the pixel value of the empty pixel includes: Preset at least 3 interpolation algorithms, and classify them into high-precision interpolation algorithms, medium-precision interpolation algorithms, and low-precision interpolation algorithms according to the accuracy of the interpolation algorithms; When it is a key frame, preselect to use a high-precision interpolation algorithm to generate the pixel value of the empty pixel, and when it is a non-key frame, preselect to use a low-precision interpolation algorithm to generate the pixel value of the empty pixel; Obtain the difference between the time when the previous frame finishes interpolation and the cut-off time of the acquisition period, and obtain the system load according to the difference; When the system load is higher than the preset reference load, reduce the accuracy of the interpolation algorithm used on the basis of preselection.

7. A UAV-borne hyperspectral acquisition method based on FPGA, characterized in that It includes the steps of: Receiving the second pulse signal and positioning data provided by the UAV, providing trigger signals to the hyperspectral camera module, visible light camera module, inertial measurement module, and zenith light module according to the second pulse signal, and generating an acquisition period according to the trigger signal, During the acquisition period, perform the following steps: Open a temporary storage area according to the preset data packet size; Receive the trigger signal to trigger the hyperspectral camera module, visible light camera module, inertial measurement module, and zenith light module to collect hyperspectral image data, visible light image data, flight attitude data, and zenith light data; Align and cache the hyperspectral image data, flight attitude data, zenith light data, and positioning data along the time axis according to the visible light image data, and write the visible light image data into the temporary storage area; Calculate the reflectivity according to the aligned zenith light data and hyperspectral image data, perform geometric correction on the hyperspectral image data according to the flight attitude data and positioning data, obtain hyperspectral data, and write the hyperspectral data into the temporary storage area; Generate a data packet according to the temporary storage area, store the data packet in a specified persistent storage device and backhaul it to a specified ground control center, clear the cache and recycle the temporary storage area.

8. An electronic device, characterized in that, It includes a processor and a memory; The processor is connected to the memory; The memory is used to store executable program codes; The processor runs the program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method as claimed in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by the processor, implements the method as claimed in claim 7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the method as claimed in claim 7.

Images