A programmable hyperspectral lidar device and real-time registration method thereof

Abstract

The present invention discloses a programmable hyperspectral laser radar device and a real-time registration method thereof. The programmable hyperspectral laser radar device includes: an optical module, a real-time signal processing module, which processes the received laser echo signal in real time, obtains a number of arbitrarily programmable spectral channels based on the window position and spectral wavelength width selected based on the converted non-laser echo signal, and then reconstructs the image information of the obtained several spectral channels through a spectral separation method to obtain different spectral images corresponding to the target scanning area; a data integration module, which is used for real-time registration of the laser point cloud and the hyperspectral image, and assigns three-dimensional coordinate information and spectral information to be retained to the laser point cloud. In the above technical solution, it supports user-defined window position and spectral wavelength width, extracts only specific bands, avoids full-band data redundancy, improves processing efficiency, and retains key spectral features, making it suitable for specific scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of laser radar, and in particular relates to a programmable hyperspectral laser radar device and a real-time registration method thereof. Background Art

[0002] LiDAR emits lasers, calculates the time information between the transmitted signal and the echo signal reflected from the surface of the illuminated object, and then uses the LiDAR's own angle and position information to obtain three-dimensional information of the target. It has the characteristics of penetration and high measurement accuracy, but it is difficult to obtain the target's attribute information, and its use in target identification and ground object monitoring is limited. Hyperspectral imaging technology can obtain rich spectral information and two-dimensional geometric spatial information of the target, and is currently the main means of obtaining target attribute information. However, traditional hyperspectral cameras obtain spectral information of the camera's full band in each shot, contain a large amount of redundant information, and have limited ability to obtain three-dimensional spatial information of the target. Summary of the Invention

[0003] The purpose of the present invention is to address the problems existing in the prior art and provide a programmable hyperspectral lidar device and a real-time registration method thereof, so as to realize the hardware integration of lidar and hyperspectral camera detection modules, and realize the real-time acquisition and registration of laser point clouds and hyperspectral images; as well as realize the efficient acquisition and registration of three-dimensional spatial information and hyperspectral data, thereby solving the limitation that single remote sensing means such as lidar and hyperspectral camera cannot simultaneously obtain accurate three-dimensional geometric information and rich spectral information of the target.

[0004] Based on the description, the present invention provides a programmable hyperspectral lidar device, including: an optical module for emitting a laser to illuminate a target, wherein the echo signal reflected by the target is incident on a laser receiving system and a non-laser receiving system respectively through a spectroscope, and the non-laser receiving system uses a linear gradient filter and an array detector to receive the non-laser echo signal;

[0005] A real-time signal processing module is used to process the laser echo signal and the non-laser echo signal in real time. The processing of the non-laser echo signal includes: selecting a window position and a spectral wavelength width, obtaining a plurality of arbitrarily programmable spectral channels, and reconstructing the image information of the plurality of spectral channels by spectral separation method to obtain different spectral images corresponding to the target scanning area;

[0006] The data integration module is used to perform real-time registration and integration of processed laser point clouds and hyperspectral images.

[0007] In the above technical solution, the non-laser wavelength echo signal is transmitted to the non-laser receiving system through a spectroscope to synchronously obtain the spectral information of the target. The data integration module directly assigns spectral information to the three-dimensional point cloud data through time synchronization and coordinate alignment, forming multimodal data of "three-dimensional coordinates + spectral attributes". This allows the three-dimensional point cloud of the lidar to be integrated with the spectral information of the hyperspectral image, making up for the defect of traditional lidar in not being able to obtain attribute information, and improving target recognition and ground object monitoring capabilities. It supports user-defined window positions and spectral wavelength widths, extracting only specific bands, avoiding full-band data redundancy, improving processing efficiency, while retaining key spectral features, and being suitable for specific scenarios. The spectroscope synchronously separates the laser echo signal and the non-laser echo signal to achieve consistent data collection in time and space. In summary, the above technical solution successfully solves the dual contradictions of lidar's lack of attribute information, high redundancy of hyperspectral imaging, and weak three-dimensional capability through three major innovations: hardware collaborative design of spectrometry / detection / scanning, programmable signal processing, and intelligent alignment algorithm.

[0008] As a further technical solution, the optical module further includes: a tower mirror with four 45° inclined reflection surfaces for achieving multi-angle scanning of the same target.

[0009] As a further technical solution, the laser echo signal is processed in real time, including: converting the laser echo signal into a current signal and sending it into a transimpedance amplifier circuit with a multi-rate transimpedance amplifier gain, selecting the channel output that is not saturated and has the highest peak voltage, and performing analog-to-digital conversion on the output voltage.

[0010] As a further technical solution, the data integration module includes:

[0011] The time synchronization unit is used to provide a sub-microsecond synchronized clock source for multiple sensors on the device;

[0012] The target calibration unit is used to extract the same-name feature points of the point cloud grayscale image and the hyperspectral image through the ground target, complete the registration using similarity transformation, and assign the point cloud spectral information to the nearest laser scanning point through interpolation. This is done until each laser scanning point is assigned point cloud spectral information, and each frame of point cloud data with spectral information is matched with its position and posture data at the corresponding timestamp.

[0013] The conversion unit is used to convert the point cloud data from the lidar coordinate system to the global coordinate system according to the position information and attitude angle information provided by the position and attitude data.

[0014] As a further technical solution, the target calibration unit is further configured to execute the following instructions:

[0015] Identify the coordinates of the target center point in the point cloud grayscale image coordinate system and the hyperspectral image coordinate system, use the FLANN algorithm to match the feature points with the same name, use the least squares method and similarity transformation to achieve the registration of the two images after the matching of the same name features, and convert the spectral image into the point cloud coordinate system.

[0016] As a further technical solution, the device is mounted on a drone and acquires the three-dimensional coordinate information of a large-area target, the signal strength value recorded by the laser radar when receiving the reflected echo, and the spectral information of the selected band through a push-scan method.

[0017] Based on the specification, the present invention provides a drone equipped with the above-mentioned programmable hyperspectral lidar device.

[0018] Based on the description, the present invention provides a real-time registration method for a programmable hyperspectral laser radar device, comprising the following steps:

[0019] The laser is emitted to the target through the optical module. The echo signal reflected by the target is incident on the laser receiving system and the non-laser receiving system respectively through the beam splitter. The non-laser receiving system uses a linear gradient filter and an array detector to receive the non-laser echo signal.

[0020] Real-time processing of laser echo signals and hyperspectral image data through real-time signal processing modules;

[0021] Establish sub-microsecond multi-sensor time synchronization benchmark through data integration module;

[0022] Generate point cloud grayscale images and hyperspectral images through the target, extract feature points with the same name and align them;

[0023] Convert the spectral image to the point cloud coordinate system and interpolate to give the laser point cloud spectral information;

[0024] Each frame of point cloud data with spectral information is matched with the position and posture data at the corresponding timestamp, and the aligned point cloud is converted to the global coordinate system based on the position and posture data.

[0025] The above technical solution covers the entire process from laser emission and echo signal reception, real-time signal processing, multi-sensor time synchronization benchmark establishment, target generation and registration, spectral image conversion and point cloud spectral information assignment, and finally, global coordinate system conversion based on position and posture data. This forms a systematic, efficient, and accurate real-time registration solution for laser point clouds and hyperspectral imagery. Through the close connection and collaborative work of each step, an integrated operation is achieved from data acquisition to processing and fusion, ensuring data accuracy and consistency, providing users with more comprehensive and accurate geospatial information, and meeting the needs of different fields for high-precision, multi-dimensional data.

[0026] As a further technical solution, the establishment of a sub-microsecond multi-sensor time synchronization benchmark includes:

[0027] The local oscillator is tamed by the satellite receiver to adjust its frequency and phase to be consistent with the satellite signal;

[0028] Direct digital frequency synthesis technology is used to output the fractional-second signal, and the multi-sensor clock synchronization error is controlled to be less than the target time.

[0029] As a further technical solution, interpolation is used to assign spectral information to the laser point cloud, which also includes:

[0030] The bilinear interpolation method is used to map the spectral information of the spectral image pixel to the nearest laser point cloud;

[0031] The registration accuracy reaches the predetermined pixel, and each point cloud contains three-dimensional coordinates, intensity and user-defined band spectral information.

[0032] Compared with the prior art, the beneficial effects of the present invention are: 1. The non-laser wavelength echo signal is transmitted to the non-laser receiving system through the spectroscope to synchronously obtain the spectral information of the target; the data integration module directly assigns the spectral information to the three-dimensional point cloud data through time synchronization and coordinate alignment, forming multi-modal data of "three-dimensional coordinates + spectral attributes", allowing the three-dimensional point cloud of the lidar to be integrated with the spectral information of the hyperspectral, making up for the defect that the traditional lidar cannot obtain attribute information, and improving the target recognition and ground object monitoring capabilities; 2. It supports user-defined window position and spectral wavelength width, only extracts specific bands, avoids full-band data redundancy, improves processing efficiency, and retains key spectral features, which is suitable for specific scenarios; 3. The spectroscope synchronously separates the laser echo and non-laser spectral signals to achieve time-space consistent data acquisition, and assigns spectral attributes to the three-dimensional point cloud by mapping the spectral image to the point cloud coordinate system. While obtaining the two-dimensional spectral information of the target, combined with the three-dimensional coordinates of the lidar, spectral data with spatial dimensions is formed, breaking through the limitation that the traditional hyperspectral can only provide two-dimensional plane information. In summary, this solution successfully solves the dual contradictions of lidar's lack of attribute information, high redundancy of hyperspectral imaging, and weak three-dimensional capabilities through three major innovations: hardware collaborative design of spectroscopy / detection / scanning, programmable signal processing, and intelligent registration algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A structural block diagram of a programmable hyperspectral laser radar device provided in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the optical path of a programmable hyperspectral lidar device provided in an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of the design of the linear gradient filter and the area array detection array provided in an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the working principle of a non-laser receiving system provided in an embodiment of the present invention;

[0037] Figure 5 A flowchart for processing spectral images and point cloud data provided by an embodiment of the present invention;

[0038] Figure 6 A flowchart of a real-time registration method for a programmable hyperspectral lidar device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In the description of the present invention, it should be noted that the terms "middle", "upper", "lower", "left", "right", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the present invention.

[0041] In order to meet the design indicators of programmable hyperspectral lidar equipment, such as high spectral resolution, high temporal resolution, high spatial resolution, and lightweight integrated equipment, the following innovations are made: a programmable non-laser receiving system, a multi-angle scanning optical module, and a joint calibration of laser echo signals and non-laser echo signals.

[0042] like Figure 1 and Figure 2 As shown, the present invention provides a programmable hyperspectral laser radar device, comprising:

[0043] An optical module is used to emit laser light onto a target. The echo signal reflected by the target is incident on a laser receiving system and a non-laser receiving system respectively through a spectroscope. The non-laser receiving system uses a linear gradient filter and an array detector (array detection array) to receive the non-laser echo signal.

[0044] A real-time signal processing module is used to process the laser echo signal and the non-laser echo signal in real time. The processing of the non-laser echo signal includes: selecting a window position and a spectral wavelength width, obtaining a plurality of arbitrarily programmable spectral channels, and reconstructing the image information of the plurality of spectral channels by spectral separation method to obtain different spectral images corresponding to the target scanning area;

[0045] The data integration module is used to perform real-time registration and integration of processed laser point clouds and hyperspectral images.

[0046] In the above technical solution, the non-laser wavelength echo signal is transmitted to the non-laser receiving system through a spectroscope to synchronously obtain the spectral information of the target. The data integration module directly assigns the spectral information to the three-dimensional point cloud data through time synchronization and coordinate registration, forming multimodal data of "three-dimensional coordinates + spectral attributes". The three-dimensional point cloud of the lidar is integrated with the spectral information of the hyperspectral system, which makes up for the defect that traditional lidar cannot obtain attribute information and improves target recognition and ground object monitoring capabilities. It supports user-defined window positions and spectral wavelength widths, extracts only specific bands, avoids full-band data redundancy, improves processing efficiency, and retains key spectral features, which is suitable for specific scenarios. The spectroscope synchronously separates the laser echo and non-laser spectral signals to achieve consistent data acquisition in time and space, maps the spectral image to the point cloud coordinate system, and assigns spectral attributes to the three-dimensional point cloud. While obtaining the two-dimensional spectral information of the target, it is combined with the three-dimensional coordinates of the lidar to form spectral data with spatial dimensions, breaking through the limitation of traditional hyperspectral systems that can only provide two-dimensional planar information. In summary, the above technical solution successfully solves the dual contradictions of lidar's lack of attribute information, high redundancy of hyperspectral imaging and weak three-dimensional capability through three major innovations: hardware collaborative design of spectroscopy / detection / scanning, programmable signal processing and intelligent registration algorithm.

[0047] It should be noted that in the optical module, the fiber laser is divided into two optical output paths: seed light and main light. The seed light has weak energy and is directly coupled into a photodetector through optical fiber, which converts it into an electrical signal as the starting signal for laser emission. A laser collimator and beam expander is connected to the main light fiber head. The ranging module calculates the time difference between the seed light emission and the main light echo, and the ToF method is used to calculate the distance value.

[0048] In this embodiment, the optical module also includes a tower mirror with four 45° inclined reflective surfaces. The four tower mirrors are driven by a brushless motor and form a 45-degree angle with the incident light. The four reflective surfaces alternately reflect the laser light in sequence to achieve multi-angle scanning of the same target.

[0049] In this embodiment, a near-zero dispersion full-spectrum light-transmitting glass material and a spectral offset corrected image-space telecentric optical path are used, and the distortion in the full field of view is less than 0.035%. At the same optical aperture, it has a higher luminous flux, realizing weak signal imaging detection, thereby improving the luminous flux and weak signal imaging detection capabilities.

[0050] Specifically, the spectrometer uses a spectrometer or other device with a spectroscopic system, and coats it with an anti-reflection film to reflect the optical signal of the laser wavelength into the laser receiving optical system, and transmit the optical signals of other wavelengths into the non-laser receiving system. For the echo signal of non-laser wavelength, a linear gradient filter is installed in front of the array detector as a beam splitting element, solving the problems of high-precision installation and spectral line bending in traditional spectrometer systems and simplifying the optical system. The center wavelength of the linear gradient filter changes linearly along the direction of filter thickness change, so that the spectral wavelength received by each row of pixels on the array detector corresponds to the transmission wavelength of the filter at the corresponding position. By mounting the system on a drone and sweeping it in the direction of flight, the sweeping direction is consistent with the wavelength gradient direction, so that each column of pixels is sampled at different spatial positions along the scanning direction. The optical signal is converted into an electrical signal by the array detector. The system is equivalent to being composed of multiple linear arrays, and multiple sets of two-dimensional spectrum information are obtained through one-dimensional motion, thereby obtaining a complete target spectral image.

[0051] In this embodiment, the laser echo signal is processed in real time: the received laser echo signal is converted into a current signal through a photodetector, and the current signal is converted into a voltage signal using a transimpedance amplifier circuit. Resistors of different resistance values ​​are connected in series with the switch circuit to form a group, and are connected in parallel with other groups. The switch circuit is controlled by the integrated processing unit to form different transimpedance values, and finally transimpedance amplification gains of different magnifications are obtained. The output of the channel with the highest peak voltage that has not yet reached saturation is preserved, so that the echo signals within the largest possible ranging range can be within the range of the AD converter and the amount of data storage is reduced. After the voltage signal passes through the high-speed AD converter, the digital signal is sent to the FPGA for subsequent data processing.

[0052] In this embodiment, the data integration module includes:

[0053] The time synchronization unit is used to provide a sub-microsecond synchronized clock source for multiple sensors on the device by combining direct digital frequency synthesis with satellite pulse-second taming technology;

[0054] The target calibration unit is used to extract the same-name feature points of the point cloud grayscale image and the hyperspectral image through the ground target, complete the registration using similarity transformation, and assign the point cloud spectral information to the nearest laser scanning point through interpolation. This is done until each laser scanning point is assigned point cloud spectral information, and each frame of point cloud data with spectral information is matched with its position and posture data at the corresponding timestamp.

[0055] The conversion unit is used to convert the point cloud data from the lidar coordinate system to the global coordinate system according to the position information and attitude angle information provided by the position and attitude data.

[0056] In this embodiment, in the target calibration unit:

[0057] The ground targets are a black and white checkerboard pattern, arranged every 20 meters along the route and not collinear;

[0058] The FAST operator is used to identify the coordinates of the target center point in the point cloud grayscale image coordinate system and the hyperspectral image coordinate system. The FLANN algorithm (fast approximate nearest neighbor algorithm) matches the feature points with the same name. The least squares method is used to achieve registration of the two images after the matching of the same-name features through similarity transformation, and the spectral image is converted to the point cloud coordinate system.

[0059] The non-laser receiving system of the present invention uses high-dispersion refined gradient filtering technology to finely align the linear gradient filter with the CMOS / InGaAs pixel, achieving arbitrary programmability of thousands of spectral channels and reducing spectral data redundancy.

[0060] The optical module of the present invention adopts high-precision digital waveform online processing and ranging and angle measurement compensation technology to meet the submillimeter ranging accuracy requirements. It uses high-precision angle measurement technology with angle subdivision and multi-reader compensation, develops angle precision subdivision and compensation algorithms, adopts precision grating and multi-reader angle measurement methods to achieve high-resolution angle measurement, studies system error calibration technology for ranging and angle measurement, establishes a ranging system error model and studies ranging error compensation schemes, establishes a LiDAR angle measurement system error model and completes self-calibration of the angle measurement system error, thereby achieving high-precision angle and ranging measurement.

[0061] The data integration module of the present invention completes the internal and external parameter calibration of the camera and lidar and the joint calibration of multiple sensors, constructs a mapping relationship model between point cloud data and spectral images, and realizes real-time mapping and data transmission between point cloud and spectral images; the integrated system is rationally designed and material selected to ensure the stability and practicality of the system.

[0062] The programmable hyperspectral lidar device proposed in the present invention is designed with a non-laser receiving system and an optical module. Through multi-sensor integration, the design of a lightweight airborne hyperspectral lidar system is realized. The device is mounted on an unmanned aerial vehicle (UAV) device and can acquire large-area, high-precision target three-dimensional coordinate information, intensity information, and spectral information of selected bands through overlapping push-scanning within a certain spatial range by the UAV device.

[0063] In the non-laser receiving system, an area array CMOS detector is used, and a linear gradient filter is used to replace the traditional grating and prism as the spectroscopic device of the hyperspectral camera, which solves the problem of prism and grating spectroscopic systems being limited by slits. Using fine spectral characteristic detection and calibration technology, a high-precision monochromator is selected to detect the spectral information obtained by the gradient filter spectrometry, and its wavelength, relative light intensity, resolution and linearity are calibrated to achieve the development of high-dispersion fine gradient filters. The linear gradient filter is installed on the area array CMOS detector, and its central wavelength changes linearly along the direction of change of the filter thickness, so that the spectral wavelength received by each row of pixels on the area array detector corresponds to the transmission wavelength of the filter at the corresponding position. Its design is as follows: Figure 3 As shown. By placing a non-laser receiving system on a drone and sweeping it in the direction of flight, with the sweep direction aligned with the wavelength gradient, each column of pixels is sampled at different spatial positions along the scanning direction. The system is equivalent to being composed of multiple linear arrays, acquiring multiple sets of two-dimensional spectral information through one-dimensional motion, thereby obtaining a complete spectral image of the target. By combining gradient filter spectroscopic technology with digital domain time delay integration technology based on a CMOS array, the imaging results of the same target at different times are collected, and the number of electrons in multiple images is added to improve the signal-to-noise ratio. This allows for imaging of 32 spectral segments and integral series within the 400-1000nm spectral range. This increases the image signal energy without increasing the aperture of the imaging spectrometer, thereby improving image accuracy.

[0064] The non-laser receiving system design also features a free-form surface with an ultra-large aperture ratio. This system utilizes near-zero dispersion, full-spectrum transparent glass and a spectral offset-corrected image-space telecentric optical path. This system achieves higher luminous flux at the same optical aperture, enabling imaging and detection of weak signals. The linear gradient filter, installed as a beamsplitting element in front of the area array detector, eliminates spectral line bending and nonlinearity during use, simplifying the optical calibration process. This eliminates the need for high-precision installation of multiple beamsplitting elements found in traditional prism and grating imaging spectrometers, resulting in an extremely small beamsplitting structure, a simple optical system, and a more compact camera system, further contributing to the system's lightweight design. The spectral segment on-orbit programming imaging technology is adopted. The UAV push-scan continuously samples in the spectral region and the spatial region along the flight direction. The spectral image data corresponding to each ground area obtained by the scan is a combination of information obtained at different times and in different rows of pixels. The non-laser receiving system can independently select the required window position and spectral wavelength width to achieve arbitrary programming of thousands of spectral channels. Taking into account the data transmission capacity and actual application requirements of the camera, the non-laser receiving system can select the band information required by the user as the output spectral band, and then reconstruct the image information of the characteristic spectrum through the spectral separation method to obtain the spectral image corresponding to the target scanning area. Its working principle is shown in Figure 4Compared with traditional hyperspectral cameras, non-laser receiving systems have the advantages of compact optical structure, wide spectral band range, and arbitrarily selectable spectral bands. They can reduce the storage capacity of data volume, avoid data redundancy, and are conducive to lightweight system design.

[0065] The optical module uses high-precision digital waveform online processing technology and implements a digital waveform processing algorithm based on GHz analog-to-digital conversion and field-programmable logic gate array chips: an FIR filter is used to filter the original waveform to reduce the impact of time jitter error and waveform amplitude signal error on subsequent echo signal processing; a CIC interpolation filter is used to interpolate the effective echo waveform to improve ranging accuracy when the pulse waveform is too narrow or the ADC sampling frequency is low; dynamic threshold waveform detection is used to achieve adaptive threshold adjustment to address the impact of problems such as different DC bias of different devices and increased bias caused by insufficient circuit discharge on waveform detection; a Gaussian fitting algorithm is used to fit the target waveform through a preset target waveform model, which can obtain high-precision ranging values ​​and echo intensity information in real time. Even when the ADC sampling frequency is low, sub-millimeter high-precision ranging can still be achieved. Develop high-precision angle measurement technology using angle subdivision and multi-reader compensation. Mathematically analyze the signal DC component error, amplitude error, phase error, harmonic component error, noise error, and quantization error generated by angle subdivision, derive error patterns and calculation formulas, and form a relatively complete mathematical analysis of photoelectric encoder subdivision error and accuracy. Furthermore, study the installation eccentricity and tilt errors generated during the installation of the angle sensor. Based on these error analysis results, develop an angle precision subdivision and compensation algorithm. To correct the systematic errors of laser ranging and angle measurement caused by device defects and structural processing and adjustment errors, calibration is performed in the order of amplitude and phase error, temperature error, multiplication coefficient error, and addition coefficient error. The ranging system error is established and a ranging error compensation scheme is studied. Based on the deviation between the theoretical and actual values ​​of azimuth and elevation angles and the reference total station error model, a self-calibration method for the angle measurement system error is designed to ultimately achieve high-precision ranging.

[0066] In the data integration module, it mainly consists of two parts: high-precision time reference and joint calibration of image point cloud data.

[0067] In this embodiment, when establishing a high-precision time reference method, the time synchronization unit compares the fixed frequency signal provided by the satellite timing receiver with the oscillation signal generated by the local oscillator to obtain a frequency difference. The local oscillator is adjusted through direct digital frequency synthesis technology to make the oscillation frequency basically consistent with the oscillation frequency of the satellite. The phase of the local oscillator is compensated by controlling the phase change rate so that the difference between the fractional-second signal output by the local oscillator and the second signal output by the receiver is within a certain range. A clock source is provided for multiple sensors in the system through a clock output interface, thereby improving the time synchronization accuracy from sub-millisecond to sub-microsecond, realizing high-precision time synchronization of multiple sensors such as laser scanners, hyperspectral cameras, inertial navigation units, and Beidou navigation units, and establishing high-accuracy timestamps for the spectral image data and laser point cloud data obtained by scanning to meet the subsequent high-precision alignment requirements of spectral data and point cloud data.

[0068] For the laser point cloud data and spectral images acquired at the same time, there is a geometric association between the coordinate systems. The image coordinates and point cloud coordinates are unified into the same coordinate system to achieve spatial synchronization of the three-dimensional point cloud and two-dimensional image data at the same timestamp, and obtain a point cloud with texture information of the user-selected spectral band.

[0069] For example, the Zhang Zhengyou calibration method is used to take 20 images to complete the calibration of the internal and external parameters of the camera. For a target point in three-dimensional space at a certain moment, the coordinates of the point in the 3D lidar coordinate system are , the coordinates in the camera coordinate system are Both coordinate systems are three-dimensional space coordinate systems, and the conversion relationship between them can be expressed as:

[0070] ,

[0071] in, Represents the translation vector from the camera coordinate system to the radar coordinate system, R is the camera's external parameter matrix, is the translation vector from the point cloud coordinate system to the camera coordinate system, is the rotation matrix from the point cloud coordinate system to the camera coordinate system, ,in:

[0072] ,

[0073] Indicates the rotation angle of the radar coordinate system relative to the camera coordinate system in the x, y, and z directions.

[0074] The two-dimensional coordinates of the physical point in the pixel coordinate system are expressed as (u, v), and the conversion relationship between the image plane coordinate system and the pixel coordinate system can be defined by the following equation:

[0075] ,

[0076] Where f is the focal length of the camera, dx and dy represent the pixel conversion units along the x-axis and y-axis respectively (unit: mm / pixel), and Represents the offset of the projection plane center relative to the optical axis.

[0077] The formula can be sorted out:

[0078] ,

[0079] in, , is the intrinsic parameter matrix of the camera, is the external parameter matrix of the camera. Using a two-dimensional checkerboard, 25 sets of feature points corresponding to the lidar and the hyperspectral image are selected for calibration, and the , the registration between the LiDAR 3D point cloud and the hyperspectral image can be completed. This establishes the mapping relationship between the laser point and the image pixel at the same time stamp.

[0080] During the scanning process, targets are evenly distributed on the ground to ensure that there are no less than 3 targets within the system scanning range on each route, and these targets are evenly distributed on both sides of the route. Black and white checkerboard targets are used to ensure that their center positions can be clearly identified in both point cloud grayscale images and hyperspectral images. The system splits the spectral images of multiple bands required by the user into multiple single-band spectral images spliced ​​in chronological order, and splices the point cloud data according to the scanning time order provided by the position and posture data. The specific implementation is as follows: Figure 5 shown.

[0081] Combining the signal strength values ​​recorded by the laser radar when receiving the reflected echo from the laser point cloud, the laser point cloud is rasterized and interpolated to generate a two-dimensional grayscale point cloud image, whose size and posture are consistent with the acquired laser point cloud. The FAST operator is used to identify the coordinates of the center points of each target on the point cloud grayscale image and hyperspectral image in the point cloud grayscale image coordinate system and the hyperspectral image coordinate system, respectively. The FLANN algorithm is used to match the same-name feature points in the two coordinate systems. The two images are then aligned using the least squares method through a similarity transformation after the same-name feature matching, and the spectral image is converted to the point cloud coordinate system.

[0082] It should be noted that the corresponding coordinates of the feature points with the same name in the hyperspectral image coordinate system and the point cloud coordinate system must be identified first because the algorithm must be used to match the points with the same name in the two coordinate systems, and then the corresponding points with the same name must be aligned through similarity transformation.

[0083] Interpolation is used to assign the spectral information at the pixel points of the spectral image to the nearest laser scanning point, mapping the spectral information to 3D point cloud data. Repeating this process assigns each laser scanning point 3D position information, the signal strength value recorded by the LiDAR when receiving the reflected echo, and the spectral information for all user-defined bands. Each frame of point cloud data with spectral information is matched with the position and attitude data at its corresponding timestamp. Based on the position and attitude angle information (pitch and roll angles) provided by the position and attitude data, the point cloud data is converted from the LiDAR coordinate system to the global coordinate system, reconstructing and aligning the scanned point cloud with the spectral image in the desired band.

[0084] It should be noted that after the LiDAR and camera are synchronized in time and space, the LiDAR point cloud data is projected and converted. This means that the point cloud data is projected from three-dimensional space onto two-dimensional image data. The mapped coordinates of the point cloud data are rounded to ensure an accurate correspondence between the three-dimensional point cloud data and the two-dimensional image data. The spectral information of the pixel points extracted from the two-dimensional point cloud data is then mapped to the three-dimensional point cloud data, giving each laser scanning point three-dimensional position information, the signal intensity value recorded by the LiDAR when receiving the reflected echo, and the hyperspectral information of the user-defined band. This achieves lightweight system integration of multiple sensors, real-time correction and registration of hyperspectral image data and point cloud data, and obtains target information with high temporal resolution, high spatial resolution, and high spectral resolution.

[0085] This embodiment also includes a hardware system with high environmental adaptability, including a waterproof and dustproof packaging structure and an anti-vibration design. In the hardware system with high environmental adaptability:

[0086] The high environmental adaptability hardware system aims to address the possible damage to the instrument caused by extremely bad weather. It designs waterproof and dustproof packaging structures for non-laser receiving systems, laser light sources, signal detectors, optical modules, rotary scanning modules, and integrated circuits. Epoxy resin materials are applied to the surface of electronic components to enhance sealing, curing, and insulation effects. A window mirror is installed outside the optical lens to enhance protection for the optical lens. To reduce the impact of a large amount of shock and vibration on mechanical properties, the system design is optimized, the main optical components are reinforced, and epoxy resin is filled in the gap between the linear gradient filter and the detector to reduce the probability of damage to precision machinery. Considering the possibility of the instrument being carried on a drone, In the case of jitter, finite element simulation is performed to analyze the inherent vibration frequency of the equipment, and it is compensated to improve the vibration resistance of the equipment; considering the research on device structure distribution optimization and heat dissipation, the influence of temperature on the entire system is complex and comprehensive. Temperature changes will affect the circuit response time, electronic device performance, etc., and even cause certain changes in the laser pulse waveform, which will affect the ranging accuracy. Low-temperature drift models are selected for key resistors, capacitors and crystal oscillators, and heat dissipation treatment is carried out on the heat-generating chip to keep it within the normal working temperature range. Temperature compensation calibration is performed on the laser that is most affected by temperature to improve the environmental adaptability of the device.

[0087] In this embodiment, the device is mounted on a drone and acquires the three-dimensional coordinate information of a large-area target, the signal strength value recorded by the laser radar when receiving the reflected echo, and the spectral information of the selected band through a push-scan method.

[0088] Based on the same technical concept as the above embodiments, the present invention provides a drone equipped with the above-mentioned programmable hyperspectral lidar device. Example

[0089] This embodiment provides a programmable hyperspectral lidar device, which is mounted on an unmanned aerial vehicle (UAV) platform. Through hardware integration and intelligent algorithm collaborative design, it can achieve real-time acquisition, processing, and registration of laser point clouds and hyperspectral images. The device mainly consists of the following modules:

[0090] 1. Optical module

[0091] A 1550nm fiber laser is used to output a laser beam with a divergence angle of 0.2mrad~0.5mrad through a collimating beam expander, and multi-angle coverage is achieved through four tower mirrors.

[0092] The beam splitter coated with a 1550nm anti-reflection coating reflects the laser echo to the receiving system, and transmits the non-laser wavelength (400~1000nm) to the hyperspectral camera.

[0093] Non-laser receiving system: It uses an array CMOS detector (64×128 pixels, pixel size 5μm), with a linear gradient filter installed at the front end. The central wavelength changes linearly along the thickness of the filter, and spectral and spatial synchronous sampling is achieved through UAV push scanning.

[0094] like Figure 4 As shown, for the image data obtained by the hyperspectral camera, the drone push-scans continuously sample the spectral and spatial regions along the flight direction. The user can independently select the required window position and spectral wavelength width to achieve arbitrary programmable acquisition of thousands of spectral channels. The image information of several spectral channels obtained is then reconstructed through the spectral separation method to obtain different spectral images corresponding to the target scanning area.

[0095] Four-sided tower mirror: A brushless motor drives the four-sided tower mirror (four groups of 45° reflecting surfaces, with slight differences in reflection angle between each surface), achieving high-speed multi-angle scanning, and combined with a dynamic balancing design to ensure speed stability.

[0096] 2. Real-time signal processing module

[0097] Laser echo processing unit: A transimpedance amplifier circuit group (multiple sets of parallel resistors and switching circuits) dynamically adjusts gain, and combines a high-speed ADC and FPGA to implement waveform filtering, interpolation, and Gaussian fitting, achieving a ranging accuracy of 2mm@100m.

[0098] Hyperspectral image processing unit: Based on the user-selected window position (any of 32 bands within 400-1000 nm) and spectral width, spectral separation method is used to reconstruct characteristic images and reduce data redundancy.

[0099] 3. Data integration module

[0100] Time synchronization unit: Based on the satellite second pulse training local oscillator, combined with direct digital frequency synthesis technology (DDS), it achieves sub-microsecond time synchronization of multiple sensors (error <0.0001s).

[0101] Target calibration unit: Using black and white checkerboard targets on the ground (placed every 20 meters along the route), the FAST operator is used to extract feature points with the same name, the FLANN algorithm is used for matching, and the least squares method is used to calculate the similarity transformation parameters. The spectral image is mapped to the point cloud coordinate system, and bilinear interpolation is used to assign spectral information to the laser point cloud (with a registration accuracy of 0.1 pixel). This is done until each laser scanning point is assigned point cloud spectral information. Each frame of point cloud data with spectral information is then matched with the position and posture data at the corresponding timestamp.

[0102] Conversion unit: used to convert point cloud data from the lidar coordinate system to the global coordinate system based on the position information and attitude angle information provided by the position and attitude data.

[0103] 4. Highly environmentally adaptable hardware system

[0104] Waterproof and dustproof packaging structure: Epoxy resin is coated on the surface of electronic components and the gap between filters, and the optical lens is equipped with an external protective window.

[0105] Anti-vibration design: Finite element simulation optimizes the natural frequency, reinforces optical components, and uses near-zero dispersion, full-spectrum transparent glass materials to reduce temperature sensitivity.

[0106] Specific implementation: The laser emits a wavelength of 1550nm. The seed light emitted by the laser diode is split into two paths: one path is directly output as a reference light, and the other path is amplified by an erbium-doped fiber amplifier to amplify the main light signal. After the main light signal is output from the optical fiber, it is collimated by a collimating beam expander to obtain a laser beam with a divergence angle of 0.2mrad to 0.5mrad. The laser beam is deflected by a reflector to the scanning structure. The laser emission direction after the reflector deflection is coaxial with the receiving optical axis. The scanning mirror uses a four-sided tower mirror and is driven by a DC brushless motor for scanning. The beamsplitter is coated with a 1550nm wavelength anti-reflection coating. The echo with a wavelength of approximately 1550nm is reflected into the laser receiving optical system, and the remaining wavelengths are transmitted to the hyperspectral camera system, achieving a coaxial design of the laser receiving optical axis and the visible light receiving optical axis. Ultimately, a three-axis coaxial design is achieved for the laser emission optical axis, the laser receiving optical axis, and the visible light receiving optical axis.

[0107] The non-laser receiving system also features a free-form surface with a large aperture ratio. It utilizes near-zero dispersion, full-spectrum transparent glass materials and a telecentric optical path for spectral offset correction. The distortion across the entire field of view is less than 0.035%. At the same optical aperture, it offers higher luminous flux, enabling imaging and detection of weak signals. The focal length of the hyperspectral camera module's focusing optical system is 8mm, while the array size of the area array CMOS detector is 64*128, with a pixel size of 5 microns, ensuring that each pixel contains at least one laser spot at any flight altitude. A linear filter is mounted on the CMOS detector, and fine spectral characteristic detection and calibration technology is employed. A high-precision monochromator is used to detect the spectral information obtained by the gradient filter's spectroscopic analysis. The wavelength, relative light intensity, resolution, and linearity are calibrated, enabling the development of a high-dispersion, fine-tuned gradient filter with a spectral detection band range of 400-1000nm and a spectral resolution of 5nm. The system combines gradient filter spectroscopic technology with digital domain time delay integration technology based on CMOS arrays. By collecting imaging results of the same target at different times, the number of electrons in multiple imaging is added together to improve the signal-to-noise ratio. The user can independently select the required window position and spectral wavelength width to achieve 32 spectral segments and integral series imaging in the 400~1000nm spectral range. The image signal energy is increased without increasing the aperture of the imaging spectrometer, thereby improving the image accuracy. The image information of the characteristic spectrum is then reconstructed through the non-negative matrix decomposition method to obtain the spectral image corresponding to the target scanning area.

[0108] A high-precision time base establishment method is proposed, which combines direct digital frequency synthesis controlled by field programmable logic gate array with satellite receiver second pulse training. The clock output interface provides a clock source for multiple sensors in the system, improving the time synchronization accuracy to sub-microsecond level. This enables high-precision time synchronization of multiple sensors such as laser scanners, hyperspectral cameras, inertial navigation units, and Beidou navigation units. High-accuracy timestamps are established for the scanned spectral image data and laser point cloud data. The time error between sensors is less than 0.0001s, meeting the subsequent high-precision alignment requirements of spectral data and point cloud data.

[0109] Targets are evenly distributed on the ground, with a target placed every 20 meters at the edge of each route, ensuring that there are more than three targets on each route and that the target positions are not collinear. A black and white checkerboard pattern is selected as the target, ensuring that its center position can be clearly identified in both the point cloud and the image. The system splits the spectral images of multiple bands required by the user into multiple single-band spectral images spliced ​​in chronological order, splices the point cloud data according to the scanning time sequence provided by the position and posture data, and rasterizes and interpolates the point cloud into a point cloud grayscale image. The point cloud grayscale image and the spectral image are aligned based on the same-name feature points on the image, and the spectral image is converted to the point cloud coordinate system. The spectral information is then assigned to each laser scanning point through interpolation processing, with a pixel registration accuracy of up to 0.1 pixel, giving each laser scanning point three-dimensional position information, intensity information, and user-defined spectral information for all bands. Each frame of point cloud data with spectral information is matched with the position and attitude data at the corresponding timestamp. Based on the position information and attitude angle information (pitch and roll angles) provided by the position and attitude data, the point cloud data is converted from the lidar coordinate system to the global coordinate system to achieve reconstruction and alignment of the scanned point cloud with the spectral image of the band required by the user.

[0110] The working principle and implementation steps of the above embodiment are as follows:

[0111] 1. The laser emits pulses, and the spectroscope separates the echo signals: the 1550nm laser echo is used for ranging, and the non-laser wavelength is transmitted to the hyperspectral camera; the four-sided tower mirror rotates at high speed to achieve multi-angle scanning coverage, and the spatial posture is recorded in combination with the position and posture data.

[0112] 2. After the laser echo is amplified by transimpedance, the ranging value is extracted through digital waveform processing (FIR filtering, CIC interpolation, and dynamic threshold detection). The hyperspectral image is intercepted according to the user-selected band, and the signal-to-noise ratio is improved using time delay integration technology to output the reconstructed characteristic spectral image.

[0113] 3. The satellite timing signal tames the local oscillator, and DDS technology outputs fractional-second pulses to ensure the clock synchronization of lidar, hyperspectral camera, and position and attitude data.

[0114] 4. Extract the same-name feature points from the grayscale and spectral images of the ground target (black and white checkerboard). Use similarity transformation to align the spectral information and map it to the point cloud. Use bilinear interpolation to assign 3D coordinates, intensity, and spectral attributes to the point cloud. Combined with the position and posture data, the data is converted to a global coordinate system to generate "3D coordinate + spectrum" multimodal data.

[0115] Based on the same technical concept of the above embodiments, the present invention provides, on one hand, a drone equipped with a programmable hyperspectral laser radar device, including a drone body, on which the programmable hyperspectral laser radar device is provided.

[0116] like Figure 6 As shown, based on the same technical concept of the above embodiment, the present invention provides a real-time registration method for a programmable hyperspectral laser radar device, including the following steps:

[0117] The optical module emits laser light and receives echo signals, and separates the laser wavelength echo signal and the non-laser wavelength echo signal;

[0118] Real-time processing of laser echo signals and hyperspectral image data through real-time signal processing modules;

[0119] Establish sub-microsecond multi-sensor time synchronization benchmark through data integration module;

[0120] Generate point cloud grayscale images and hyperspectral images through the target, extract feature points with the same name and align them;

[0121] Convert the spectral image to the point cloud coordinate system and interpolate to give the laser point cloud spectral information;

[0122] Each frame of point cloud data with spectral information is matched with the position and posture data at the corresponding timestamp, and the aligned point cloud is converted to the global coordinate system based on the position and posture data.

[0123] The above technical solution covers the entire process from laser emission and echo signal reception, real-time signal processing, multi-sensor time synchronization benchmark establishment, target generation and registration, spectral image conversion and point cloud spectral information assignment, and finally, global coordinate system conversion based on position and posture data. This forms a systematic, efficient, and accurate real-time registration solution for laser point clouds and hyperspectral imagery. Through the close connection and collaborative work of each step, an integrated operation is achieved from data acquisition to processing and fusion, ensuring data accuracy and consistency, providing users with more comprehensive and accurate geospatial information, and meeting the needs of different fields for high-precision, multi-dimensional data.

[0124] In this embodiment, establishing a sub-microsecond multi-sensor time synchronization benchmark includes:

[0125] The local oscillator is tamed by the satellite receiver to adjust its frequency and phase to be consistent with the satellite signal;

[0126] Direct digital frequency synthesis technology is used to output the fractional-second signal, controlling the multi-sensor clock synchronization error to be less than 0.0001 seconds.

[0127] In this embodiment, when interpolating spectral information:

[0128] The bilinear interpolation method is used to map the spectral information of the spectral image pixel to the nearest laser point cloud;

[0129] The registration accuracy reaches 0.1 pixel, and each point cloud contains three-dimensional coordinates, intensity and user-defined band spectral information.

[0130] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A programmable hyperspectral laser radar device, characterized in that: include: An optical module is used to emit laser light onto a target. The echo signal reflected by the target is incident on a laser receiving system and a non-laser receiving system respectively through a spectroscope. The non-laser receiving system uses a linear gradient filter and an array detector to receive the non-laser echo signal. A real-time signal processing module is used to process the laser echo signal and the non-laser echo signal in real time. The processing of the non-laser echo signal includes: selecting a window position and a spectral wavelength width, obtaining a plurality of arbitrarily programmable spectral channels, and reconstructing the image information of the plurality of spectral channels by spectral separation method to obtain different spectral images corresponding to the target scanning area; The data integration module is used to perform real-time registration and integration of processed laser point clouds and hyperspectral images.

2. The programmable hyperspectral laser radar device according to claim 1, characterized in that: The optical module further comprises: The tower mirror has four 45° inclined reflecting surfaces, which is used to achieve multi-angle scanning of the same target.

3. The programmable hyperspectral laser radar device according to claim 1, characterized in that: The laser echo signal is processed in real time, including: converting the laser echo signal into a current signal and sending it into a transimpedance amplifier circuit with a multi-rate transimpedance amplifier gain, selecting the channel output that is not saturated and has the highest peak voltage, and performing analog-to-digital conversion on the output voltage.

4. The programmable hyperspectral laser radar device according to claim 1, characterized in that: The data integration module includes: The time synchronization unit is used to provide a sub-microsecond synchronized clock source for multiple sensors on the device; The target calibration unit is used to extract the same-name feature points of the point cloud grayscale image and the hyperspectral image through the ground target, complete the registration using similarity transformation, and assign the point cloud spectral information to the nearest laser scanning point through interpolation. This is done until each laser scanning point is assigned point cloud spectral information, and each frame of point cloud data with spectral information is matched with its position and posture data at the corresponding timestamp. The conversion unit is used to convert the point cloud data from the lidar coordinate system to the global coordinate system according to the position information and attitude angle information provided by the position and attitude data.

5. The programmable hyperspectral laser radar device according to claim 4, characterized in that: The target calibration unit is further configured to execute the following instructions: Identify the coordinates of the target center point in the point cloud grayscale image coordinate system and the hyperspectral image coordinate system, use the fast approximate nearest neighbor algorithm to match the feature points with the same name, use the least squares method and similarity transformation to achieve the registration of the two images after the matching of the same name features, and convert the spectral image into the point cloud coordinate system.

6. The programmable hyperspectral laser radar device according to claim 1, characterized in that: The device is mounted on a drone and acquires the three-dimensional coordinate information of a large-area target, the signal strength value recorded by the laser radar when receiving the reflected echo, and the spectral information of the selected band through a push-scan method.

7. A drone, characterized in that: A programmable hyperspectral lidar device is provided according to any one of claims 1 to 6.

8. A real-time registration method for a programmable hyperspectral laser radar device, characterized in that: The following steps are involved: The laser is emitted to the target through the optical module. The echo signal reflected by the target is incident on the laser receiving system and the non-laser receiving system respectively through the beam splitter. The non-laser receiving system uses a linear gradient filter and an array detector to receive the non-laser echo signal. Real-time processing of laser echo signals and hyperspectral image data through real-time signal processing modules; Establish sub-microsecond multi-sensor time synchronization benchmark through data integration module; Generate point cloud grayscale images and hyperspectral images through the target, extract feature points with the same name and align them; Convert the spectral image to the point cloud coordinate system and interpolate to give the laser point cloud spectral information; Each frame of point cloud data with spectral information is matched with the position and posture data at the corresponding timestamp, and the aligned point cloud is converted to the global coordinate system based on the position and posture data.

9. The real-time registration method of a programmable hyperspectral laser radar device according to claim 8, characterized in that: The establishment of a sub-microsecond multi-sensor time synchronization benchmark includes: The local oscillator is tamed by the satellite receiver to adjust the frequency and phase of the local oscillator to be consistent with the satellite signal; Direct digital frequency synthesis technology is used to output the fractional-second signal, and the multi-sensor clock synchronization error is controlled to be less than the target time.

10. The real-time registration method of a programmable hyperspectral laser radar device according to claim 8, characterized in that: Interpolation gives the laser point cloud spectral information, including: The bilinear interpolation method is used to map the spectral information of the spectral image pixel to the nearest laser point cloud; The registration accuracy reaches the predetermined pixel, and each point cloud contains three-dimensional coordinates, intensity and user-defined band spectral information.

Images