A hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information

Abstract

The present application relates to a kind of fusion hyperspectral three-dimensional reconstruction method of laser radar, vision and inertial information, mainly used to solve the technical problems that using hyperspectral depth sensor combines hyperspectral information and depth data, lack of synchronous mechanism in fusion registration, difficult to realize space-time alignment, real-time and general poor, and unable to carry out data acquisition in equipment motion state.A kind of fusion hyperspectral three-dimensional reconstruction method of laser radar, vision and inertial information, by fusing industrial camera, laser radar and inertial measurement unit IMU acquisition multi-source data, reconstructs the three-dimensional RGB point cloud model of target scene;Extract the RGB information and geometric structure information of each point in three-dimensional RGB point cloud model, expand RGB channel, map the spectral information obtained after channel expansion to geometric structure information, generate hyperspectral three-dimensional point cloud model.

Description

TECHNICAL FIELD

[0001] The application relates to a hyperspectral three-dimensional reconstruction method, in particular to a hyperspectral three-dimensional reconstruction method fusing laser radar, vision and inertial information. BACKGROUND

[0002] Hyperspectral imaging technology can capture spectral information in multiple narrow bands. Compared with traditional RGB images, it provides more accurate spectral features, making it show excellent application value in many fields. However, the current hyperspectral imaging system usually stays at the two-dimensional level, lacks the three-dimensional geometric information of the target, can only provide image-based analysis, and cannot directly monitor and evaluate the target in all directions. In recent years, in the field of frontier research such as agricultural phenotype, the technology combining hyperspectral information and three-dimensional depth data is gradually becoming a new development trend.

[0003] Currently, there are two methods to combine hyperspectral information and depth data. One is to fuse hyperspectral images and depth sensor data to obtain three-dimensional hyperspectral information. Nieto et al. [Nieto J, Monteiro S, Viejo D. 3D geological modelling using laser and hyperspectral data [C] / / 2010 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2010: 4568-4571] combined laser radar and hyperspectral imaging to construct a three-dimensional geological model, but the registration accuracy was insufficient in complex terrain. Brell et al. [Brell M, Segl K, Guanter L, et al. 3D hyperspectral point cloud generation: Fusing airborne laser scanning and hyperspectral imaging sensors for improved object-based information extraction [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 149: 200-214] proposed a reconstruction method by integrating hyperspectral cameras and laser radars on a UAV platform. Although progress was made in classification accuracy, it was pointed out that due to the significant difference in resolution between point clouds and spectral images, parallax errors were prone to occur during the fusion process, which was the main difficulty restricting the system accuracy. Edmonds et al. [Edmonds M, Yi J, Singa N K, et al. Generation of high-density hyperspectral point clouds of crops with robotic multi-camera planning [C] / / 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE). IEEE, 2019: 1475-1480] designed a system that fused multispectral cameras and depth cameras, but the accuracy was affected by the robot's moving speed and positioning system during the mapping process, which easily caused inconsistent data in dynamic scenes.

[0004] Another method of combining hyperspectral information with depth data is to use a hyperspectral depth sensor. The most direct method of using a hyperspectral depth sensor is to use a professional hyperspectral lidar imaging device. However, due to the poor flexibility of the hyperspectral lidar imaging device, it is difficult to freely adjust the waveband range, field of view, and other parameters according to different applications. Moreover, the hardware design is complex and the price is high, which limits large-scale application and popularization. Patent CN105675549B obtains multispectral images of crop canopy through multiple waveband visible / near-infrared cameras, simultaneously reconstructs the three-dimensional structure of the canopy through spatial interpolation, and then registers the image with the reconstructed model to generate multispectral point clouds containing elevation information. Patent CN117671228A registers and fuses hyperspectral images and depth images to generate canopy hyperspectral three-dimensional point clouds. CN119672215A obtains multispectral image data and corresponding LiDAR point cloud data of a target area, constructs a three-dimensional multispectral point cloud fusion model, inputs the shallow and deep multispectral feature maps and the original LiDAR point cloud into the model, and the model outputs a three-dimensional point cloud with spectral properties, i.e., each spatial point in the original point cloud is assigned a corresponding multispectral reflectance value. However, the fusion registration part lacks a synchronization mechanism, making it difficult to achieve spatio-temporal alignment, poor real-time performance and universality, and unable to collect data in a device motion state. SUMMARY

[0005] The purpose of the present application is to solve the technical problems of using a hyperspectral depth sensor to combine hyperspectral information with depth data, lack of synchronization mechanism in fusion registration, difficulty in achieving spatio-temporal alignment, poor real-time performance and universality, and inability to collect data in a device motion state, and to propose a hyperspectral three-dimensional reconstruction method that fuses lidar, vision, and inertial information.

[0006] To solve the above technical problems, the technical solution provided by the present application is as follows:

[0007] A hyperspectral three-dimensional reconstruction method that fuses lidar, vision, and inertial information, characterized by the following steps:

[0008] Step 1, reconstructing a three-dimensional RGB point cloud model

[0009] Step 11, using an industrial camera to collect images of the target scene, using a lidar to collect original point clouds of the target scene, and using an IMU to collect pose data of the lidar and the industrial camera. The pose data collected by the inertial measurement unit (IMU) is used to provide motion compensation when the industrial camera and the lidar are moving to collect data.

[0010] Step 12, spatially aligning the industrial camera and the lidar.

[0011] Step 13, time synchronization is realized by using the PPS pulse signal and the GPRMC data based on the laser radar and the industrial camera;

[0012] Step 14, the laser radar combines the IMU to build a frame by frame map in a tightly coupled manner; meanwhile, the industrial camera combines the IMU to obtain the RGB information of the current frame and project the RGB information to the original point cloud of the current frame; until all frames are traversed, the global map P containing the RGB information of the target scene is obtained RGB , that is, an RGB three-dimensional point cloud model;

[0013] Step 2, reconstructing a hyperspectral three-dimensional point cloud model

[0014] Step 21, extracting the global map P RGB , the RGB information C of each point N×3 , and the geometric structure information S N×3 ,

[0015] , and mapping the RGB information C N×3 and the geometric structure information S N×3 to a two-dimensional image to form a picture I H×W×3 ; wherein N is the total number of point clouds;

[0016] Step 22, using a deep learning method to perform spectral reconstruction on the picture I H×W×3 , expanding the 3 channels of the RGB corresponding to each pixel to U channels to obtain a hyperspectral image Y∈R H*W*U ;

[0017] Step 23, training the hyperspectral image Y∈R H*W*U using the ARAD-HS dataset;

[0018] Step 24, mapping the spectral information in the trained hyperspectral image Y∈R H*W*U to the geometric structure information S N×3 to generate a hyperspectral three-dimensional point cloud model, and completing the hyperspectral three-dimensional reconstruction.

[0019] Further, the step 12 specifically is:

[0020] Step 12, jointly calibrating the industrial camera and the laser radar to make the corner points in the original point cloud and the image have a one-to-one correspondence;

[0021] Given four sets of laser radar-industrial camera corresponding corner points, the transformation rotation matrix R and the translation vector t are adjusted through a nonlinear optimization library to calculate the re-projection error ε(R,t) of all points in the original point cloud in the pixel coordinate system of the image. The optimization objective function of the re-projection error ε(R,t) is:

[0022]

[0023] wherein X i is the three-dimensional position of the i-th point in the LIDAR coordinate system, x i is the two-dimensional coordinate of the i-th point in the pixel coordinate system of the image, and K is the intrinsic parameter of the industrial camera;

[0024] obtaining the rotation matrix R and the translation vector t corresponding to the minimum re-projection error ε(R, t) so as to realize the spatial alignment of the LIDAR and the industrial camera.

[0025] Further, the step 13 is specifically:

[0026] Step 131, using the single-chip microcomputer to convert the PPS pulse signal output by the GPS module into a 1Hz square wave signal and a 10Hz square wave signal, and transmitting them to the LIDAR and the industrial camera respectively for realizing time triggering;

[0027] Step 132, after the LIDAR receives the rising edge of the k-th 1Hz square wave signal, combining the UTC time information in the GPRMC data received by the LIDAR the internal time stamp and the UTC time information aligning and taking as the reference time t base , the LIDAR writes the reference time t base into the shared memory shm; wherein:

[0028]

[0029] Step 133, the shared memory shm transmits the reference time t base to the industrial camera; the industrial camera finds the frame of image closest to the reference time t base , denoted as the j * -th frame, and resets the time of the frame to the reference time t base , i.e.

[0030]

[0031] realizing the time synchronization of the two sensors; wherein:

[0032]

[0033] is the time stamp of the j -th frame of image of the industrial camera.

[0034] Further, the step 14 is specifically:

[0035] Step A1, the LIDAR collects the first frame of original point cloud, and performs time compensation on the original point cloud to eliminate motion distortion;

[0036] Step A2, constructing a local map of the first frame according to the original point cloud of the first frame as a constructed local map;

[0037] Step A3, the laser radar collects the original point cloud of the next frame as the current frame original point cloud; time compensation is performed on the current frame original point cloud to eliminate motion distortion; the IMU collects the laser radar pose data of the current frame and performs state pre-integration to obtain the current pose of the laser radar;

[0038] Step A4, using a point-to-plane ICP algorithm to register the current frame original point cloud with the constructed local map, performing state updating on the current pose to obtain the optimal pose estimation result of the laser radar of the current frame;

[0039] Step A5, correcting the current frame original point cloud according to the optimal pose estimation result, fusing the corrected current frame original point cloud with the constructed local map to generate a local map of the current frame, and taking it as a new constructed local map; at the same time, the industrial camera combined with the IMU obtains the corresponding RGB value of each point in the current frame original point cloud;

[0040] Step A6, return to step A3 until all frames are traversed, complete the three-dimensional RGB point cloud model reconstruction of the target scene, and obtain a global map P containing RGB information of the target scene RGB .

[0041] Further, in step A5, the industrial camera combined with the IMU obtains the corresponding RGB value of each point in the current frame original point cloud, which is specifically:

[0042] Step B1, the industrial camera combined with the IMU constructs a visual inertial odometer, which includes a frame-frame visual inertial odometer and a frame-map visual inertial odometer; the industrial camera collects the current frame image I k ; the map in the frame-map visual inertial odometer is a visual sparse map;

[0043] Step B2, in the frame-frame visual inertial odometer, first extract a group of corner points in the previous frame image I k-1 using the FAST algorithm Then use the Lucas-Kanade optical flow method to track the corner points in the current frame image I k to obtain the corresponding pixel positions

[0044] Step B3, the three-dimensional coordinates of each pixel in the previous frame image I k-1 are P i , minimizing the 3D-2D re-projection error of the current frame image I k to obtain the pose parameters R kt k Then, based on the pose parameters R at this time... k t k Calculate the current frame pose T of the industrial camera k =[R k |t k ]; where, minimize the current frame image I k The 3D–2D reprojection error is expressed by the following formula:

[0045]

[0046] Among them, R k Let t be the rotation matrix of the current frame. k Let be the translation vector of the current frame; π is the projection function of the industrial camera, and the formula for calculating π is:

[0047]

[0048] Where (X,Y,Z) are the coordinates of a point in space, f x f y c is the lens focal length. x c y Principal point coordinates;

[0049] Step B4: Based on the current frame pose T of the industrial camera k Correcting pixel position F k Obtain the registered current frame image I' k ;

[0050] Step B5: In frame-map visual inertial odometry, the visual inertial odometry uses a visual sparse map as a reference and selects a set of sparsely distributed points from it. Figure Three Dimension And projected onto the registered current frame image I' k Establish a photometric error model:

[0051]

[0052] Among them, C j For the land Figure Three Dimension P j The corresponding RGB information, I1 k For the current frame image I' k The pixel grayscale value, r j This is photometric error;

[0053] Step B6: Minimize photometric error r j Obtain the pose parameters R at this time. k t k Then, based on the photometric error r j Minimum pose parameter Rk , t k , compute a new current pose of the industrial camera as the best pose of the current frame of the industrial camera; wherein the photometric error r j is minimized

[0054]

[0055] Step B7, project the current frame raw point cloud P = {p i} collected by the LiDAR to the current frame image I' k according to the best pose of the current frame of the industrial camera. i Each point p k in the raw point cloud corresponds to a pixel position

[0056]

[0057] where T CL is the extrinsic matrix of the LiDAR to the industrial camera.

[0058] Step B8, use bilinear interpolation to obtain the RGB value RGB of each pixel position i :

[0059]

[0060] where Interp bilinear is the bilinear interpolation function.

[0061] Step B9, assign the RGB value RGB of each pixel position i to the corresponding point p i in the raw point cloud.

[0062] Further, step A4 is specifically:

[0063] Construct the point-to-plane residual term e i :

[0064]

[0065] where P i is the raw point cloud of the i-th frame, q i-1 is the raw point cloud in the local map constructed in the i-1-th frame, is the plane normal vector of the plane where q i-1 is located; R is a rotation matrix and t is a translation vector.

[0066] Use the Lie group and Lie algebra method to calculate the residual term e iLinear error model E of point-to-plane ICP i :

[0067] E i = H i δξ + e i

[0068] where H i is the i-th frame Jacobian matrix, ξ is Lie algebra, δ is the increment of Lie algebra ξ and translation vector t;

[0069] Minimize linear error model E i , and the linear error model E i is minimized when H i is taken as the observation input H; combined with the state pre-integration in step A3, input into the error state iterative Kalman filter for state update and optimization, and the formula of Kalman filter gain K is:

[0070]

[0071] where the observation input H is the derivative of ICP observation residual to state, R1 is the observation noise covariance, P - is the state prediction value; the obtained Kalman filter gain K is substituted into the state increment calculation:

[0072]

[0073] where z is the actual observation value of point-to-plane residual from ICP calculation, is the ICP observation residual derived from the predicted state; x is the state increment; according to the formula , the state of the current pose corresponding to the i-th frame is updated:

[0074]

[0075] where, is the attitude; is the velocity, δv is the velocity increment; is the position, δp is the position increment; is the gyroscope bias, b g is the gyroscope bias increment; is the accelerometer bias, b a is the accelerometer bias increment; the left side of the arrow is the updated state, which is the optimal pose estimation result.

[0076] Further, the step 22 is specifically:

[0077] Step 221, the deep learning method adopts the MST++ method, including a plurality of single-stage SSTs in cascade; each SST adopts a U-shaped structure, including an encoder, a bottleneck layer and a decoder;

[0078] Picture I H×W×3 The corresponding RGB image is I ∈ R H*W*3 , and the RGB image I ∈ R H*W*3 is input into a 3x3 convolutional layer to realize feature embedding and generate an initial feature map X0;

[0079] Step 222, the encoder includes a 4x4 first convolutional layer with a step of 4, N1 spectral attention blocks SABs, a 4x4 second convolutional layer with a step of 4, a 4x4 third convolutional layer with a step of 4, and N2 spectral attention blocks SABs, which sequentially perform first downsampling, first spectral feature extraction, second downsampling, third downsampling and second spectral feature extraction on the initial feature map X0 to obtain a first feature map and output to the bottleneck layer;

[0080] Step 223, the bottleneck layer includes N3 spectral attention blocks SABs, which sequentially perform spectral feature extraction on the first feature map to obtain a second feature map and output to the decoder;

[0081] Step 224, the decoder and the encoder are in a symmetrical structure, including a first deconvolutional layer, a second deconvolutional layer and a third deconvolutional layer, which sequentially perform upsampling on the second feature map; the first deconvolutional layer, the second deconvolutional layer and the third deconvolutional layer are also connected with the first convolutional layer, the second convolutional layer and the third convolutional layer in a skip connection manner to fuse the features of the encoder; the output channel number U of the third deconvolutional layer is 31, and a reconstructed hyperspectral image Y ∈ R H*W*31 is output; the hyperspectral image Y ∈ R H*W*31 covers a wavelength range of 400-700 nm with a spectral resolution of 10 nm.

[0082] Further, the geometric structure information S N×3 in step 21 is mapped as follows:

[0083]

[0084] The RGB information C N×3 is mapped as follows:

[0085]

[0086] wherein 255 is the maximum brightness value of a pixel; H is the image height, W is the image width, and 3 in Nx3 represents 3 RGB channels;

[0087] In step 23, the ARAD-HS dataset contains 510 groups of data, each group of data containing 3 channels of RGB information and corresponding 31 channels of hyperspectral data.

[0088] Further, in step 222, the spectral attention block SAB includes a multi-head self-attention mechanism S-MSA, an FFN, and a normalization layer.

[0089] After the first downsampling, the feature map is X∈R H*W-C , which is input into N1 spectral attention blocks SABs and flattened into a matrix X∈R HW*C , and the query Q matrix, the key Key matrix, and the value V matrix are obtained through linear mapping:

[0090]

[0091] where W Q ,W Key ,W V are learnable parameters.

[0092] Then, the features are divided into multiple subspaces in the channel dimension, and each head of the multi-head self-attention mechanism S-MSA independently performs attention calculation on a subspace:

[0093]

[0094] head j =V j A j

[0095] where A j is the attention weight matrix of the jth head head j , σ j is a learnable scaling factor for adapting the spectral density distribution of different bands; is the key Key matrix of the jth head head j , Q j is the query Q matrix of the jth head, and V j is the value V matrix of the jth head head j .

[0096] Finally, each head head is spliced to form an output through linear transformation and position encoding:

[0097]

[0098] where f p represents the spectral position encoding generated by two layers of 3×3 deep separable convolution.

[0099] Further, step 3 performance evaluation is also included, specifically:

[0100] Step 31, the hyperspectral image Y obtained in step 22 is expanded to obtain a hyperspectral image Y H*W*31 As a predicted value Data of the target scene collected by the push-broom spectrometer is used as the true value Y i ;

[0101] The predicted value is calculated The root mean square error RMSE between the predicted value and the true value Y i :

[0102]

[0103] Where Y i is the true value, is the predicted value, and N is the number of all values;

[0104] The predicted value is calculated The mean absolute error MAE between the predicted value and the true value Y i :

[0105]

[0106] The predicted value is calculated The goodness of fit coefficient GFC between the predicted value and the true value Y i :

[0107]

[0108] Step 32, if the root mean square error RMSE, the mean absolute error MAE and the goodness of fit coefficient GFC meet the requirements, the hyperspectral three-dimensional reconstruction is completed; if they do not meet the requirements, return to step 22, change the number of expansion channels U, and re-perform spectral reconstruction.

[0109] Compared with the prior art, the beneficial effects of the present application are:

[0110] 1. The present application is a hyperspectral three-dimensional reconstruction method fusing laser radar, vision and inertial information, which reconstructs a three-dimensional RGB point cloud model of a target scene by fusing multi-source data collected by an industrial camera, a laser radar and an inertial measurement unit IMU; extracts the RGB information and geometric structure information of each point in the three-dimensional RGB point cloud model, expands the RGB channel, maps the spectral information obtained after expansion of the channel to the geometric structure information, and generates a hyperspectral three-dimensional point cloud model.

[0111] 2.The hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information, solves the technical bottleneck that multi-source data is difficult to realize space-time alignment and fusion in a moving state, avoids the problems of high equipment cost and complex calibration process existing in traditional hyperspectral and depth fusion systems, and has good flexibility and practicability.The technology can be applied to the fields of visible and near-infrared detection such as agricultural and forestry monitoring, ecological environment monitoring, fine topographic mapping and resource investigation, and has good popularization value and application prospect.

[0112] 3.In the hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information, the spatial alignment of the industrial camera and the lidar is realized by optimizing the re-projection error of all points in the original point cloud in the pixel coordinate system of the image; and the time synchronization of the lidar and the industrial camera is realized based on the PPS pulse signal and the GPRMC data. High coupling is realized in space synchronization and time synchronization, and the consistency of multi-source data in space-time is ensured, which provides a solid foundation for real-time mapping.

[0113] 4.In the hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information, in the process of reconstructing the RGB three-dimensional point cloud model, the lidar combines the IMU to obtain the optimal pose estimation result of each frame of the lidar by using the point-to-plane ICP algorithm, and then the corresponding frame of the original point cloud is corrected according to the optimal pose estimation result. Thus, the lidar can adapt to stable operation in a handheld or mobile state, has excellent flexibility and deployment convenience, and is especially suitable for plant monitoring and environment perception tasks in complex outdoor environments.

[0114] 5.The hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information restores the spectral information of the target by using a spectral reconstruction technology, avoids the step of directly fusing the hyperspectral data and the depth information, thereby significantly reducing the overall hardware cost and complexity, improving the data acquisition and processing efficiency, and having good practicability and popularization value. BRIEF DESCRIPTION OF DRAWINGS

[0115] Figure 1 The flowchart of the hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information;

[0116] Figure 2 The calibration board collected by the industrial camera and the lidar in the hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information; wherein (a) is the calibration board collected by the industrial camera, and (b) is the calibration board collected by the lidar.

[0117] Figure 3An effect picture of the point cloud coloring in step 12 of the embodiment of the application, wherein the effect pictures from left to right are respectively effect pictures of the point cloud coloring using the corresponding internal parameter and external parameter matrices of 1-5m;

[0118] Figure 4 An RGB three-dimensional point cloud model obtained in step 14 of the embodiment of the application, wherein the target scenes of plant 1-3 from top to bottom are respectively cypress, pine forest and mountain jujube;

[0119] Figure 5 A hyperspectral three-dimensional point cloud model diagram of the same target scene reconstructed under different wave bands in step 24 of the embodiment of the application;

[0120] Figure 6 A curve comparison diagram between the predicted value and the true value in step 3 of the embodiment of the application. DETAILED DESCRIPTION

[0121] The application will be further described below in combination with the drawings and embodiments.

[0122] The application is a hyperspectral three-dimensional reconstruction method fusing laser radar, vision and inertial information, in the embodiment, the target scene is three green trees, which are respectively cypress, pine forest and mountain jujube, as shown in the figure, comprising the following steps: Figure 1

[0123] Step 1, reconstructing a three-dimensional RGB point cloud model

[0124] Step 11, using an industrial camera, a laser radar and an inertial measurement unit (IMU) built-in the laser radar as sensors to realize high-precision three-dimensional reconstruction of a target object. The industrial camera is used to collect images of the target scene, the laser radar is used to collect original point clouds of the target scene, and the IMU is used to collect pose data of the laser radar and the industrial camera.

[0125] In the embodiment, a Hikvision robot industrial camera of model MV-CA013-A0UC and a Livox MID-360 laser radar are used, the IMU is a radar built-in IMU, and the industrial camera and the laser radar are rigidly fixed through a customized tool. In other embodiments, the industrial camera is replaced by a multispectral or hyperspectral camera, which is fused with the laser radar and the IMU, so that multispectral or hyperspectral point clouds containing near-infrared and infrared information can be directly reconstructed.

[0126] Step 12, spatially aligning the industrial camera and the laser radar; ​

[0127] To ensure the spatial consistency and time synchronization between sensors in the data fusion and reconstruction process, first, the industrial camera and the lidar are jointly calibrated. The key goal of joint calibration is to achieve spatial alignment, that is, to reproject the original point cloud into the pixel coordinate system of the image, so that the original point cloud and the corner points in the image have a one-to-one correspondence. As shown in the figure, (a) is the calibration board collected by the industrial camera, and (b) is the calibration board collected by the lidar. Figure 2

[0128] Given four sets of corresponding corner points of the lidar-industrial camera, the transformation rotation matrix R and the translation vector t are adjusted through the nonlinear optimization library Ceres Solver to calculate the re-projection error ε(R, t) of all points in the original point cloud in the pixel coordinate system of the image. The optimization objective function of the re-projection error ε(R, t) is:

[0129]

[0130] Where X i is the three-dimensional position of the i-th point in the lidar coordinate system, x i is the two-dimensional coordinate of the i-th point in the pixel coordinate system of the image, and K is the intrinsic parameter of the industrial camera.

[0131] The rotation matrix R and the translation vector t corresponding to the minimum re-projection error ε(R, t) are obtained. At this time, the rotation matrix R and the translation vector t are the extrinsic matrix that can make the lidar and the industrial camera achieve spatial alignment.

[0132] To ensure the stability of joint calibration at different distances, calibration is performed every meter in the range of 1-5m, and the point cloud color rendering method is used to verify the calibration effect. As shown in the figure, from left to right, the point cloud coloring effect is performed using the intrinsic and extrinsic matrix corresponding to 1-5m. Finally, the intrinsic and extrinsic matrix obtained by observing the data collected at a distance of two meters is used for the best coloring effect. Figure 3

[0133] Step 13, based on the PPS pulse signal and GPRMC data to realize time synchronization of lidar and industrial camera. The PPS pulse signal is a pulse signal generated once per second, and the GPRMC data contains UTC time information; Specifically:

[0134] Step 131, since the lidar and the industrial camera have different frequency requirements for the synchronization pulse, the STM32 single-chip microcomputer is used to convert the PPS pulse signal output by the GPS module into a 1Hz square wave signal and a 10Hz square wave signal, which are transmitted to the lidar and the industrial camera respectively, for time triggering;

[0135] ​​Step 132: After receiving the rising edge of the k-th 1Hz square wave signal, the lidar combines the UTC time information from the received GPRMC data. Mix internal timestamp with UTC time information Alignment, and as a reference time t base The lidar will use the reference time t base Write to the shared memory shm; where:

[0136]

[0137] Step 133: To align with the LiDAR, the shared memory shm will use the reference time t base Transmitted to industrial camera; industrial camera locates distance reference time t base The most recent frame, denoted as j-th. * Frame, and the time of that frame Reset to base time t base ,Right now

[0138]

[0139] Achieve time synchronization between two sensors; where:

[0140]

[0141] This is the timestamp of the j-th frame of the industrial camera image.

[0142] Step 14: The LiDAR, combined with the IMU, uses a tightly coupled approach to build a map frame by frame; simultaneously, the industrial camera, combined with the IMU, acquires the RGB information of the current frame and projects the RGB information onto the original point cloud of the current frame; this process continues until all frames are traversed, resulting in a global map P of the target scene containing RGB information. RGB , i.e., RGB 3D point cloud model;

[0143] Step 14 specifically refers to:

[0144] Step A1: The lidar acquires the first frame of the original point cloud and performs time compensation on the original point cloud to eliminate motion distortion.

[0145] Step A2: Construct a local map of the first frame based on the original point cloud of the first frame, as the constructed local map;

[0146] Step A3: The lidar acquires the original point cloud of the next frame as the original point cloud of the current frame; time compensation is performed on the original point cloud of the current frame to eliminate motion distortion; the IMU acquires the lidar pose data of the current frame and performs state pre-integration to obtain the current pose of the lidar.

[0147] Step A4, the current frame of the original point cloud is registered with the constructed local map by using a point-to-plane ICP algorithm (Point-to-Plane Iterative Closest Point), the state of the current pose is updated, and the optimal pose estimation result of the current frame of the laser radar is obtained; specifically:

[0148] Construct a point-to-plane residual term e i :

[0149]

[0150] where P i is the original point cloud of the i-th frame, q i-1 is the original point cloud in the constructed local map of the i-1-th frame, is the plane normal vector of the plane where q i-1 is located; R is a rotation matrix, and t is a translation vector;

[0151] Using the Lie group and Lie algebra method, the residual term e i of the i-th frame is converted into a linear error model E i of the point-to-plane ICP:

[0152] E i = H i δξ + e i

[0153] where H i is the Jacobian matrix of the i-th frame, corresponds to the partial derivative of the ICP residual with respect to the attitude, ξ is the Lie algebra, and δ is the increment of the Lie algebra ξ and the translation vector t;

[0154] Minimize the linear error model E i , and when the linear error model E i is minimized, the corresponding H i is taken as the observation input H; combined with the state pre-integration in step A3, input into an error state iterative Kalman filter (ESIKF) for state updating and optimization, and the formula of the Kalman filter gain K is:

[0155]

[0156] where the observation input H is the derivative of the ICP observation residual with respect to the state, R1 is the observation noise covariance, P - is the state prediction value; the obtained Kalman filter gain K is substituted into the state increment calculation:

[0157]

[0158] where z is the actual observation value of the point-to-plane residual from the ICP calculation, ICP observation residual derived from the predicted state; x is the state increment; according to the formula State update on the current pose corresponding to the i-th frame:

[0159]

[0160] wherein, is the pose; is the velocity, and δv is the velocity increment; is the position, and δp is the position increment; is the gyroscope bias, b g is the gyroscope bias increment; is the accelerometer bias, b a is the accelerometer bias increment; the left side of the arrow is the updated state, which is the optimal pose estimation result.

[0161] Step A5, correct the current frame original point cloud according to the optimal pose estimation result, fuse the corrected current frame original point cloud with the constructed local map, generate a local map of the current frame, and take it as a new constructed local map; at the same time, the industrial camera combines the IMU to obtain the RGB value corresponding to each point in the current frame original point cloud;

[0162] The industrial camera combines the IMU to obtain the RGB value corresponding to each point in the current frame original point cloud specifically as follows:

[0163] Step B1, the industrial camera combines the IMU to construct a visual inertial odometer, and the visual inertial odometer includes a frame-to-frame visual inertial odometer (Frame-to-Frame) and a frame-to-map visual inertial odometer (Frame-to-Map); the industrial camera collects a current frame image I k ; the map in the frame-to-map visual inertial odometer is a visual sparse map;

[0164] Step B2, in the frame-to-frame visual inertial odometer, first, a group of corner points are extracted from a previous frame image I k-1 using the FAST algorithm Then, the corner points are tracked in a current frame image I k using the Pyramidal Lucas-Kanade Optical Flow method (Lucas-Kanade optical flow method), to obtain corresponding pixel positions

[0165] Step B3, the three-dimensional coordinates of each pixel in the previous frame image I k-1 are P i , and the current frame image I kThe 3D–2D reprojection error was calculated, and the pose parameter R was obtained when the 3D–2D reprojection error was minimized. k t k Then, based on the pose parameters R at this time... k t k Calculate the current frame pose T of the industrial camera k =[R k |t k ]; where, minimize the current frame image I k The 3D–2D reprojection error is expressed by the following formula:

[0166]

[0167] Among them, R k Let t be the rotation matrix of the current frame. k Let be the translation vector of the current frame; π is the projection function of the industrial camera, and the formula for calculating π is:

[0168]

[0169] Where (X,Y,Z) are the coordinates of a point in space, f x f y c is the lens focal length. x c y Principal point coordinates;

[0170] Step B4: Based on the current frame pose T of the industrial camera k Correct pixel position F k Obtain the registered current frame image I' k ;

[0171] Step B5: In frame-map visual inertial odometry, the visual inertial odometry uses a visual sparse map as a reference and selects a set of sparsely distributed points from it. Figure Three Dimension And projected onto the registered current frame image I' k Establish a photometric error model:

[0172]

[0173] Among them, C j For the land Figure Three Dimension P j The corresponding RGB information, I1 k For the current frame image I' k The pixel grayscale value, r j This is photometric error;

[0174] Step B6: Minimize photometric error r j Obtain the pose parameters R at this time. k tk ; according to the photometric error r j the minimum pose parameter R k , t k , the new current pose of the industrial camera is calculated as the best pose of the current frame of the industrial camera; wherein the photometric error r j is minimized

[0175]

[0176] Step B7, according to the best pose of the current frame of the industrial camera, project the current frame of the original point cloud P = {p i} collected by the laser radar to the current frame image I' k , each point p i in the original point cloud corresponds to a pixel position k on the current frame image I'

[0177]

[0178] wherein T CL is the external parameter matrix of the laser radar to the industrial camera;

[0179] Step B8, use bilinear interpolation to obtain the RGB value RGB of each pixel position i :

[0180]

[0181] wherein Interp bilinear is the bilinear interpolation function;

[0182] Step B9, assign the RGB value RGB of each pixel position i to the corresponding point p i in the original point cloud.

[0183] Step A6, return to step A3 until all frames are traversed, complete the three-dimensional RGB point cloud model reconstruction of the target scene, and obtain the global map P RGB containing RGB information of the target scene.

[0184] Three different types of green trees were selected as the research objects, and a handheld laser radar and an industrial camera were bound together to collect data around each green tree. The high-resolution industrial camera obtained RGB images for color information extraction, while the laser radar collected three-dimensional point cloud data of the target to obtain its depth information. The inertial measurement unit (IMU) built into the laser radar can effectively compensate for errors caused by device motion during the collection process. Through the internal and external parameter matrices obtained from the previous calibration at a distance of 2 meters, the spatial synchronization between the camera and the laser radar was realized, and the color information of the image was accurately mapped to the corresponding three-dimensional point cloud, thereby generating an RGB point cloud. As shown in FIG. 8, the leftmost side is a frame collected by a visual inertial odometer (VIO), the middle is a frame collected by a laser radar combined with an IMU, and the rightmost side is a three-dimensional RGB point cloud model after fusion. The target scenes of Plant1-3 from top to bottom are respectively cypress, pine forest, and mountain jinzis. Figure 4

[0185] Step 2, reconstruction of hyperspectral three-dimensional point cloud model

[0186] Step 21, extraction of global map P RGB RGB information C N×3 and geometric structure information S N×3 ,

[0187] RGB information C N×3 and geometric structure information S N×3 are mapped into a two-dimensional image, and the RGB information of each point is converted into a pixel to form a picture I H×W×3 ; wherein N is the total number of point clouds;

[0188] Geometric structure information S N×3 mapping process is:

[0189]

[0190] RGB information C N×3 mapping process is:

[0191]

[0192] wherein 255 is the maximum brightness value of the pixel; H is the image height, W is the image width, and 3 in N x 3 represents 3 RGB channels;

[0193] Step 22, spectral reconstruction of picture I H×W×3 by a deep learning method, which expands the 3 channels of RGB corresponding to each pixel to U channels to obtain a hyperspectral image Y ∈ R H*W*U ;

[0194] ​Step 221, the deep learning method adopts the MST++ (Multi-Stage Spectral Perceptive Transformer) method, which includes a plurality of single-stage SSTs (Spectral-wise Transformer) in cascade; each SST adopts a U-shaped structure, including an encoder, a bottleneck layer, and a decoder;

[0195] Picture I H×W×3 The corresponding RGB image is I ∈ R H*W*3 , the RGB image I ∈ R H*W*3 is input into a 3x3 convolutional layer to realize feature embedding (Embedding) and generate an initial feature map X0;

[0196] Step 222, the encoder includes a 4x4 first convolutional layer with a step size of 4, N1 spectral attention blocks SABs, a 4x4 second convolutional layer with a step size of 4, a 4x4 third convolutional layer with a step size of 4, and N2 spectral attention blocks SABs, which sequentially perform first downsampling, first spectral feature extraction, second downsampling, third downsampling, and second spectral feature extraction on the initial feature map X0, for deepening spectral information modeling, obtaining a first feature map, and outputting to the bottleneck layer;

[0197] The calculation process of the spectral attention block SABs in step 222 is the same, and the spectral attention block SABs as the core calculation unit can effectively capture the spectral self-similarity in the feature map, thereby improving the accuracy and stability of spectral reconstruction. The spectral attention block SABs includes a multi-head self-attention mechanism S-MSA (Spectral Multi-Head Self-Attention), an FFN (Feed Forward Network, feed-forward neural network), and a normalization layer (Normalization Layer);

[0198] S-MSA is the most core block of SAB, which models the dependency relationship between spectral channels and adaptively adjusts the attention strength.

[0199] Taking N1 spectral attention blocks SABs as an example, the feature map after the first downsampling is X ∈ R H*W*C , which is input into N1 spectral attention blocks SABs, flattened into a matrix X ∈ R HW*C in the spatial dimension, and the query Q matrix, the key Key matrix, and the value V matrix are obtained through linear mapping:

[0200]

[0201] where W Q ,W Key ,WV are learnable parameters;

[0202] Then, the features are divided into multiple subspaces in the channel dimension, and each head of the multi-head self-attention mechanism S-MSA independently performs attention calculation on one subspace:

[0203]

[0204] wherein A j is the attention weight matrix of the jth head head j , σ j is a learnable scaling factor for adapting the spectral density distribution of different wavebands; is the Key matrix of the jth head head j , Q j is the Query Q matrix of the jth head head, V j is the Value V matrix of the jth head head j ;

[0205] Finally, each head head is spliced to form an output through linear transformation and position encoding:

[0206]

[0207] wherein f p represents the spectral position encoding generated by using two layers of 3x3 deep separable convolution.

[0208] In other embodiments, an infrared, ultraviolet deep learning network is used to expand the waveband range, and a hyperspectral point cloud with higher spectral resolution and wider waveband coverage can be reconstructed.

[0209] Step 223, the bottleneck layer includes N3 spectral attention blocks SABs to enhance the global spectral modeling capability; the first feature map is sequentially subjected to spectral feature extraction to obtain a second feature map, and the second feature map is output to the decoder; the calculation process of the N3 spectral attention blocks SABs is the same as that of the spectral attention block SAB in step 222;

[0210] Step 224, the decoder and the encoder are symmetric structures, including a first deconvolution layer with a step of 4, a second deconvolution layer, and a third deconvolution layer, which sequentially perform upsampling on the second feature map; the first deconvolution layer, the second deconvolution layer, and the third deconvolution layer are also connected with the first convolution layer, the second convolution layer, and the third convolution layer through skip connections, to fuse the features of the encoder; the output channel number U of the third deconvolution layer is 31, and a reconstructed hyperspectral image Y∈R H*W*31 is output; the hyperspectral image Y∈R H*W*31The wavelength range is 400-700 nm, and the spectral resolution is 10 nm.

[0211] Step 23, using the ARAD-HS dataset to process the hyperspectral image Y ∈ R H*W*U for training; the ARAD-HS dataset contains 510 groups of data, each group of data contains 3 channels of RGB information, and 31 channels of hyperspectral data corresponding thereto;

[0212] Step 24, mapping the spectral information in the trained hyperspectral image Y ∈ R H*W*31 to the geometric structure information S N×3 , to generate a hyperspectral three-dimensional point cloud model and complete the hyperspectral three-dimensional reconstruction. Through this process, not only the geometric structure characteristics of the point cloud can be preserved, but also rich spectral information can be given, so that the fusion of hyperspectral data and three-dimensional reconstruction technology is realized.

[0213] As shown in Figure 5 , the images in the same row respectively show the hyperspectral three-dimensional point cloud model of the same green vegetation reconstructed at 450 nm, 550 nm and 650 nm bands, which intuitively shows the reconstruction ability of the method at different bands.

[0214] Step 3 performance evaluation, specifically:

[0215] Step 31, using the hyperspectral image Y ∈ R H*W*31 obtained in step 22 as the predicted value ; the data of the target scene collected by the push-broom spectrometer is used as the true value Y i ; in this embodiment, the SIMSPEC HL10 push-broom hyperspectral imager is selected to collect three kinds of green leaf samples in the field, and the hyperspectral reflectance curve is obtained for verifying the authenticity of the reconstruction result.

[0216] To quantify the reconstruction effect, three evaluation indexes are used to evaluate the performance of the method. The first one is the root mean square error RMSE, which is used to measure the overall size of the error between the predicted value and the true value, the smaller the better, 0 represents complete consistency; the root mean square error RMSE between the predicted value and the true value Y i is calculated as:

[0217]

[0218] where Y i is the true value, is the predicted value, and N is the number of all values;

[0219] The second one is the mean absolute error MAE, which represents the degree of deviation of the average spectral reconstruction result from the true value, the closer to 0, the smaller the error; the mean absolute error MAE between the predicted value Compared with the true value Y i Mean Absolute Error (MAE) between:

[0220]

[0221] The third is the goodness-of-fit coefficient (GFC), which measures the structural similarity between spectral curves; a value approaching 1 indicates perfect similarity. Predicted values ​​are then calculated. Compared with the true value Y i Goodness-of-fit coefficients (GFCs) between them:

[0222]

[0223] Step 32: If the root mean square error (RMSE), mean absolute error (MAE), and goodness-of-fit coefficient (GFC) meet the requirements, the hyperspectral three-dimensional reconstruction is completed; if they do not meet the requirements, return to step 22, change the number of extended channels U, and reconstruct the spectrum.

[0224] like Figure 6 As shown, the leftmost curve is the comparison between the reconstructed spectrum of Plant 1 (Chinese juniper) and the reference spectrum measured by the spectrometer; the middle curve is the comparison between the reconstructed spectrum of Plant 2 (pine forest) and the reference spectrum measured by the spectrometer; and the rightmost curve is the comparison between the reconstructed spectrum of Plant 3 (wild privet) and the reference spectrum measured by the spectrometer.

[0225]

[0226] Table 1 lists the quantitative evaluation indicators of the reconstruction results relative to the true value, including mean squared error (RMSE), mean absolute error (MAE) ratio, and goodness of fit coefficient (GFC), to comprehensively measure the reconstruction performance of this method.

Claims

1. A hyperspectral three-dimensional reconstruction method fusing lidar, vision and inertial information, characterized in that, The method comprises the following steps: Step 1, reconstructing a three-dimensional RGB point cloud model Step 11, collecting images of a target scene by using an industrial camera, collecting original point clouds of the target scene by using a laser radar, and collecting pose data of the laser radar and the industrial camera by using an IMU; Step 12, performing spatial alignment on the industrial camera and the laser radar; Step 13, realizing time synchronization of the laser radar and the industrial camera based on PPS pulse signals and GPRMC data; Step 14, the laser radar combines the IMU, adopts the close-coupling mode to build a map frame by frame; meanwhile, the industrial camera combines the IMU, obtains the RGB information of the current frame, projects the RGB information to the original point cloud of the current frame; until all frames are traversed, the global map of the target scene containing the RGB information is obtained , that is, the RGB three-dimensional point cloud model; Step 2, reconstructing a hyperspectral three-dimensional point cloud model Step 21, extract global map RGB information of each point and geometry information , RGB information and geometry information is mapped into a two-dimensional image, constituting a picture ; wherein N is the total number of point clouds; Step 22, performing spectral reconstruction on the picture by a deep learning method to expand the 3 channels of RGB corresponding to each pixel to U channels, to obtain a hyperspectral image ; Step 23, training using ARAD-HS dataset on hyperspectral images ; Step 24, mapping the spectral information in the trained hyperspectral image to the geometric structure information Above, generate a hyperspectral three-dimensional point cloud model, complete hyperspectral three-dimensional reconstruction; The step 22 is specifically: Step 221, the deep learning method adopts an MST++ method, comprising a plurality of single-stage SSTs in cascade; each SST adopts a U-shaped structure, comprising an encoder, a bottleneck layer and a decoder; Picture The corresponding RGB image is The RGB image is input into a 3x3 convolutional layer to realize feature embedding and generate an initial feature map X0. Step 222, the encoder includes a first convolutional layer of 4x4 with a step of 4, a second convolutional layer of 4x4 with a step of 4, a third convolutional layer of 4x4 with a step of 4, a first spectral attention block SAB, sequentially performing first downsampling, first spectral feature extraction, second downsampling, third downsampling and second spectral feature extraction on the initial feature map X0 to obtain a first feature map, and outputting to a bottleneck layer; Step 223, the bottleneck layer includes a plurality of spectral attention blocks SABs, sequentially performing spectral feature extraction on the first feature map to obtain a second feature map, and outputting to the decoder; The decoder and the encoder are in a symmetrical structure, comprising a first deconvolution layer, a second deconvolution layer and a third deconvolution layer, which sequentially perform up-sampling on the second feature map; The first deconvolution layer, the second deconvolution layer and the third deconvolution layer are also in a skip connection with the first convolution layer, the second convolution layer and the third convolution layer, and fuse the features of the encoder; The output channel number U of the third deconvolution layer is 31, and a reconstructed hyperspectral image is output ; the hyperspectral image covers a wavelength range of 400-700 nm with a spectral resolution of 10 nm; H is the image height, and W is the image width.

2. The fusion of lidar, vision and inertial information for hyperspectral 3D reconstruction method according to claim 1, characterized in that, The step 12 is specifically: Step 12, jointly calibrating the industrial camera and the laser radar, so that the original point clouds and the corner points in the images have a one-to-one correspondence; Given four sets of laser-radar-industrial camera corresponding corner points, adjust the transformation rotation matrix through the nonlinear optimization library and translation vector , calculate the re-projection error of all points in the original point cloud in the pixel coordinate system of the image , re-projection error The optimization objective function of is: ； wherein, is a three-dimensional position of the i-th point in the laser radar coordinate system, is a two-dimensional coordinate of the i-th point in the pixel coordinate system of the image, is an intrinsic parameter of the industrial camera; Obtaining reprojection error Rotation matrix corresponding to the minimum And translation vector To enable spatial alignment of the lidar and industrial camera.

3. The fusion of lidar, vision and inertial information for hyperspectral 3D reconstruction method according to claim 1, characterized in that, The step 13 is specifically: Step 131, using a single-chip microcomputer to convert the PPS pulse signals output by a GPS module into 1Hz square wave signals and 10Hz square wave signals, and transmitting the signals to the laser radar and the industrial camera respectively, so as to realize time triggering; Step 132, after receiving the rising edge of the kth 1Hz square wave signal, the laser radar combines the UTC time information in the GPRMC data it receives , the internal timestamp is aligned with the UTC time information , and serves as the reference time , the laser radar writes the reference time into the shared memory shm; wherein: ； Step 133, the shared memory shm transmits the reference time to the industrial camera; the industrial camera finds the distance reference time of the last image, denoted as the frame, and resets the time of this frame to the reference time , i.e. ； The time synchronization of the two sensors is realized; wherein: ； is the timestamp of the jth frame of the industrial camera.

4. The fusion of lidar, vision and inertial information for hyperspectral 3D reconstruction method according to claim 1, characterized in that, The step 14 is specifically: Step A1, the laser radar collects a first frame of original point clouds, and performs time compensation on the original point clouds to eliminate motion distortion; Step A2, constructing a first frame of local map according to the first frame of original point clouds, as a constructed local map; Step A3, the laser radar collects a next frame of original point clouds as a current frame of original point clouds; time compensation is performed on the current frame of original point clouds to eliminate motion distortion; the IMU collects the pose data of the laser radar in the current frame and performs state pre-integration to obtain the current pose of the laser radar; Step A4, using a point-to-plane ICP algorithm to register the current frame of original point clouds and the constructed local map, performing state updating on the current pose to obtain an optimal pose estimation result of the laser radar in the current frame; Step A5, correcting the current frame of original point clouds according to the optimal pose estimation result, fusing the corrected current frame of original point clouds and the constructed local map to generate a local map of the current frame, and taking the local map as a new constructed local map; at the same time, the industrial camera combines the IMU to obtain the corresponding RGB value of each point in the current frame of original point clouds; Step A6, return to step A3 until all frames are traversed, complete the three-dimensional RGB point cloud model reconstruction of the target scene, and obtain a global map of the target scene containing RGB information .

5. The fusion of lidar, vision and inertial information for hyperspectral 3D reconstruction method according to claim 4, characterized in that, In step A5, the industrial camera combines the IMU to obtain the corresponding RGB value of each point in the current frame of original point clouds, which is specifically: Step B1, an industrial camera combined with an IMU constructs a visual inertial odometer, the visual inertial odometer including a frame-frame visual inertial odometer and a frame-map visual inertial odometer; the industrial camera collects a current frame image ; a map in the frame-map visual inertial odometer is a visual sparse map; Step B2, in frame-to-frame visual-inertial odometry, first a set of corner points are extracted in the previous frame image using the FAST algorithm ={ }, then the corner points ={ } are tracked in the current frame image using the Lucas-Kanade optical flow method, obtaining the corresponding pixel positions ={ }. Step B3, previous frame image The three-dimensional coordinates of each pixel are Minimizing the 3D-2D re-projection error of the current frame image to obtain the pose parameters when the 3D-2D re-projection error is minimized , ; According to the pose parameters at this time , Calculate the current frame pose of the industrial camera ; wherein the 3D-2D re-projection error of the current frame image is minimized using the following equation: ； wherein, is a rotation matrix of the current frame, is a translation vector of the current frame; is a projection function of the industrial camera, The calculation formula of is: ； wherein, is a spatial point coordinate, , is a lens focal length, is a principal point coordinate; Step B4, obtaining a current frame image registered correcting pixel positions , obtaining a registered current frame image ; Step B5, in frame-map visual-inertial odometry, the visual-inertial odometry takes a set of sparse distributed map three-dimensional points selected from the visual sparse map as a reference , and projects them to the registered current frame image , and establishes a photometric error model: ； wherein, a map three-dimensional point corresponding RGB information, a current frame image a pixel gray value, photometric error; Step B6, minimizing photometric error , obtaining the pose parameter at this time , ; and then according to the photometric error , the pose parameter at the time of minimum photometric error , , calculating the new current pose of the industrial camera as the optimal pose of the current frame of the industrial camera; wherein the photometric error is minimized using the following formula: ； Step B7. According to the best pose of the current frame of the industrial camera, the current frame raw point cloud collected by the LiDAR projected to the current frame image , each point in the raw point cloud corresponds to a pixel position on the current frame image ; ； wherein, is the extrinsic matrix of the laser radar to the industrial camera; Step B8. Obtain the RGB value for each pixel location using bilinear interpolation :​ ； wherein is a bilinear interpolation function; Step B9: Position each pixel RGB values Assign points to the corresponding points in the original point cloud .

6. The fusion of lidar, vision and inertial information for hyperspectral 3D reconstruction method according to claim 5, characterized in that, Step A4 is specifically: Constructing point-to-plane residual terms : ； wherein, is the original point cloud of the i-th frame, is the original point cloud in the constructed local map of the i-1-th frame, is the local map of the i-1-th frame, is the plane normal of the face where the point p;is located; R is a rotation matrix, and t is a translation vector. Using the Lie group, Lie algebra method, the residual term of the ith frame is converted into a linear error model of point-to-plane ICP :​ ； wherein is the Jacobian matrix of the ith frame, is a Lie algebra, is the increment of the Lie algebra ξ and the translation vector t; Minimizing linear error model and linear error model Minimizing linear error model As the observation input H; combined with the state pre-integration in step A3, input to the error state iteration Kalman filter for state update and optimization, the formula of Kalman filter gain K is: ； where the observation input H is the derivative of the ICP observation residual with respect to the state, 1 is the observation noise covariance, is the state prediction; and the Kalman filter gain K is calculated as follows: ； where, is the actual observation of the point-to-plane residual from ICP computation, is the ICP observation residual derived from the predicted state; is the state increment; according to the formula state update for the current pose corresponding to the i-th frame: ； wherein, is the pose; is the velocity, is the velocity increment; is the position, is the position increment; is the gyroscope bias, is the gyroscope bias increment; is the accelerometer bias, is the accelerometer bias increment; the left side of the arrow is the updated state as the optimal pose estimate.

7. The hyperspectral three-dimensional reconstruction method fusing laser radar, vision and inertial information according to claim 1, characterized in that: The geometry information in the step 21 The mapping process is: ； RGB information The mapping process is: ； wherein 255 is the maximum brightness value of a pixel; and 3 in N*3 represents 3 RGB channels. In the step 23, the ARAD-HS dataset contains 510 groups of data, each group of data containing 3 channels of RGB information and 31 channels of hyperspectral data corresponding thereto.

8. The hyperspectral three-dimensional reconstruction method of fusing lidar, vision and inertial information according to claim 7, characterized in that: In step 222, the spectral attention block SABs includes a multi-head self-attention mechanism S-MSA, an FFN and a normalization layer; The feature map after the first downsampling is , input to spectral attention blocks SABs, which are flattened into a matrix in the spatial dimension, and the query Q matrix, the key Key matrix, and the value V matrix are obtained through linear mapping: ； wherein , , are learnable parameters; Then, the features are divided into multiple subspaces in the channel dimension, and each head of the multi-head self-attention mechanism S-MSA independently performs attention calculation on a subspace: ； ； in, Is it the j-th head? Attention weight matrix, It is a learnable scaling factor used to adapt to the spectral density distribution of different bands; Is it the j-th head? The key matrix, It is the query Q matrix of the j-th head. Is it the j-th head? The value of matrix V; Finally, each head is spliced to form an output through linear transformation and position encoding: ； wherein represents a spectral position encoding generated using a two-layer 3x3 depthwise separable convolution.

9. The fusion of lidar, vision and inertial information for hyperspectral 3D reconstruction method according to claim 8, characterized in that, It also includes step 3 performance evaluation, specifically: Step 31, obtaining hyperspectral images from step 22 as a predicted value , using a pushbroom spectrometer to collect data of the target scene as a true value ; The root mean square error RMSE between the calculated prediction values and the true values ​ ； wherein is the true value, is the predicted value, N is the number of all values; The mean absolute error MAE between the calculated prediction values and the true values is: ； The coefficient of goodness of fit GFC between the calculated prediction values and the true values is calculated. ； In step 32, if the root mean square error RMSE, the mean absolute error MAE and the goodness of fit coefficient GFC meet the requirements, the hyperspectral three-dimensional reconstruction is completed; if they do not meet the requirements, step 22 is returned, the number of extended channels U is changed, and the spectral reconstruction is re-performed.

Images