A method and system for generating hyperspectral point cloud data with three-dimensional atlas integration

Abstract

The present invention provides a method and system for generating hyperspectral point cloud data with integrated three-dimensional maps, which relates to the technical fields of RGB image spectral super-resolution reconstruction and three-dimensional point cloud reconstruction. After obtaining RGB image data using a mobile sensor device, a multi-level spectral transformation network and a depth prediction network are constructed and trained. After inputting a single-frame RGB image, a hyperspectral image and the initial depth information of the RGB image are respectively reconstructed and generated. The initial focal length of the camera of the mobile sensor device is determined and a distorted 3D point cloud is generated. A 3D point cloud generation network is constructed and the distorted 3D point cloud data is used as input to reconstruct and generate 3D point cloud data. The spectral information and the point cloud band information are obtained using the generated hyperspectral image and 3D point cloud data. The spectral information of multiple bands of the corresponding pixel points of the hyperspectral image is mapped onto the point cloud band information to generate hyperspectral point cloud data with integrated three-dimensional maps, avoiding the problems of independent acquisition and fusion of two heterogeneous data, namely spectra and point clouds.

Description

Technical Field

[0001] The present invention relates to the technical field of RGB image spectral super-resolution reconstruction and 3D point cloud reconstruction, and more specifically, to a method and system for generating hyperspectral point cloud data with three-dimensional atlas integration. Background Art

[0002] In recent years, with the continuous development of high spectral resolution and high spatial resolution technologies, hyperspectral images have become an important information source for China's economy and agricultural life. Hyperspectral remote sensing images have the advantages of rich information content, high spectral resolution, and wide band range, and contain more accurate identifiable ground object feature information. However, although the hyperspectral images collected by current imaging spectrometers for hyperspectral image acquisition have rich spectral information, they do not have high-resolution spatial three-dimensional characteristics.

[0003] To solve the problem that the hyperspectral images collected by current imaging spectrometers for hyperspectral image acquisition do not have high-resolution spatial three-dimensional characteristics, an existing technology discloses an intrinsic hyperspectral point cloud generation method, system, and device. The invention first obtains an intrinsic mapping matrix through a hyperspectral image and a lidar point cloud, then segments and extracts the pixel set belonging to the building surface in the hyperspectral image H, and determines the vector L of the incident light direction using the pixels in the set Q; finally, performs hyperspectral image-lidar point cloud joint intrinsic decomposition to generate an intrinsic hyperspectral point cloud, and the generated spectral information-spatial three-dimensional information integrated data has high accuracy and is not affected by light. However, in this solution, spectral and point cloud two heterogeneous data need to be independently collected and fused, so the acquisition process is complex, which means that the generation of hyperspectral point clouds still requires a relatively high cost. Summary of the Invention

[0004] To solve the problem that the generation of current hyperspectral point clouds requires independent collection and fusion of spectral and point cloud two heterogeneous data, and the acquisition process is complex and difficult and the required cost is relatively high, the present invention proposes a method and system for generating hyperspectral point cloud data with three-dimensional atlas integration, which not only realizes the accurate acquisition of the high-resolution spatial three-dimensional characteristics of the target or scene, but also obtains high spectral resolution while obtaining high spatial resolution, so as to obtain the data characteristics of three-dimensional atlas integration to guide the production in fields such as agriculture.

[0005] To achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] S1. Use a mobile sensor device to obtain RGB image data;

[0007] S2. Construct a multi-level spectral transformation network and train it, and input a single-frame RGB image into the trained multi-level spectral transformation network to reconstruct and generate a hyperspectral image;

[0008] S3. Construct a depth prediction network and train it. Input a single-frame RGB image into the trained depth prediction network to obtain the initial depth information of the single-frame RGB image;

[0009] S4. Determine the initial focal length of the camera of the mobile sensor device, and calculate and generate a distorted 3D point cloud based on the initial depth information and the initial camera focal length;

[0010] S5. Construct a 3D point cloud generation network. Input the distorted 3D point cloud data into the 3D point cloud generation network to obtain a depth offset adjustment factor and a focal length adjustment factor to adjust and predict the final depth information and the camera focal length, and reconstruct and generate 3D point cloud data;

[0011] S6. Perform data processing on the hyperspectral image generated in S2 to obtain the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image. Perform point cloud processing on the 3D point cloud data generated in S5 to obtain the band information corresponding to each pixel point on the 3D point cloud;

[0012] S7. End-to-end map the spectral information of multiple bands of the corresponding pixel points of the hyperspectral image to the band information of the pixel points of the 3D point cloud to obtain a hyperspectral point cloud with multi-band spectral information.

[0013] This technical solution uses data-driven learning single-frame RGB image spectral super-resolution reconstruction and three-dimensional reconstruction technologies to generate a hyperspectral point cloud. First, a multi-level spectral transformation network is constructed to perform spectral super-resolution reconstruction using a single-frame RGB image. Then, a depth prediction network and a 3D point cloud generation network are constructed to perform 3D point cloud reconstruction using a single-frame RGB image. Finally, the spectral information of multiple bands of the hyperspectral image reconstructed by spectral super-resolution is end-to-end mapped into the 3D point cloud to generate hyperspectral point cloud data, thereby obtaining the data characteristics of the three-dimensional map integration. It not only realizes the accurate acquisition of the high-resolution spatial three-dimensional characteristics of the target or scene, but also obtains high spectral resolution while obtaining high spatial resolution to guide production in fields such as agriculture.

[0014] Preferably, in step S1, it further includes the process of using a hyperspectral imager to obtain an RGB-HSI image pair. The RGB image data in the RGB-HSI image pair includes GNSS coordinates and the names of each band, which are stored in the metadata of the image; perform preprocessing operations on the obtained RGB image data. The preprocessing operations include distortion correction and stereo correction, so as to subsequently use the RGB image to construct a depth prediction network and a 3D point cloud generation network, and use the RGB-HSI image pair to train the model.

[0015] Preferably, in step S2, the multi-level spectral transformation network is composed of a cascade of multiple single-level spectral transformation modules. Each single-level spectral transformation network is composed of a U-shaped structure module of Encoder-Decoder. The U-shaped structure module is composed of multiple spectral attention modules. Each spectral attention module includes a spectral multi-head self-attention module with the properties of HSI space sparsity and spectral self-similarity and a feed-forward network module.

[0016] Preferably, after the multi-level spectral transformation network is constructed, it is trained as follows:

[0017] Based on the optimizer Adam, the self-collected dataset of 128×128 RGB-HSI image pairs obtained in step S1 and the NTIRE 2022 public dataset are used to train the multi-level spectral transformation network. During the training process, the RGB image data is linearly rescaled to [0,1], and then 128×128 RGB-HSI sample pair datasets are cropped from the NTIRE 2022 dataset, and the self-collected data of 128×128 RGB-HSI image pairs are supplemented to the cropped dataset. The training data is enhanced by random rotation and flipping.

[0018] Preferably, the depth prediction network constructed in step S3 includes a standard backbone ResNet50 for feature extraction and a Decoder decoder for predicting unknown scale depth information. After the depth prediction network is constructed, the depth prediction network is trained;

[0019] Obtain RGB-D image data from the open-source dataset, and use the RGB-D image data to train the depth prediction network. During the training process, the depth range of the RGB-D image data is normalized to [0,1], and image normalization regression loss processing is performed. The image normalization regression loss ILNR function is:

[0020]

[0021] where, L ILNR represents the image normalization regression loss ILNR function, d i represents the predicted depth information, represents the ground truth depth information; μ trim and σ trim represent the mean and standard deviation of the depth map respectively;

[0022] The pairwise normal regression PWN loss is used to improve the performance of local geometric feature learning. The expression of the pairwise normal regression PWN loss function is:

[0023]

[0024] Among them, L PWN For the paired normal regression PWN loss function, let the sampling points be {(A i , B i ), i = 0, ..., N}, then the surface normals corresponding to the sampling points are {(n Ai , n Bi ), i = 0, ..., N}; n * represents the ground truth surface normal; is the ground truth surface normal of (A i , B i ); The image normalization regression loss ILNR function and the normal regression PWN loss function together constitute the total loss function of the depth prediction network. Then, the depth prediction network is trained using the Adam optimizer to optimize its network parameters until the total loss function of the depth prediction network converges, and a trained depth prediction network is obtained.

[0025] Here, fully considering the factor that the depth ranges of the image data provided by the multi-source dataset used in the training of the depth prediction network are different, image normalization regression loss processing is performed, the performance of local geometric feature learning is improved, and the performance of the depth prediction network is optimized.

[0026] Preferably, the calculation process of the distorted 3D point cloud in step S4 is as follows:

[0027] Let the initial depth information of the single-frame RGB image predicted by the depth prediction network be d init and the initial camera focal length be f init . The predicted initial depth information and initial camera focal length are transformed through the coordinate transformation formula to obtain the distorted 3D point cloud data:

[0028] Among them,

[0029] Among them, (u, v) is an arbitrary coordinate point in the image coordinate system, u 0 , v 0 are respectively the central coordinates of the single-frame RGB image, (x w , y w , z w ) represents the three-dimensional coordinate point in the world coordinate system, z c represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera, R and T are both external parameter matrices, d init is the initial depth information, and f init is the initial focal length.

[0030] Preferably, the 3D point cloud generation network constructed in step S5 consists of an offset point cloud network N d () and a focal length point cloud network Nf Composition;

[0031] After constructing the 3D point cloud generation network, use the Adam optimizer to train the 3D point cloud generation network and optimize its network parameters;

[0032] Among them, the input network architecture of the 3D point cloud generation network is the PVCNN network. Input the distorted 3D point cloud data into the 3D point cloud generation network to obtain the depth offset adjustment factor and the focal length adjustment factor, and adjust and predict the final depth information d pred And reconstruct the 3D point cloud data;

[0033] When adjusting and predicting the depth information, assume the distorted 3D point cloud is Use it as the input data of the offset point cloud network N d () to construct the training objective function as:

[0034]

[0035] Among them, θ is the network weight, f * is the ground truth focal length, d * is the ground truth depth, is the depth offset adjustment factor;

[0036] When adjusting and predicting the camera focal length, use the distorted 3D point cloud as the input data of the focal length point cloud network N f () to construct the training objective function:

[0037]

[0038] Among them, θ is the network weight, f * is the ground truth focal length, d * is the ground truth depth, is the focal length adjustment factor;

[0039] Repeat adjusting the depth information and the focal length several times. Then, use the adjusted and predicted depth information d pred and the camera focal length f pred to reconstruct and generate the 3D point cloud data through the coordinate transformation formula. The coordinate transformation formula is as follows:

[0040] Among them,

[0041] Among them, (u, v) is an arbitrary coordinate point in the image coordinate system, u 0 , v 0 are the central coordinates of the image respectively, (x w , y w , z w) represents a three-dimensional coordinate point in the world coordinate system, z c represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera. R and T are the external parameter matrices respectively, d pred is the finally predicted depth information, f pred is the finally predicted camera focal length.

[0042] Preferably, in step S6, load and read the hyperspectral image data generated in step S2, and convert the hyperspectral image data into a multi-dimensional matrix of the hyperspectral image. Then loop through each element in the multi-dimensional matrix of the hyperspectral image. Each such element is a pixel point of the hyperspectral image. During the traversal, obtain the matrix corresponding to each pixel point. This matrix represents the spectral information of multiple bands corresponding to each pixel point. Finally, obtain the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image;

[0043] The steps of performing point cloud processing operations on the 3D point cloud generated by three-dimensional reconstruction in step S5 are as follows:

[0044] Load and read the point cloud data, convert the point cloud data into a multi-dimensional matrix of the point cloud, perform point cloud outlier filtering operations, and then loop through each element in the multi-dimensional matrix of the point cloud. Each such element is a pixel point of the point cloud. Remove the NAN points in the point cloud. During the traversal, obtain the matrix corresponding to each pixel point in the point cloud. This matrix represents the information of multiple bands corresponding to each point. Finally, obtain the band information corresponding to each pixel point on the 3D point cloud.

[0045] Preferably, in step S7, loop through each pixel point on the 3D point cloud and the hyperspectral image, use the SIFT feature matching algorithm to extract and match the pixel point features of the 3D point cloud and the hyperspectral image, find the matching pixel point pairs. After the matching is completed, map the spectral information of multiple bands of the corresponding pixel point of the hyperspectral image end-to-end to the band information of the matching 3D point cloud pixel point, and store all the band information in the form of a point cloud for data storage, so as to obtain the hyperspectral point cloud data with three-dimensional map integration and multi-band spectral information.

[0046] This application also proposes a system for generating hyperspectral point cloud data with three-dimensional map integration. The system includes:

[0047] A mobile sensor device for acquiring RGB image data;

[0048] A hyperspectral image reconstruction and generation unit for constructing and training a multi-level spectral transformation network, inputting a single-frame RGB image into the trained multi-level spectral transformation network, and reconstructing and generating a hyperspectral image;

[0049] An image depth information acquisition unit, which is used to construct and train a depth prediction network, input a single-frame RGB image into the trained depth prediction network, and obtain the initial depth information of the single-frame RGB image;

[0050] A distorted 3D point cloud generation unit, which is used to determine the initial focal length of the camera of the mobile sensor device, and calculate and generate a distorted 3D point cloud according to the initial depth information and the initial camera focal length;

[0051] A 3D point cloud generation unit, which is used to construct a 3D point cloud generation network, take the distorted 3D point cloud data as input, obtain a depth offset adjustment factor and a focal length adjustment factor to adjust and predict the final depth information and the camera focal length, and reconstruct and generate 3D point cloud data;

[0052] A band information acquisition unit, which is used to perform data processing on the hyperspectral image obtained by the hyperspectral image reconstruction generation unit, obtain the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image, perform point cloud processing on the 3D point cloud data generated by the 3D point cloud generation unit, and obtain the band information corresponding to each pixel point on the 3D point cloud;

[0053] A three-dimensional map integration unit, which maps the spectral information of multiple bands corresponding to the pixel points of the hyperspectral image end-to-end to the band information of the pixel points of the 3D point cloud to obtain a hyperspectral point cloud with multi-band spectral information.

[0054] In this technical solution, RGB image data is collected by a sensor device, a hyperspectral image is reconstructed and generated by a hyperspectral image reconstruction generation unit, a single-frame RGB image is input into the constructed and trained depth prediction network by an image depth information acquisition unit to obtain the initial depth information of the RGB image, then a distorted 3D point cloud is generated by a distorted 3D point cloud generation unit according to the initial depth information and the initial camera focal length, and then a 3D point cloud generation network is constructed by a 3D point cloud generation unit to reconstruct and generate 3D point cloud data, omitting the process of collecting hyperspectral image data and 3D point cloud data by external devices, simplifying the collection process and reducing the cost.

[0055] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0056] The present invention provides a method and system for generating hyperspectral point cloud data with three-dimensional map integration. First, RGB image data is acquired using a mobile sensor device. A multi-level spectral transformation network and a depth prediction network are respectively constructed and trained. A single-frame RGB image is input, a hyperspectral image is reconstructed and generated, and the initial depth information of the single-frame RGB image is obtained. Then, the initial focal length of the camera of the mobile sensor device is determined, a distorted 3D point cloud is calculated and generated. A 3D point cloud generation network is constructed and the distorted 3D point cloud data is used as input to reconstruct and generate 3D point cloud data. The generated hyperspectral image and 3D point cloud data are processed to obtain spectral information and band information. Finally, the multi-band spectral information of the corresponding pixels of the hyperspectral image is mapped end-to-end to the band information of the 3D point cloud pixels to generate a hyperspectral point cloud with multi-band spectral information. It can not only accurately obtain the high-resolution spatial three-dimensional characteristics of the target or scene, but also obtain high spectral resolution while retaining the high spatial resolution of the RGB image, thereby obtaining the data characteristics of three-dimensional map integration, providing rich spectral information and physical geometry information. In addition, the generation process of the hyperspectral point cloud does not require independent acquisition and fusion of two heterogeneous data of spectrum and point cloud, simplifies the acquisition process and reduces the acquisition cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 FIG. is a schematic flow chart showing the method for generating hyperspectral point cloud data with three-dimensional map integration proposed in Embodiment 1 of the present invention;

[0058] Figure 2 FIG. is a schematic flow framework diagram showing the method implementation of the method for generating hyperspectral point cloud data with three-dimensional map integration proposed in Embodiment 1 of the present invention;

[0059] Figure 3 FIG. shows the RGB image acquired by the mobile sensor device in Embodiment 1 of the present invention;

[0060] Figure 4 FIG. shows that in Embodiment 1 of the present invention, according to Figure 3 the RGB image acquired, the visualized image of the hyperspectral image cube reconstructed;

[0061] Figure 5 FIG. shows that in Embodiment 1 of the present invention, according to Figure 3 the RGB image acquired, the depth map predicted;

[0062] Figure 6 FIG. shows that in Embodiment 1 of the present invention, according to Figure 3 the RGB image acquired, the visualized image of the 3D point cloud data reconstructed;

[0063] Figure 7 FIG. shows that in Embodiment 1 of the present invention, according to Figure 4 the hyperspectral image and Figure 6Pseudo-color visualization images of hyperspectral point cloud data of different bands generated from 3D point cloud data;

[0064] Figure 8 Schematic diagram of the overall structure of the system representing the method for generating hyperspectral point cloud data with three-dimensional spectrum integration proposed in Embodiment 2 of the present invention. Detailed implementation manners

[0065] The accompanying drawings are only for illustrative purposes and should not be construed as limitations on this patent;

[0066] To better illustrate this embodiment, some parts of the accompanying drawings will be omitted, enlarged or reduced, and do not represent the actual size;

[0067] For those skilled in the art, it is understandable that some well-known content descriptions in the accompanying drawings may be omitted.

[0068] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0069] The description of the positional relationship in the accompanying drawings is only for illustrative purposes and should not be construed as limitations on this patent;

[0070] Embodiment 1

[0071] As Figure 1 shown, this embodiment proposes a method for generating hyperspectral point cloud data with three-dimensional spectrum integration. The schematic diagram of the process framework implemented by this method is as Figure 2 shown. Refer to Figure 1 and Figure 2 . The method includes the following steps:

[0072] S1. Use a mobile sensor device to obtain RGB image data, and obtain the RGB image as Figure 3 shown;

[0073] In this embodiment, it is also necessary to use a hyperspectral imager to collect RGB-HSI image pairs. The RGB image data in the RGB-HSI image pairs includes GNSS coordinates and the names of each band, and is stored in the metadata of the image; perform preprocessing operations on the obtained Figure 3 shown RGB image data. The preprocessing operations include distortion correction and stereo correction.

[0074] S2. Construct and train a multi-level spectral transformation network, input a single-frame RGB image into the trained multi-level spectral transformation network, reconstruct and generate a hyperspectral image. The hyperspectral image generated in this process is as Figure 4 shown;

[0075] S3. Construct and train a depth prediction network, input a single-frame RGB image into the trained depth prediction network, and obtain the initial depth information of the single-frame RGB image, asFigure 5 as shown;

[0076] S4. Determine the initial focal length of the camera of the mobile sensor device, and calculate and generate a distorted 3D point cloud based on the initial depth information and the initial camera focal length;

[0077] The calculation process of the distorted 3D point cloud is as follows:

[0078] Let the initial depth information of a single-frame RGB image predicted by the depth prediction network be d init and the initial camera focal length be f init . Through the coordinate transformation formula, the predicted initial depth information and the initial camera focal length are used to obtain the distorted 3D point cloud data:

[0079] where,

[0080] where, (u, v) is an arbitrary coordinate point in the image coordinate system, u 0 , v 0 are the center coordinates of the single-frame RGB image respectively, (x w , y w , z w ) represents the three-dimensional coordinate point in the world coordinate system, z c represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera, R and T are both external parameter matrices, d init is the initial depth information, and f init is the initial focal length.

[0081] S5. Construct a 3D point cloud generation network, input the distorted 3D point cloud data into the 3D point cloud generation network, obtain a depth offset adjustment factor and a focal length adjustment factor to adjust the predicted final depth information and camera focal length, and reconstruct and generate 3D point cloud data, as Figure 6 shown;

[0082] The 3D point cloud generation network mentioned above is composed of an offset point cloud network N d () and a focal length point cloud network N f ();

[0083] Among them, the input network architecture of the 3D point cloud generation network is a PVCNN network. Input the distorted 3D point cloud data into the 3D point cloud generation network to obtain a depth offset adjustment factor and a focal length adjustment factor, and adjust the predicted final depth information d pred and reconstruct the 3D point cloud data;

[0084] When adjusting and predicting the depth information, let the distorted 3D point cloud be and use it as the input data of the offset point cloud network N d (), and construct the training objective function as:

[0085]

[0086] Among them, θ is the network weight, and f * is the ground truth focal length, and d * is the ground truth depth, is the depth offset adjustment factor;

[0087] When adjusting and predicting the focal length, the distorted 3D point cloud is used as the input data of the focal length point cloud network N f (), and the training objective function is constructed:

[0088]

[0089] Among them, θ is the network weight, and f * is the ground truth focal length, and d * is the ground truth depth, is the focal length adjustment factor;

[0090] Repeat adjusting the depth information and the focal length several times, and then use the adjusted and predicted depth information d pred and the camera focal length f pred to reconstruct and generate 3D point cloud data through the coordinate transformation formula. The coordinate transformation formula is as follows:

[0091] Among them,

[0092] Among them, (u, v) is an arbitrary coordinate point in the image coordinate system, u 0 , v 0 are respectively the central coordinates of the image, (x w , y w , z w ) represents the three-dimensional coordinate point in the world coordinate system, z c represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera, R and T are the external parameter matrices respectively, d pred is the finally predicted depth information, and f pred is the finally predicted camera focal length.

[0093] In this embodiment, when training the 3D point cloud generation network, first sample 121K RGB-D images from the calibrated stereo DIML, sample data from RGB-D images from KITTI and 51K synthetic RGB-D images from the 3D Ken Burns paper. During training, retain 1000 images from all datasets as the validation set, resize the images to 448×448, and horizontally flip them with a 50% probability. Each batch evenly loads image data from different datasets during the training process. In addition, use the Adam optimizer for network optimization, with the momentum set to 0.8. For hyperparameter settings, the initial learning rate of the network is fixed at 0.02, the batch size batch is 40, and the learning rate decays by 0.1.

[0094] S6. Process the hyperspectral image generated in S2 to obtain the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image, and process the 3D point cloud data generated in S5 to obtain the band information corresponding to each pixel point on the 3D point cloud;

[0095] Load and read the hyperspectral image data generated in step S2, and convert the hyperspectral image data into a multi-dimensional matrix of the hyperspectral image. Then loop through each element in the multi-dimensional matrix of the hyperspectral image. Each such element is a pixel point of the hyperspectral image. During the traversal, obtain the matrix corresponding to each pixel point, and this matrix represents the spectral information of 31 bands corresponding to each pixel point. Finally, obtain the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image;

[0096] The steps for performing point cloud processing on the 3D point cloud generated by three-dimensional reconstruction in step S5 are as follows:

[0097] Load and read the point cloud data, convert the point cloud data into a multi-dimensional matrix of the point cloud, perform point cloud outlier filtering operations, and then loop through each element in the multi-dimensional matrix of the point cloud. Each such element is a pixel point of the point cloud. Remove the NAN points in the point cloud. During the traversal, obtain the matrix corresponding to each pixel point in the point cloud, and this matrix represents the corresponding multiple band information for each point. Finally, obtain the 6-band information (x, y, z, R, G, B) corresponding to each pixel point on the 3D point cloud.

[0098] S7. End-to-end map the spectral information of multiple bands corresponding to the pixel points of the hyperspectral image onto the band information of the pixel points of the 3D point cloud to obtain a hyperspectral point cloud with multi-band spectral information, as Figure 7 shown.

[0099] Loop through each pixel point on the 3D point cloud and hyperspectral image, use the SIFT feature matching algorithm to extract and match the pixel point features of the 3D point cloud and hyperspectral image, find the matching pixel point pairs. After the matching is completed, map the spectral information of multiple bands of the corresponding pixel points of the hyperspectral image end-to-end to the band information of the matching 3D point cloud pixel points, and store all the band information in the form of point cloud data, so as to obtain the hyperspectral point cloud data with three-dimensional spectrum integration with multi-band spectral information.

[0100] Embodiment 2

[0101] This embodiment describes the process of constructing the multi-level spectral transformation network, training in step S2, inputting a single-frame RGB image into the trained multi-level spectral transformation network, reconstructing and generating a hyperspectral image, the process of constructing the depth prediction network in step S3, training, and inputting a single-frame RGB image into the trained depth prediction network to obtain the initial depth information of the single-frame RGB image.

[0102] In step S2, the multi-level spectral transformation network is composed of a cascade of multiple single-level spectral transformation modules. Each single-level spectral transformation network is composed of a U-shaped structure module of Encoder-Decoder. The U-shaped structure module is composed of multiple spectral attention modules. Each spectral attention module includes a spectral multi-head self-attention module with the properties of HSI space sparsity and spectral self-similarity and a feed-forward network module;

[0103] In the multi-level spectral transformation network, first read the input RGB image data and convert the RGB image data into multi-dimensional matrix representation information , as the input of the spectral multi-head self-attention module S-MSA, where H represents the height of the multi-dimensional matrix, W represents the width of the multi-dimensional matrix, and C represents the spectral channel number of the multi-dimensional matrix; the spectral multi-head self-attention module S-MSA reshapes X in into , linearly projects X to and , where Q is query, K is key, and V is value, and the linear projection satisfies:

[0104] Q = XW Q , K = XW K , V = XW V , where are learnable parameters;

[0105] Along the spectral channel, Q, K, and V are respectively divided into N heads, and the dimension of each head where C is the number of spectral channels; according to each spectral representation, calculate the spectral self-attention A of head j ​j , meanwhile, using learnable parameters and reweighted matrix multiplication to adjust the spectral self-attention A j , the expression is:

[0106]

[0107] where represents the transposed matrix of K j ; subsequently, by introducing the position embedding function f p () to encode the position information of different spectral channels, and connecting the output information of N heads for linear projection, satisfying:

[0108]

[0109] where is a learnable parameter, and f p (·) is the position embedding function;

[0110] Finally, by changing the dimension of the data, the output result of S-MSA(X) is reshaped to obtain the output feature information ; using the LN normalization layer to normalize the feature information to obtain , taking as the input of the feed-forward network module, using Conv 1×1 to perform channel dimensionality reduction on , introducing the GELU activation function to increase its non-linearity factor, satisfying the expression:

[0111] ; obtaining a feature map with the same dimension as through DW Conv 3×3, using the GELU activation function and Conv 1×1 to construct a bottleneck layer for feature map channel dimensionality increase, and finally outputting the feature information

[0112] In the spectral attention module, first, the obtained by reducing the dimension of the original RGB image data through the Conv 3×3 layer and the Embedding layer in the single-stage spectral transformation network module is used as the input of the spectral attention module, and is normalized using the LN layer to obtain the feature information , then X in obtains the feature information after passing through the spectral multi-head self-attention module, and X out and X 0 perform a matrix addition operation to obtain the feature information . After normalization by the LN layer, the feature information is obtained Finally, X out_0_LN After passing through the feedforward network module, the feature information X is output out_0_LN_FFN , and it is combined with to perform a matrix addition operation to obtain the feature information finally output by the spectral attention module

[0113] In the single-level spectral transformation network module, the single-level spectral transformation network is composed of the U-shaped structure module of Encoder-Decoder. Among them, the Encoder encoder is composed of multiple spectral attention modules, an Embedding layer, and a DownSample layer; the Decoder encoder is composed of multiple spectral attention modules, a Mapping layer, and an UpSample layer; first, the data obtained after dimensionality reduction of the original RGB image data through the Embedding layer is used as the input data of the Encoder encoder. After encoding and downsampling by the Encoder, the low-dimensional feature information X is output 2 , then the spectral attention module is set as the Bottleneck layer; X 2 After passing through the Bottleneck layer, the feature information X is output 2_o , then X 2_o is used as the input of the Decoder decoder. After decoding and upsampling by the Decoder decoder, the feature information is output , then X 0 is mapped to obtain the feature information X with multi-resolution context information SST_out .

[0114] Finally, the multi-level spectral transformer network model is composed of N s cascaded single-level spectral transformation network modules, which can gradually improve the spectral super-resolution reconstruction quality from coarse to fine; first, the single-frame RGB image data is used as the input data of the multi-level spectral transformation network model, and the Conv 3×3 is used to extract the feature information of the RGB image , then X RGB is input into the single-level spectral transformation network module, and the feature information X with multi-resolution context information is output SST_out ; then the Conv 3×3 is used to extract the spectral feature information from X SST_out , and then it is combined with X RGB to perform a matrix addition operation to obtain a hyperspectral image with multi-band spectral information.

[0115] After the multi-level spectral transformation network is constructed, it is trained as follows:

[0116] Use the self - collected dataset of 128×128 RGB - HSI image pairs obtained in step S1 and the NTIRE 2022 public dataset to train the multi - level spectral transformation network. During the training process, linearly rescale the RGB image data to [0, 1], then crop a dataset of 128×128 RGB - HSI sample pairs from the NTIRE 2022 dataset, and supplement the self - collected data of 128×128 RGB - HSI image pairs to the cropped dataset. The training data is enhanced by random rotation and flipping; in addition, set the batch size batch to 25, and select the optimizer Adam with parameters β 1 = 0.8 and β 2 = 0.888, initialize the learning rate to 0.0003, and adopt the Cosine Annealing scheme for 300 epochs.

[0117] The depth prediction network constructed in step S3 includes the standard backbone ResNet50 for feature extraction and a Decoder decoder for predicting unknown - scale depth information. After completing the construction of the depth prediction network, train the depth prediction network;

[0118] Obtain RGB - D image data from the open - source dataset, and use the RGB - D image data to train the depth prediction network. During the training process, normalize the depth range of the RGB - D image data to [0, 1], and perform image normalization regression loss processing. The image normalization regression loss ILNR function is:

[0119]

[0120] where L ILNR represents the image normalization regression loss ILNR function, , d i represents the predicted depth information, represents the ground - truth depth information; μ trim and σ trim represent the mean and standard deviation of the depth map respectively;

[0121] Use the pairwise normal regression PWN loss to improve the performance of local geometric feature learning. The expression of the pairwise normal regression PWN loss function is:

[0122]

[0123] where L PWN is the pairwise normal regression PWN loss function. Let the sampling points be {(A i , B i ), i = 0,..., N}, then the surface normals corresponding to the sampling points are {(n Ai , nBi ) where \(i = 0,\cdots,N\); \(n\) * represents the ground truth surface normal; is the ground truth surface normal of \((A\) i , \(B\) i ). The image normalization regression loss \(I_{LNR}\) function and the normal regression \(P_{WN}\) loss function together constitute the total loss function of the depth prediction network;

[0124] In this embodiment, when training the depth prediction network, first sample data from the calibrated stereo DIML's 121K RGB - D images, RGB - D images from KITTI, and 51K synthetic RGB - D images from 3D Ken Burns. During training, reserve 1000 images from all datasets as the validation set, resize the images to \(448\times448\), and flip them horizontally with a 50% probability. Load data from different datasets evenly for each batch, and use the Adam optimizer with a batch size of 40 for network optimization, with the momentum set to 0.8. For hyperparameter settings, the initial learning rate of the network is fixed at 0.02, and the learning rate decays by 0.1. After inputting the single - frame RGB image data into the trained depth prediction network, the initial depth information of the single - frame RGB image is obtained.

[0125] Embodiment 3

[0126] In this embodiment, as Figure 8 shown, a system for generating three - dimensional atlas - unified hyperspectral point cloud data is proposed. The system includes:

[0127] A mobile sensor device 101 for acquiring RGB image data;

[0128] A hyperspectral image reconstruction and generation unit 102 for constructing and training a multi - level spectral transformation network, inputting a single - frame RGB image into the trained multi - level spectral transformation network, and reconstructing and generating a hyperspectral image;

[0129] An image depth information acquisition unit 103 for constructing and training a depth prediction network, inputting a single - frame RGB image into the trained depth prediction network, and obtaining the initial depth information of the single - frame RGB image;

[0130] A distorted 3D point cloud generation unit 104 for determining the initial focal length of the camera of the mobile sensor device, and calculating and generating a distorted 3D point cloud based on the initial depth information and the initial camera focal length;

[0131] A 3D point cloud generation unit 105 for constructing a 3D point cloud generation network, using the distorted 3D point cloud data as input, obtaining depth offset adjustment factors and focal length adjustment factors to adjust the predicted final depth information and camera focal length, and reconstructing and generating 3D point cloud data;

[0132] The band information acquisition unit 106 is configured to perform data processing on the hyperspectral image obtained by the hyperspectral image reconstruction generation unit, acquire the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image, perform point cloud processing on the 3D point cloud data generated by the 3D point cloud generation unit, and acquire the band information corresponding to each pixel point on the 3D point cloud;

[0133] The three-dimensional spectrum integration unit 107 maps the spectral information of multiple bands corresponding to the pixel points of the hyperspectral image end-to-end to the band information of the pixel points of the 3D point cloud, and obtains a hyperspectral point cloud with multi-band spectral information.

Claims

1. A method for generating hyperspectral point cloud data with three-dimensional spectrum integration, characterized in that, it includes: S1. Use a mobile sensor device to obtain RGB image data; S2. Construct and train a multi-level spectral transformation network, input a single-frame RGB image into the trained multi-level spectral transformation network, and reconstruct and generate a hyperspectral image; S3. Construct and train a depth prediction network, input a single-frame RGB image into the trained depth prediction network, and obtain the initial depth information of the single-frame RGB image; S4. Determine the initial focal length of the camera of the mobile sensor device, and calculate and generate a distorted 3D point cloud based on the initial depth information and the initial camera focal length; S5. Construct a 3D point cloud generation network, input the distorted 3D point cloud data into the 3D point cloud generation network, obtain a depth offset adjustment factor and a focal length adjustment factor to adjust the predicted final depth information and camera focal length, and reconstruct and generate 3D point cloud data; The 3D point cloud generation network constructed in step S5 consists of an offset point cloud network and a focal length point cloud network and is composed of; After constructing the 3D point cloud generation network, use the Adam optimizer to train the 3D point cloud generation network to optimize its network parameters; Among them, the input network architecture of the 3D point cloud generation network is the PVCNN network. The distorted 3D point cloud data is input into the 3D point cloud generation network to obtain the depth offset adjustment factor and the focal length adjustment factor, and finally the predicted depth information is adjusted and the camera focal length as well as the reconstructed 3D point cloud data; When adjusting and predicting depth information, assume that the distorted 3D point cloud is , which is used as the input data of the offset point cloud network , and the training objective function is constructed as: |； where θ is the network weight, is the ground truth focal length, is the ground truth depth, is the depth offset adjustment factor; is the center coordinate of the image; When adjusting and predicting the camera focal length, the distorted 3D point cloud is used as the input data of the focal length point cloud network to construct the training objective function: ； where θ is the network weight, is the ground truth focal length, is the ground truth depth, is the focal length adjustment factor; S6. Perform data processing on the hyperspectral image generated in S2 to obtain spectral information of multiple bands corresponding to each pixel point of the hyperspectral image, perform point cloud processing on the 3D point cloud data generated in S5 to obtain the band information corresponding to each pixel point on the 3D point cloud; S7. End-to-end map the spectral information of multiple bands corresponding to the pixel points of the hyperspectral image to the band information of the pixel points of the 3D point cloud to obtain a hyperspectral point cloud with multi-band spectral information.

2. The method for generating hyperspectral point cloud data with three-dimensional spectrum integration according to claim 1, characterized in that, in step S1, it further includes the process of using a hyperspectral imager to obtain an RGB-HSI image pair. The RGB image data in the RGB-HSI image pair includes GNSS coordinates and the names of each band, which are stored in the metadata of the image; perform preprocessing operations on the obtained RGB image data, and the preprocessing operations include distortion correction and stereo correction.

3. The method for generating hyperspectral point cloud data with three-dimensional spectrum integration according to claim 2, characterized in that, in step S2, the multi-level spectral transformation network is composed of a cascade of multiple single-level spectral transformation modules. Each single-level spectral transformation network is composed of a U-shaped structure module of Encoder-Decoder. This U-shaped structure module is composed of multiple spectral attention modules. Each spectral attention module includes a spectral multi-head self-attention module with the properties of HSI space sparsity and spectral self-similarity and a feed-forward network module.

4. The method for generating hyperspectral point cloud data with three-dimensional spectrum integration according to claim 3, characterized in that, after the multi-level spectral transformation network is constructed, train it in the following way: Based on the Adam optimizer, use the self - collected dataset of 128×128 RGB - HSI image pairs obtained in step S1 and the NTIRE 2022 public dataset to train the multi - level spectral transformation network. During the training process, linearly rescale the RGB image data to [0, 1], then crop out the 128×128 RGB - HSI sample pair dataset from the NTIRE 2022 dataset, and supplement the self - collected data of 128×128 RGB - HSI image pairs to the cropped dataset. The training data is enhanced by random rotation and flipping.

5. The method for generating hyperspectral point cloud data with three - dimensional map integration according to claim 1, characterized in that, the depth prediction network constructed in step S3 includes a standard backbone ResNet50 for feature extraction and a Decoder decoder for predicting unknown - scale depth information. After completing the construction of the depth prediction network, train the depth prediction network; Obtain RGB - D image data from an open - source dataset, and use the RGB - D image data to train the depth prediction network. During the training process, normalize the depth range of the RGB - D image data to [0, 1], and perform image normalization regression loss processing. The image normalization regression loss ILNR function is: Among them, represents the image normalization regression loss ILNR function, , represents the predicted depth information, represents the ground truth depth information; and respectively represent the mean and standard deviation of the depth map; Use the pairwise normal regression (PWN) loss to improve the performance of local geometric feature learning. The expression of the pairwise normal regression PWN loss function is: ； Among them, For the paired normal regression PWN loss function, assume that the sampling points are {( , ), i = 0, ..., N}, then the surface normals corresponding to the sampling points are {( , ), i = 0, ..., N}; denotes the ground truth surface normal; ([[]] , ) is the ground truth surface normal of ( , ); The image normalization regression loss ILNR function and the normal regression PWN loss function together constitute the total loss function of the depth prediction network. Then, the Adam optimizer is used to train the depth prediction network to optimize its network parameters until the total loss function of the depth prediction network converges, and a trained depth prediction network is obtained.​ 6. The method for generating hyperspectral point cloud data with three - dimensional map integration according to claim 5, characterized in that, the calculation process of the distorted 3D point cloud in step S4 is as follows: Let the initial depth information of a single-frame RGB image predicted by the depth prediction network be , the initial camera focal length be . The predicted initial depth information and initial camera focal length are used to obtain distorted 3D point cloud data through the coordinate transformation formula: , wherein, , Among them, is an arbitrary coordinate point in the image coordinate system, are the central coordinates of a single-frame RGB image respectively, represents the three-dimensional coordinate point in the world coordinate system, represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera. Both R and T are external parameter matrices, is the initial depth information, is the initial focal length.

7. The method for generating hyperspectral point cloud data with three - dimensional map integration according to claim 6, characterized in that, Adjust the depth information and the camera focal length repeatedly for several times, and then adjust the predicted depth information with the camera focal length Reconstruct and generate 3D point cloud data through the coordinate transformation formula as follows: , wherein, , Among them, is an arbitrary coordinate point in the image coordinate system, are the central coordinates of the image respectively, represents the three-dimensional coordinate point in the world coordinate system, represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera, and R and T are the external parameter matrices respectively, is the finally predicted depth information, is the finally predicted camera focal length.

8. The method for generating hyperspectral point cloud data with three - dimensional map integration according to claim 7, characterized in that, In step S6, load and read the hyperspectral image data generated in step S2, and convert the hyperspectral image data into a multi - dimensional matrix of the hyperspectral image. Then loop through each element in the multi - dimensional matrix of the hyperspectral image. Each element is a pixel point of the hyperspectral image. During the traversal, obtain the matrix corresponding to each pixel point, and this matrix represents the spectral information of multiple bands corresponding to each pixel point. Finally, obtain the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image; The steps for performing point cloud processing operations on the 3D point cloud generated by three - dimensional reconstruction in step S5 are as follows: Load and read the point cloud data, convert the point cloud data into a multi - dimensional matrix of the point cloud, perform point cloud outlier filtering operations, and then loop through each element in the multi - dimensional matrix of the point cloud. Each element is a pixel point of the point cloud. Remove the NAN points in the point cloud. During the traversal, obtain the matrix corresponding to each pixel point in the point cloud, and this matrix represents the information of multiple bands corresponding to each point. Finally, obtain the band information corresponding to each pixel point on the 3D point cloud.

9. The method for generating hyperspectral point cloud data with three - dimensional map integration according to claim 8, characterized in that, In step S7, the 3D point cloud and each pixel point on the hyperspectral image are traversed in a loop. The SIFT feature matching algorithm is used to extract and match the pixel point features of the 3D point cloud and the hyperspectral image, and the matching pixel point pairs are found. After the matching is completed, the spectral information of multiple bands corresponding to the pixel points of the hyperspectral image is mapped end-to-end to the band information of the pixel points of the matched 3D point cloud, and all the band information is stored in the form of point cloud data, so as to obtain the hyperspectral point cloud data with multi-band spectral information and three-dimensional map integration.

10. A system for generating hyperspectral point cloud data with three-dimensional map integration, characterized in that, the system includes: A mobile sensor device for acquiring RGB image data; A hyperspectral image reconstruction and generation unit for constructing and training a multi-level spectral transformation network, inputting a single-frame RGB image into the trained multi-level spectral transformation network, and reconstructing and generating a hyperspectral image; An image depth information acquisition unit for constructing and training a depth prediction network, inputting a single-frame RGB image into the trained depth prediction network, and obtaining the initial depth information of the single-frame RGB image; A distorted 3D point cloud generation unit for determining the initial focal length of the camera of the mobile sensor device, and calculating and generating a distorted 3D point cloud according to the initial depth information and the initial camera focal length; A 3D point cloud generation unit for constructing a 3D point cloud generation network, taking the distorted 3D point cloud data as input, obtaining a depth offset adjustment factor and a focal length adjustment factor to adjust and predict the final depth information and camera focal length, and reconstructing and generating 3D point cloud data; The constructed 3D point cloud generation network consists of an offset point cloud network and a focal length point cloud network and is composed of them; After constructing the 3D point cloud generation network, use the Adam optimizer to train the 3D point cloud generation network to optimize its network parameters; Among them, the input network architecture of the 3D point cloud generation network is the PVCNN network. The distorted 3D point cloud data is input into the 3D point cloud generation network to obtain the depth offset adjustment factor and the focal length adjustment factor, and finally the predicted depth information, the camera focal length and the reconstructed 3D point cloud data are adjusted. ​ When adjusting and predicting depth information, assume that the distorted 3D point cloud is , and use it as the input data of the offset point cloud network . The training objective function is constructed as follows: |； where θ is the network weight, is the ground truth focal length, is the ground truth depth, is the depth offset adjustment factor; When adjusting and predicting the camera focal length, the distorted 3D point cloud is used as the input data of the focal length point cloud network to construct the training objective function: ； where θ is the network weight, is the ground truth focal length, is the ground truth depth, is the focal length adjustment factor; is the center coordinate of the image; A band information acquisition unit for processing the data of the hyperspectral image obtained by the hyperspectral image reconstruction and generation unit, acquiring the spectral information of multiple bands corresponding to each pixel point of the hyperspectral image, and performing point cloud processing on the 3D point cloud data generated by the 3D point cloud generation unit to acquire the band information corresponding to each pixel point on the 3D point cloud; A three-dimensional map integration unit that maps the spectral information of multiple bands corresponding to the pixel points of the hyperspectral image end-to-end to the band information of the pixel points of the 3D point cloud to obtain a hyperspectral point cloud with multi-band spectral information.

Images