Shallow and deep multispectral point cloud generation method based on multispectral image data and laser radar point cloud data

Abstract

The invention discloses a shallow-layer and deep-layer multispectral point cloud generation method based on multispectral image data and laser radar point cloud data, and relates to the field of remote sensing data processing, and the method comprises the steps: obtaining a multispectral image and laser radar point cloud data of a target region; the method comprises the following steps: calculating a shallow multispectral feature map through ENVI software, constructing a three-dimensional multispectral point cloud fusion model, inputting a shallow / deep multispectral feature map and point cloud data into the three-dimensional multispectral point cloud fusion model, and realizing spatial registration of a multispectral image grid coordinate system and a point cloud plane coordinate system through affine transformation. And respectively generating shallow multi-spectral point cloud data and deep multi-spectral point cloud data. According to the invention, shallow and deep multispectral feature maps are respectively extracted, efficient fusion of the multispectral image and the laser radar point cloud is realized based on the constructed three-dimensional multispectral point cloud fusion model, and the defects that the point cloud data lacks spectral information and the hyperspectral point cloud data is high in cost are overcome.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing data processing, and specifically is a method for fusing and generating shallow multispectral point clouds and deep multispectral point clouds based on multispectral image data and laser radar point cloud data. Background Art

[0002] With the rapid development of remote sensing science, active and passive remote sensing technologies have been widely used in the classification and change monitoring of mangrove wetland vegetation, mainly including various satellite / airborne multispectral, hyperspectral and synthetic aperture radar (SAR) images. Multispectral and hyperspectral data with high spatial resolution can accurately restore the essential characteristics of the spectral and geometric properties of objects on a two-dimensional basis, but are easily affected by external factors such as climate and light. SAR images can effectively avoid the influence of external factors and provide all-day and all-weather remote sensing data even at night and in bad weather conditions. However, to date, active and passive remote sensing in the study of mangrove species classification is mostly traditional two-dimensional species classification, and its classification results still have obvious shortcomings: (1) Optical images have limited classification capabilities for mangrove species with small spectral differences but large structural differences; (2) Two-dimensional mapping results can only provide mangrove canopy information, lack spatial structure information, and cannot achieve three-dimensional monitoring of mangroves. Three-dimensional mapping of mangrove species provides a new idea for solving the above problems.

[0003] LiDAR point cloud provides a new direction for 3D mapping of mangrove species, but the current 3D classification research based on LiDAR point cloud is mostly focused on indoor scenes, urban scenes, intelligent driving and forest protection monitoring. Most of the research in mangrove scenes is to rasterize LiDAR point cloud and then combine it with other optical images for 3D classification and inversion mapping of mangroves. Moreover, it is impossible to achieve fine identification of mangrove species with similar canopy structures but different spectral characteristics using only point cloud. Therefore, it is necessary to construct a multi-source fused point cloud with spectral information and spatial structure information by fusing optical images and point cloud data to achieve high-precision 3D mapping of mangrove species. At present, the multi-source fused point cloud data used for vegetation classification in natural scenes are mostly hyperspectral-point clouds, but high-resolution hyperspectral data are usually accompanied by high acquisition cost, high noise sensitivity, large data volume and complex processing, which are severely limited in large-scene monitoring and time-series dynamic monitoring. Multispectral images just make up for the above shortcomings. Fusion of multispectral images and LiDAR point cloud data provides a new option for realizing large-scale 3D mapping of mangrove populations.

[0004] Due to the huge differences between different multispectral features, their classification results for three-dimensional point clouds may also be very different. In order to improve the classification accuracy and efficiency of multispectral point clouds in mangrove scenes, it is imperative to explore the optimal feature combination of multispectral point clouds. In addition, many studies have shown that two-dimensional deep learning classification algorithms can effectively extract deep features from local to global images, thereby achieving high-precision two-dimensional mapping, but whether these deep features can effectively improve the classification ability of three-dimensional point clouds remains to be further studied. In the existing multi-source fusion point cloud research based on data-level fusion methods, shallow spectral features (original bands, vegetation indices, and texture features, etc.) are usually directly projected into point clouds, and no attempt has been made to use the backbone network of a two-dimensional deep learning segmentation model to extract deep multispectral features and then project them into point clouds. The impact of two-dimensional deep multispectral features with deep local and global features and shallow multispectral features on three-dimensional mapping of mangrove species is still unclear.

[0005] Therefore, how to generate shallow multispectral point clouds and deep multispectral point clouds based on the fusion of multispectral image data and lidar point cloud data is an urgent problem to be solved. Summary of the invention

[0006] Based on this, the primary purpose of the present invention is to provide a shallow and deep multispectral point cloud generation method based on multispectral image data and lidar point cloud data, which realizes the precise fusion of multispectral image and lidar point cloud.

[0007] Another object of the present invention is to provide a shallow and deep multispectral point cloud generation method based on multispectral image data and lidar point cloud data, the method extracts the shallow multispectral feature map and the deep multispectral feature map of the multispectral original image data respectively; and constructs a three-dimensional multispectral point cloud fusion model based on the affine transformation principle, and realizes the efficient fusion of multispectral images and lidar point clouds by establishing a mapping index between pixels in multispectral images and points in point clouds, and successfully generates shallow and deep multispectral point clouds for three-dimensional fine-grained mapping, which makes up for the defects of lack of spectral information in point cloud data and high cost of hyperspectral point cloud data.

[0008] Another object of the present invention is to provide a shallow and deep multispectral point cloud generation method based on multispectral image data and lidar point cloud data, which can flexibly and efficiently obtain low-cost multispectral point clouds, realize large-scale three-dimensional mapping of mangrove species, and restore the three-dimensional structure and spatial distribution of mangroves.

[0009] To achieve the above object, the technical solution of the present invention is:

[0010] A method for generating shallow and deep multispectral point clouds based on multispectral image data and lidar point cloud data, the specific steps are as follows:

[0011] Step (1): Obtain multispectral image data and point cloud data of the target area;

[0012] Step (2): preprocessing the multispectral image data and point cloud data;

[0013] Step (3): Calculate the shallow multispectral feature map using ENVI software;

[0014] The shallow multispectral feature map includes three category features:

[0015] Original multispectral bands: red, green, blue, red edge and near infrared, a total of 5 bands;

[0016] Multispectral texture features: 8 types of texture features (Mean, Variance, Homogeneity, Contrast, Dissimilarity, Entropy, Second Moment, Correlation) for each of the 5 bands of the multispectral image, a total of 40 texture features are extracted;

[0017] Vegetation Index: ARVI, BGRI, BNDVI, BVI, CARI, CCRI, CIBlue, CIgreen, Cire, CIred, CIVE, CMRI, DGBVI, DRBVI, DVI, EVI, EVI2 , EXG, EXGR, FDI, GARI, GBDI, GCC, GDVI, GEMI, GIPVI, GNDVI, GRVI, GVI, IKAW, IMFI, IPVI, LAI, MCARI2, MGRVI, MNGR BVI, MNGRVI, MNLI, MRBVI, MSAVI, MSR, MSRre, MTCI, MTVI, MTVI2, NDCI, NDDI, NDVI, NDVIre, NDWI, NGBDI, NGBDI1, N GRDI, NGRDI1, NLI, OSAVI, RBRI, RDVI, RGBVI, RGRI, SAVI, SIPI, SR, SRRed, TDVI, TVI, VARI, VDVI, VEG, a total of 69 vegetation indices;

[0018] The three types of shallow multi-spectral feature maps were respectively subjected to high correlation elimination (Pearson correlation coefficient) and feature optimization (MUVR variable selection algorithm) to screen out shallow multi-spectral feature maps that meet the requirements;

[0019] Step (4): extracting deep multispectral feature maps through deep learning algorithms;

[0020] The shallow multispectral features are refined using the backbone networks of the HRNet algorithm of the CNN architecture and the Swin Transformer algorithm of the Transformer architecture to obtain the corresponding deep multispectral features, and the number of deep multispectral features output by each scheme is ensured to be consistent with the number of shallow multispectral features input;

[0021] Step (5): construct a three-dimensional multispectral point cloud fusion model;

[0022] The constructed 3D multispectral point cloud fusion model can efficiently project 2D raster image information into 3D point cloud and generate high-quality 3D multispectral point cloud data. The specific fusion process of the model is as follows:

[0023] The fusion process of multispectral point cloud is mainly divided into three parts:

[0024] ①Evenly select 9 corresponding control points in the grid coordinate system of the multispectral image and the xy plane of the point cloud, and record the grid coordinates (X i , Y i ) and the xy plane coordinates of the point cloud (x i ,y i )where (i=1, 2, L, 9);

[0025] ② Select the three control points with the widest coverage after connecting the lines from the nine control points to calculate the six unknown parameters in the affine transformation matrix M;

[0026] ③ The remaining 6 control points are used for affine transformation inverse calculation to verify the conversion accuracy of the affine transformation matrix, and the error range is set to 0.2m (the minimum interval between two points after point cloud downsampling);

[0027] ④ The xy plane coordinates of the three-dimensional point cloud are matched one by one with the grid coordinates of the two-dimensional multispectral image through the affine transformation matrix M, and a unique index between the point cloud and the pixel is constructed. The pixel value (multispectral feature information) of the two-dimensional multispectral image is projected into the three-dimensional point cloud according to the index to form a new attribute of the point cloud, and finally a multispectral point cloud is generated.

[0028] Affine transformation formula:

[0029]

[0030] B * =M -1 A(i=4,…,9)

[0031] Step (6): Input the shallow / deep multispectral feature map and point cloud data into the three-dimensional multispectral point cloud fusion model to generate shallow multispectral point cloud and deep multispectral point cloud data respectively.

[0032] Furthermore, in step (1), the multispectral image data and the point cloud data are acquired respectively by two non-integrated and non-rigidly fixed airborne multispectral sensors and an airborne lidar sensor.

[0033] Furthermore, in step (2), the pretreatment process includes:

[0034] The original multispectral image data is subjected to radiation calibration, orthorectification, image matching, image calibration, image stitching, geographic registration, image stacking and image cropping.

[0035] The point cloud data is spatially downsampled, subjected to statistical filtering and denoising processing, and the three-dimensional geometric features (normal vector, curvature and density) of the point cloud are calculated.

[0036] Furthermore, in step (4), the main process of HRNet extracting deep multispectral features is as follows:

[0037] ① The input shallow multispectral image is first downsampled by 4 times through the convolution layer and then input into the stage structure;

[0038] ② The four Bottleneck modules of the stage structure adjust the number of channels and extract features for each scale branch without changing the size of the feature layer;

[0039] ③ The features extracted by each branch in each stage are scaled by upsampling and downsampling, and then element-wise add is used for feature fusion. Then, nonlinear transformation is introduced through the ReLU activation function, and finally multi-scale fusion features are output;

[0040] ④ The multi-scale fusion features output by the smallest scale branch of each stage are added with a double down-sampled scale branch through the Transition structure, and are input into the next stage structure in parallel together with the multi-scale fusion features of other scale branches;

[0041] ⑤Finally, the multi-scale fusion features output by the four different scale branches are upsampled to the original image size and the images are stacked (concatenated) to obtain deep multispectral features.

[0042] The main process of Swin Transformer extracting deep multispectral features is as follows:

[0043] ① Patch Partition divides the input image into non-overlapping fixed-size local patches, and regards each patch as a patch token;

[0044] ② Adjust the feature dimension of the patch token through Linear Embedding and then input it into the Stage structure;

[0045] ③ Each stage consists of Patch Merging and Swin Transformer Block. Patch Merging downsamples the patch token. Swin Transformer Block can maintain sensitivity to local details while integrating global information, thereby enhancing feature representation capabilities.

[0046] ④The four stages generate hierarchical features of different resolutions respectively, which are upsampled to the original image size, and then image stacking operations are performed to finally obtain deep multispectral features.

[0047] Furthermore, in step (6), both the shallow multispectral point cloud and the deep multispectral point cloud are generated based on the three-dimensional multispectral point cloud fusion model, and the specific process is as follows:

[0048] The shallow multispectral feature map and point cloud data selected through high correlation elimination and feature optimization are input into the three-dimensional multispectral point cloud fusion model, and the feature information of the shallow multispectral feature map is projected into the corresponding point cloud to generate a shallow multispectral point cloud dataset.

[0049] The shallow multispectral feature map is input into two 2D deep learning segmentation models, HRNet and Swin Transformer. The shallow multispectral features are extracted from local to global two-dimensional deep features through the backbone network of two classic two-dimensional raster image segmentation models, and a two-dimensional deep multispectral feature map with the same depth as the input shallow multispectral feature map is output. Finally, the feature information of the deep multispectral feature map is projected onto the point cloud through the three-dimensional multispectral point cloud fusion model to generate a deep multispectral point cloud dataset.

[0050] Furthermore, a shallow and deep multispectral point cloud generation method based on multispectral image data and lidar point cloud data requires the use of various modules such as a data acquisition module, specifically including:

[0051] Data acquisition module: obtain multispectral image data and point cloud data of the target area;

[0052] Data preprocessing module: preprocessing the multispectral image data and point cloud data;

[0053] Shallow and deep multispectral feature extraction module: Calculate the shallow multispectral feature map of the preprocessed multispectral image through ENVI software, and perform feature screening; Input the shallow multispectral image after feature screening into the deep learning algorithm (HRNet and Swin Transformer), and use the backbone network architecture of the deep learning algorithm to extract the deep multispectral feature map;

[0054] Multispectral image and point cloud fusion module: The shallow / deep multispectral feature map and point cloud data are input into the three-dimensional multispectral point cloud fusion model, and the spatial alignment of the multispectral image grid coordinate system and the point cloud plane coordinate system is realized through affine transformation. The spectral feature data of the multispectral feature map pixel corresponding to each point in the aligned point cloud is mapped to the point, and the shallow multispectral point cloud and deep multispectral point cloud data are generated respectively.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention respectively extracts the shallow multispectral feature map and the deep multispectral feature map of the multispectral original image data; and constructs a three-dimensional multispectral point cloud fusion model based on the affine transformation principle. By establishing a mapping index between pixels in the multispectral image and points in the point cloud, the efficient fusion of the multispectral image and the lidar point cloud is achieved, and the shallow and deep multispectral point clouds for three-dimensional fine-grained mapping are successfully generated, which makes up for the defects of the lack of spectral information in the point cloud data and the high cost of the hyperspectral point cloud data.

[0057] At the same time, the shallow and deep multispectral point cloud generation method based on multispectral image data and lidar point cloud data provided by the present invention can flexibly and efficiently obtain low-cost multispectral point clouds, realize large-scale three-dimensional mapping of mangrove species, and restore the three-dimensional structure and spatial distribution of mangroves. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The figure is an overall flow chart of the implementation of the present invention.

[0059] Figure 2 It is a schematic diagram of the deep multi-spectral feature extraction process implemented by the present invention.

[0060] Figure 3 It is a multispectral point cloud fusion flow chart of the three-dimensional multispectral point cloud fusion model architecture and embodiment implemented by the present invention.

[0061] Figure 4 It is a comparison chart of the three-dimensional classification results of mangrove species using different shallow and deep multispectral point cloud data sets implemented in the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical scheme and advantages of the present invention more clear, the technical method in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, what is described is a part of the embodiments of the present application, not all of the embodiments. In addition, in the following description, the description of the known structure and the prior art is omitted to avoid unnecessary confusion of the concept of the present invention.

[0063] Figure 1 As shown, the embodiment of the present invention discloses a method for generating shallow and deep multispectral point clouds based on multispectral image data and laser radar point cloud data, and the specific steps are as follows:

[0064] Step (1): Obtain multispectral image data and point cloud data of the target area.

[0065] The multispectral image data and the point cloud data are acquired respectively by two non-integrated and non-rigidly fixed airborne multispectral sensors and an airborne laser radar sensor.

[0066] Step (2): pre-processing the multispectral image data and point cloud data.

[0067] Specifically, the original multispectral image data described in step (1) is subjected to radiation calibration, orthorectification, image matching, image calibration, image stitching, geographic registration, image stacking and image cropping.

[0068] The point cloud data is spatially downsampled, subjected to statistical filtering and denoising processing, and the three-dimensional geometric features (normal vector, curvature and density) of the point cloud are calculated.

[0069] Step (3): Calculate the shallow multispectral feature map using ENVI software.

[0070] Specifically, the multispectral image data processed in step (2) is input into ENVI software for calculation to obtain a shallow multispectral feature map, including three category features:

[0071] Original multispectral bands: red, green, blue, red edge and near infrared, a total of 5 bands.

[0072] Multispectral texture features: 8 types of texture features (Mean, Variance, Homogeneity, Contrast, Dissimilarity, Entropy, Second Moment, Correlation) for each of the 5 bands of the multispectral image, a total of 40 texture features are extracted.

[0073] Vegetation Index: ARVI, BGRI, BNDVI, BVI, CARI, CCRI, CIBlue, CIgreen, Cire, CIred, CIVE, CMRI, DGBVI, DRBVI, DVI, EVI, EVI2 , EXG, EXGR, FDI, GARI, GBDI, GCC, GDVI, GEMI, GIPVI, GNDVI, GRVI, GVI, IKAW, IMFI, IPVI, LAI, MCARI2, MGRVI, MNGR BVI, MNGRVI, MNLI, MRBVI, MSAVI, MSR, MSRre, MTCI, MTVI, MTVI2, NDCI, NDDI, NDVI, NDVIre, NDWI, NGBDI, NGBDI1, N GRDI, NGRDI1, NLI, OSAVI, RBRI, RDVI, RGBVI, RGRI, SAVI, SIPI, SR, SRRed, TDVI, TVI, VARI, VDVI, VEG, a total of 69 vegetation indices.

[0074] The three types of shallow multi-spectral feature maps were respectively eliminated based on high correlation (Pearson correlation coefficient) and feature optimization (MUVR variable selection algorithm) to screen out shallow multi-spectral feature maps that meet the requirements.

[0075] Step (4): Extract deep multispectral feature maps through deep learning algorithms.

[0076] The shallow multispectral features obtained in step (3) are refined using the backbone networks of the HRNet algorithm (CNN architecture) and the Swin Transformer algorithm (Transformer architecture) to obtain the corresponding deep multispectral features, and to ensure that the number of deep multispectral features output by each scheme is consistent with the number of shallow multispectral features input.

[0077] Figure 2 The specific extraction process is as follows:

[0078] The main process of HRNet to extract deep multispectral features is as follows:

[0079] ① The input shallow multispectral image is first downsampled by 4 times through the convolution layer and then input into the stage structure;

[0080] ② The four Bottleneck modules of the stage structure adjust the number of channels and extract features for each scale branch without changing the size of the feature layer;

[0081] ③ The features extracted by each branch in each stage are scaled by upsampling and downsampling, and then element-wise add is used for feature fusion. Then, nonlinear transformation is introduced through the ReLU activation function, and finally multi-scale fusion features are output;

[0082] ④ The multi-scale fusion features output by the smallest scale branch of each stage are added with a double down-sampled scale branch through the Transition structure, and are input into the next stage structure in parallel together with the multi-scale fusion features of other scale branches;

[0083] ⑤Finally, the multi-scale fusion features output by the four different scale branches are upsampled to the original image size and the images are stacked (concatenated) to obtain deep multispectral features.

[0084] The main process of Swin Transformer extracting deep multispectral features is as follows:

[0085] ① Patch Partition divides the input image into non-overlapping fixed-size local patches, and regards each patch as a patch token;

[0086] ② Adjust the feature dimension of the patch token through Linear Embedding and then input it into the Stage structure;

[0087] ③ Each stage consists of Patch Merging and Swin Transformer Block. Patch Merging downsamples the patch token. Swin Transformer Block can maintain sensitivity to local details while integrating global information, thereby enhancing feature representation capabilities.

[0088] ④The four stages generate hierarchical features of different resolutions respectively, which are upsampled to the original image size, and then image stacking operations are performed to finally obtain deep multispectral features.

[0089] Step (5): Construct a three-dimensional multispectral point cloud fusion model.

[0090] The fusion process of the model is as follows Figure 3 As shown, the details are as follows:

[0091] The fusion process of multispectral point cloud is mainly divided into three parts:

[0092] ①Evenly select 9 corresponding control points in the grid coordinate system of the multispectral image and the xy plane of the point cloud, and record the grid coordinates (X i , Y i ) and the xy plane coordinates of the point cloud (x i ,y i )where (i=1, 2, L, 9);

[0093] ② Select the three control points with the widest coverage after connecting the lines from the nine control points to calculate the six unknown parameters in the affine transformation matrix M;

[0094] ③ The remaining 6 control points are used for affine transformation inverse calculation to verify the conversion accuracy of the affine transformation matrix, and the error range is set to 0.2m (the minimum interval between two points after point cloud downsampling);

[0095] ④ The xy plane coordinates of the three-dimensional point cloud are matched one by one with the grid coordinates of the two-dimensional multispectral image through the affine transformation matrix M, and a unique index between the point cloud and the pixel is constructed. The pixel value (multispectral feature information) of the two-dimensional multispectral image is projected into the three-dimensional point cloud according to the index to form a new attribute of the point cloud, and finally a multispectral point cloud is generated.

[0096] Among them, the affine transformation formula is:

[0097]

[0098] B * =M -1 A(i=4,…,9)

[0099] Step (6): Input the shallow / deep multispectral feature map and point cloud data into the three-dimensional multispectral point cloud fusion model to generate shallow multispectral point cloud and deep multispectral point cloud data respectively.

[0100] Specifically, both the shallow multispectral point cloud and the deep multispectral point cloud are generated based on the three-dimensional multispectral point cloud fusion model, and the process is as follows:

[0101] The shallow multispectral feature map and point cloud data selected through high correlation elimination and feature optimization are input into the three-dimensional multispectral point cloud fusion model, and the feature information of the shallow multispectral feature map is projected into the corresponding point cloud to generate a shallow multispectral point cloud dataset.

[0102] The shallow multispectral feature map is input into two 2D deep learning segmentation models, HRNet and Swin Transformer. The shallow multispectral features are extracted from local to global two-dimensional deep features through the backbone network of two classic two-dimensional raster image segmentation models, and a two-dimensional deep multispectral feature map with the same depth as the input shallow multispectral feature map is output. Finally, the feature information of the deep multispectral feature map is projected onto the point cloud through the three-dimensional multispectral point cloud fusion model to generate a deep multispectral point cloud dataset.

[0103] like Figure 4 As shown, the present invention generates shallow multispectral point cloud and deep multispectral point cloud datasets of the target area respectively. In order to evaluate the performance of shallow and deep multispectral point cloud datasets in classification tasks, this specific embodiment uses two three-dimensional point cloud deep learning segmentation algorithms, PointNeXt and DWS-KP-FCNN, to systematically explore the three-dimensional mapping capabilities of shallow and deep multispectral point clouds in mangrove scenes. The results show that the generated shallow and deep multispectral point cloud datasets can effectively improve the three-dimensional classification effect of mangrove species.

[0104] The implementation of the present invention also includes various modules, specifically including:

[0105] Data acquisition module: obtain multispectral image data and point cloud data of the target area.

[0106] Data preprocessing module: preprocess the multispectral image data and point cloud data.

[0107] Shallow and deep multispectral feature extraction module: The shallow multispectral feature map of the preprocessed multispectral image is calculated by ENVI software, and feature screening is performed; the shallow multispectral image after feature screening is input into the deep learning algorithm (HRNet and Swin Transformer), and the deep multispectral feature map is extracted using the backbone network architecture of the deep learning algorithm.

[0108] Multispectral image and point cloud fusion module: The shallow / deep multispectral feature map and point cloud data are input into the three-dimensional multispectral point cloud fusion model, and the spatial alignment of the multispectral image grid coordinate system and the point cloud plane coordinate system is realized through affine transformation. The spectral feature data of the multispectral feature map pixel corresponding to each point in the aligned point cloud is mapped to the point, and the shallow multispectral point cloud and deep multispectral point cloud data are generated respectively.

[0109] In summary, the advantages of the present invention are:

[0110] The present invention respectively extracts the shallow multispectral feature map and the deep multispectral feature map of the multispectral original image data; and constructs a three-dimensional multispectral point cloud fusion model based on the affine transformation principle. By establishing a mapping index between pixels in the multispectral image and points in the point cloud, the efficient fusion of the multispectral image and the lidar point cloud is achieved, and the shallow and deep multispectral point clouds for three-dimensional fine-grained mapping are successfully generated, which makes up for the defects of the lack of spectral information in the point cloud data and the high cost of the hyperspectral point cloud data.

[0111] The present invention implements a shallow and deep multispectral point cloud generation method based on multispectral image data and lidar point cloud data, realizes large-scale three-dimensional mapping of mangrove species, and can restore the three-dimensional structure and spatial distribution of mangroves.

[0112] The specific embodiments described above are exemplary and non-limiting. Various solutions and all changes that can be thought of by those skilled in the art under the inspiration of the disclosure of the present invention are included in the present invention.

Claims

1. A method for generating shallow and deep multispectral point clouds based on multispectral image data and lidar point cloud data, characterized in that: Here are the steps: Step (1): Obtain multispectral image data and point cloud data of the target area; Step (2): preprocessing the multispectral image data and point cloud data; Step (3): Calculate the shallow multispectral feature map using ENVI software; The shallow multispectral feature map includes three category features: Original multispectral bands: red, green, blue, red edge and near infrared, a total of 5 bands; Multispectral texture features: 8 types of texture features of each of the 5 bands of the multispectral image, a total of 40 texture features are extracted; Vegetation Index: ARVI, BGRI, BNDVI, BVI, CARI, CCRI, CIBlue, CIgreen, Cire, CIred, CIVE, CMRI, DGBVI, DRBVI, DVI, EVI, EVI2 , EXG, EXGR, FDI, GARI, GBDI, GCC, GDVI, GEMI, GIPVI, GNDVI, GRVI, GVI, IKAW, IMFI, IPVI, LAI, MCARI2, MGRVI, MNGR BVI, MNGRVI, MNLI, MRBVI, MSAVI, MSR, MSRre, MTCI, MTVI, MTVI2, NDCI, NDDI, NDVI, NDVIre, NDWI, NGBDI, NGBDI1, N GRDI, NGRDI1, NLI, OSAVI, RBRI, RDVI, RGBVI, RGRI, SAVI, SIPI, SR, SRRed, TDVI, TVI, VARI, VDVI, VEG, a total of 69 vegetation indices; The three types of shallow multi-spectral feature maps are respectively subjected to high correlation elimination and feature optimization to screen out shallow multi-spectral feature maps that meet the requirements; Step (4): extracting deep multispectral feature maps through deep learning algorithms; The shallow multispectral features are refined using the backbone networks of the HRNet algorithm of the CNN architecture and the Swin Transformer algorithm of the Transformer architecture to obtain the corresponding deep multispectral features, and the number of deep multispectral features output by each scheme is ensured to be consistent with the number of shallow multispectral features input; Step (5): construct a three-dimensional multispectral point cloud fusion model; The constructed 3D multispectral point cloud fusion model can efficiently project 2D raster image information into 3D point cloud and generate high-quality 3D multispectral point cloud data. The specific fusion process of the model is as follows: The fusion process of multispectral point cloud is mainly divided into three parts: ①Evenly select 9 corresponding control points in the grid coordinate system of the multispectral image and the xy plane of the point cloud, and record the grid coordinates (X i , Y i ) and the xy plane coordinates of the point cloud (x i ,y i ), where (i=1, 2, L, 9); ② Select the three control points with the widest coverage after connecting the lines from the nine control points to calculate the six unknown parameters in the affine transformation matrix M; ③ The remaining 6 control points are used for affine transformation inverse calculation to verify the conversion accuracy of the affine transformation matrix; ④ The xy plane coordinates of the three-dimensional point cloud are matched one by one with the grid coordinates of the two-dimensional multispectral image through the affine transformation matrix M of the affine transformation formula, and a unique index between the point cloud and the pixel is constructed. The pixel value of the two-dimensional multispectral image is projected into the three-dimensional point cloud according to the index to form a new attribute of the point cloud, and finally a multispectral point cloud is generated; The affine transformation formula is: B * =M -1 A (i=4,…,9) Step (6): Input the shallow / deep multispectral feature map and point cloud data into the three-dimensional multispectral point cloud fusion model to generate shallow multispectral point cloud and deep multispectral point cloud data respectively.

2. The method for generating shallow and deep multispectral point clouds based on multispectral image data and lidar point cloud data according to claim 1, characterized in that: In step (3), the eight types of texture features are Mean, Variance, Homogeneity, Contrast, Dissimilarity, Entropy, Second Moment, and Correlation.

3. The method for generating shallow and deep multispectral point clouds based on multispectral image data and laser radar point cloud data according to claim 1, characterized in that: In step (3), the high correlation elimination of the above three types of shallow multi-spectral feature maps is achieved through the Pearson correlation coefficient, and the feature optimization is achieved through the MUVR variable selection algorithm to screen out the shallow multi-spectral feature maps that meet the requirements.

4. The method for generating shallow and deep multispectral point clouds based on multispectral image data and lidar point cloud data according to claim 1, characterized in that: In step (2), the pre-processing process includes: Performing radiometric calibration, orthorectification, image matching, image calibration, image stitching, geographic registration, image stacking and image cropping on the original multispectral image data; The point cloud data is spatially downsampled, subjected to statistical filtering and denoising, and three-dimensional geometric features of the point cloud are calculated, including normal vectors, curvature, and density.

5. The method for generating shallow and deep multispectral point clouds based on multispectral image data and laser radar point cloud data according to claim 1, characterized in that: In step (4), the main process of HRNet extracting deep multispectral features is as follows: ① The input shallow multispectral image is first downsampled by 4 times through the convolution layer and then input into the stage structure; ② The four Bottleneck modules of the stage structure adjust the number of channels and extract features for each scale branch without changing the size of the feature layer; ③ The features extracted by each branch in each stage are scaled by upsampling and downsampling, and then element-wise add is used for feature fusion. Then, nonlinear transformation is introduced through the ReLU activation function, and finally multi-scale fusion features are output; ④ The multi-scale fusion features output by the smallest scale branch of each stage are added with a double down-sampled scale branch through the Transition structure, and are input into the next stage structure in parallel together with the multi-scale fusion features of other scale branches; ⑤Finally, the multi-scale fusion features output by the four different scale branches are upsampled to the original image size and image stacked (concatenated) to obtain deep multispectral features; The main process of Swin Transformer extracting deep multispectral features is as follows: ① Patch Partition divides the input image into non-overlapping fixed-size local blocks, and regards each patch as a patch token; ② Adjust the feature dimension of the patch token through Linear Embedding and then input it into the Stage structure; ③ Each stage consists of Patch Merging and Swin Transformer Block. Patch Merging downsamples the patch token. Swin Transformer Block can maintain sensitivity to local details while integrating global information, thereby enhancing feature representation capabilities. ④The four stages generate hierarchical features of different resolutions respectively, which are upsampled to the original image size, and then image stacking operations are performed to finally obtain deep multispectral features.

6. The method for generating shallow and deep multispectral point clouds based on multispectral image data and lidar point cloud data according to claim 1, characterized in that: In the step (6), both the shallow multispectral point cloud and the deep multispectral point cloud are generated based on the three-dimensional multispectral point cloud fusion model. The specific process is as follows: The shallow multispectral feature map and point cloud data that have been filtered out through high correlation elimination and feature optimization are input into the 3D multispectral point cloud fusion model, and the feature information of the shallow multispectral feature map is projected onto the corresponding point cloud to generate a shallow multispectral point cloud data set. The shallow multispectral feature map is input into two 2D deep learning segmentation models, HRNet and Swin Transformer. The shallow multispectral features are extracted from local to global two-dimensional deep features through the backbone networks of two classic two-dimensional raster image segmentation models, and a two-dimensional deep multispectral feature map with the same depth as the input shallow multispectral feature map is output; then, the feature information of the deep multispectral feature map is projected onto the point cloud through the three-dimensional multispectral point cloud fusion model to generate a deep multispectral point cloud dataset.

Images