Classification method, training method and device of hyperspectral lidar point cloud data

Abstract

A hyperspectral laser radar point cloud data classification method, a training method and a device, the classification method comprising: determining the spectral reflectivity of a detection point according to the point cloud data of the detection point detected by the hyperspectral laser radar; determining a first image according to the spectral reflectivity of the detection point; determining the mean value of the spectral reflectivity of the spectral neighborhood point; determining a second image according to the mean value of the spectral reflectivity; inputting the first image into a first convolutional neural network branch of a trained hyperspectral laser radar point cloud data classification model to extract a first feature; inputting the second image into a second convolutional neural network branch of the trained hyperspectral laser radar point cloud data classification model to extract a second feature; inputting the first feature and the second feature into a splicing layer of the trained hyperspectral laser radar point cloud data classification model to splice a spliced feature; and inputting the spliced feature into a classifier of the trained hyperspectral laser radar point cloud data classification model to output a classification result.

Description

Technical Field

[0001] This disclosure relates to the field of lidar, and in particular to classification methods, training methods and devices for hyperspectral lidar point cloud data. Background Technology

[0002] LiDAR, as a modern optical remote sensing technology that utilizes active detection, combines traditional radar technology with modern laser technology, becoming a new technological development direction in the field of Earth observation. LiDAR point cloud classification involves acquiring semantic labels for each point in the LiDAR point cloud data, enabling the description and analysis of semantic attributes such as scene structure and object function. Accurate LiDAR point cloud classification is the foundation for complex indoor and outdoor visual interpretation and scene understanding.

[0003] Currently, the research focus on point cloud classification using lidar data both domestically and internationally primarily involves obtaining high-density and high-precision 3D point cloud data and then calculating geometric quantities such as distance variation, normal variation, curvature variation, elevation difference, density, and topological relationships to achieve point cloud classification and feature extraction. Lidar typically acquires very large amounts of point cloud data, making point cloud classification and feature extraction computationally intensive, inefficient, and inaccurate. Furthermore, some algorithms fail to yield accurate results when geometric differences are small. Additionally, traditional lidar systems are generally single-wavelength, limiting their ability to acquire target attribute information. The spectral information in the laser backscattering intensity data is relatively insufficient, resulting in limited detection capabilities for ground object categories. Fusion of passive optical hyperspectral remote sensing data with lidar data can compensate for the lack of spectral information in lidar, effectively improving the accuracy of ground object identification and classification. However, the biggest challenge in jointly interpreting data from different sensors is accurately registering different types of data to the same spatial coordinate system. Simultaneously, hyperspectral imaging is constrained by passive light source illumination conditions.

[0004] Hyperspectral lidar, as a novel remote sensing method developed in recent years, can overcome the limitations of natural lighting conditions and possesses both high spatial detection capabilities and the ability to detect the physical properties of ground objects. Besides acquiring the three-dimensional point cloud geometric information of the target surface, it can also simultaneously obtain the spectral information of ground objects, resulting in lidar point cloud data with hyperspectral information. The hyperspectral lidar system generates a point cloud [x,y,z,I] that combines hyperspectral backscattering intensity. i (λ)](i=1,2,…n), (x,y,z) represent the three-dimensional coordinates of a point in the point cloud data, I i(λ) represents the backscattering intensity at that point, where the backscattering intensity is a continuous function of wavelength λ. Since different target surfaces have different reflectivities for different wavelengths of laser light, the laser intensity values ​​obtained for different targets vary. Hyperspectral lidar echo intensity characterizes the reflection spectral characteristics of a target for laser light of different wavelengths.

[0005] In recent years, many researchers have studied point cloud classification for LiDAR based on spectral information, which mainly comes from unnormalized raw point cloud intensity information or normalized spectral information. Currently, commonly used methods for point cloud classification in hyperspectral LiDAR include Support Vector Machines (SVMs), Decision Trees, Logistic Regression, Naive Bayes, and Random Forests. These traditional machine learning methods primarily rely on manually designed features as input to the classifier, representing shallow classifier approaches. However, these shallow classifier methods have weak feature representation capabilities, making it difficult to bridge the semantic gap between the spectral data features of the low-level point cloud and the high-level logical information, resulting in often low classification accuracy. Furthermore, specific classification rules and parameter designs have poor adaptability to point clouds in complex scenes, and the model's generalization ability fails to achieve satisfactory results when handling complex scenarios. Summary of the Invention

[0006] In view of the above problems, this disclosure provides a classification method, training method, and apparatus for hyperspectral lidar point cloud data. By utilizing convolutional neural networks to classify hyperspectral lidar point cloud data, the problems of low classification accuracy and poor generalization ability in existing classification algorithms for hyperspectral lidar point cloud classification are solved.

[0007] According to embodiments of this disclosure, as a first aspect of this disclosure, a method for classifying hyperspectral lidar point cloud data is provided, comprising:

[0008] The spectral reflectance of each detection point is determined based on point cloud data from multiple detection points detected by hyperspectral lidar; a first image is determined based on the spectral reflectance of the detection points, wherein the first image includes information on the spectral reflectance of each detection point;

[0009] Traverse each detection point and determine the average spectral reflectance of the spectral neighborhood points of each detection point; determine the second image based on the average spectral reflectance, wherein the second image includes information about the average spectral reflectance of the spectral neighborhood points of each detection point.

[0010] The first image is input into the first convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the first feature.

[0011] The second image is input into the second convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the second feature.

[0012] The first and second features are input into the stitching layer of the trained hyperspectral lidar point cloud data classification model to obtain the stitched features; and

[0013] The stitched features are input into the classifier of the trained hyperspectral lidar point cloud data classification model, and the classification result is output.

[0014] According to embodiments of this disclosure, the above classification method further includes:

[0015] The spectral characteristics of the detection point are obtained based on its spectral reflectance, and these spectral characteristics are different from both the first and second characteristics.

[0016] The first and second features are input into the stitching layer of the trained hyperspectral lidar point cloud data classification model, which includes:

[0017] The spectral features, the first feature, and the second feature are input into the stitching layer of the trained hyperspectral lidar point cloud data classification model.

[0018] According to embodiments of this disclosure, the point cloud data includes the back reflection intensity of the hyperspectral laser at the detection point. The spectral reflectance of the detection point is obtained from the point cloud data of multiple detection points detected by the hyperspectral lidar, including:

[0019] The spectral reflectance of the detection point is obtained by measuring the intensity of the back reflection of the laser at the detection point.

[0020] According to embodiments of this disclosure, determining the first image based on the spectral reflectance of the detection point includes:

[0021] The spectral reflectance of the detection point is corrected to obtain the corrected spectral reflectance of the detection point.

[0022] The spectral reflectance of the corrected detection points is extracted to obtain a one-dimensional vector of the reflectance of the detection points;

[0023] Convert the one-dimensional vector of reflectivity at the detection point into a two-dimensional matrix;

[0024] Convert the two-dimensional matrix into the first image.

[0025] According to embodiments of this disclosure, determining the average spectral reflectance of the spectral neighborhood points of the detection point includes:

[0026] Search the spectral neighborhood of the detection point to determine the spectral neighborhood points of the detection point;

[0027] The mean reflectance of the spectral neighborhood points is determined based on the corrected spectral reflectance.

[0028] According to embodiments of this disclosure, as a second aspect of this disclosure, a training method for a hyperspectral lidar point cloud data classification model is also provided, comprising:

[0029] Training samples are obtained, including a first sample image and a second sample image. The first sample image is obtained by the spectral reflectance of the detection point detected by the hyperspectral lidar, and the second sample image is obtained by the average spectral reflectance of the spectral neighborhood points of the detection point.

[0030] The first sample image is input into the first convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the first prediction feature;

[0031] The second sample image is input into the second convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the second prediction feature.

[0032] The first and second predicted features are input into the stitching layer of the hyperspectral lidar point cloud data classification model and stitched together to obtain the predicted stitched features.

[0033] The predicted stitched features are input into the classifier of the hyperspectral lidar point cloud data classification model, and the predicted classification results are output.

[0034] Based on the predicted classification results, the parameters of the hyperspectral lidar point cloud data classification model are continuously adjusted through backpropagation to obtain a trained hyperspectral lidar point cloud data classification model.

[0035] According to embodiments of this disclosure, inputting the first predicted feature and the second predicted feature into the stitching layer of the hyperspectral lidar point cloud data classification model includes:

[0036] The spectral features, the first predicted feature, and the second predicted feature are input into the stitching layer of the hyperspectral lidar point cloud data classification model; wherein, the spectral features are obtained based on the spectral reflectance of the detection point.

[0037] According to embodiments of this disclosure, as a lower third aspect of this disclosure, a classification device for hyperspectral lidar point cloud data is also provided, comprising:

[0038] The determination module is used to determine the spectral reflectance of the detection points based on the point cloud data of multiple detection points detected by the hyperspectral lidar, determine the first image based on the spectral reflectance of the detection points, and traverse the detection points to determine the average spectral reflectance of the spectral neighborhood points of the detection points and determine the second image based on the average spectral reflectance.

[0039] The classification module is used to classify the first and second images using a trained hyperspectral lidar point cloud data classification model to obtain the classification results.

[0040] According to embodiments of this disclosure, as a fourth aspect of this disclosure, a training apparatus for a hyperspectral lidar point cloud data classification model is also provided, comprising:

[0041] The sample acquisition module is used to acquire training samples. The first training samples include a first sample image and a second sample image. The first sample image is obtained by measuring the spectral reflectance of the detection point detected by the hyperspectral lidar, and the second sample image is obtained by measuring the average spectral reflectance of the spectral neighborhood points of the detection point.

[0042] The training module is used to input the first sample image into the first convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the first predicted feature, input the second sample image into the second convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the second predicted feature, input the first and second predicted features into the stitching layer of the hyperspectral lidar point cloud data classification model to stitch them together to obtain the predicted stitched features, input the predicted stitched features into the classifier of the hyperspectral lidar point cloud data classification model, and continuously adjust the model parameters based on the predicted classification results. When the accuracy of the predicted classification results meets the preset threshold, the trained hyperspectral lidar point cloud data classification model is obtained.

[0043] According to embodiments of this disclosure, as a fifth aspect of this disclosure, an electronic device is also provided, comprising:

[0044] One or more processors;

[0045] Memory, used to store one or more instructions.

[0046] When one or more instructions are executed by one or more processors, they cause one or more processors to implement the method described above.

[0047] According to embodiments of this disclosure, the reflectance of each detection point and the average reflectance of its spectral neighborhood points are determined based on point cloud data from multiple detection points detected by a hyperspectral lidar. A first image is obtained based on the reflectance of each detection point, and a second image is determined based on the average reflectance of its spectral neighborhood points. Features are extracted from the first and second images using different branches of a trained hyperspectral lidar point cloud data classification model, and then the extracted features are concatenated to complete the classification of the hyperspectral lidar point cloud data. This classification method has the advantage of high classification accuracy. It solves the problems of low classification accuracy and poor generalization ability in existing classification algorithms during hyperspectral lidar point cloud classification. Attached Figure Description

[0048] Figure 1This diagram illustrates an application scenario of a hyperspectral lidar point cloud data classification method provided according to an embodiment of the present disclosure.

[0049] Figure 2 A flowchart illustrating a method for classifying hyperspectral lidar point cloud data according to an embodiment of the present disclosure is shown schematically.

[0050] Figure 3 This schematically illustrates the mapping sequence of spectral reflectance converted into a two-dimensional image according to an embodiment of the present disclosure;

[0051] Figure 4 A convolutional neural network structure diagram for classifying hyperspectral lidar point cloud data according to an embodiment of the present disclosure is illustrated schematically.

[0052] Figure 5 A convolutional neural network structure diagram of a non-plant classification method provided according to embodiments of the present disclosure is illustrated schematically.

[0053] Figure 6 This diagram schematically illustrates a convolutional neural network structure for a classification method in a plant classification scenario according to another embodiment of the present disclosure;

[0054] Figure 7 This schematic diagram illustrates a structural block diagram of a hyperspectral lidar point cloud data classification device according to an embodiment of the present disclosure;

[0055] Figure 8 This schematic diagram illustrates the structural block diagram of a training device for a hyperspectral lidar point cloud data classification model according to an embodiment of the present disclosure;

[0056] Figure 9 A block diagram of an electronic device provided according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0057] In recent years, deep learning-based methods have achieved a series of breakthroughs in fields such as image classification and object recognition, demonstrating stronger feature learning and representation capabilities compared to traditional machine learning methods. Deep learning-based methods can automatically extract point cloud feature information from data through an end-to-end learning mechanism, without requiring manually defined features. This effectively avoids the influence of subjective factors in feature extraction by traditional machine learning methods, thus adaptively extracting features from the data and classifying them, resulting in strong model generalization ability. LiDAR point cloud data is massive, and given sufficient labeled training samples, deep learning networks can often achieve good classification accuracy. Convolutional Neural Networks (CNNs), as classic deep learning models, perform exceptionally well in vision-based problems. Due to their hierarchical learning architecture, CNNs are a more efficient method for extracting non-linear features and have the advantage of handling large datasets.

[0058] Deep learning networks are data-driven methods, highly dependent on data learning, and lack human-like cognitive abilities, failing to fully utilize the rich logical information between different spectra in spectral data. Ground features, due to their inherent physical characteristics, possess spectral features that distinguish them from other ground features. These mainly include: ratio indices, absorption or reflectance properties, and characteristic spectral locations. Based on spectral features, prior knowledge and other information can be effectively used to strengthen the most separable features to assist classification. Based on the above analysis, combining data-driven deep learning with knowledge-driven spectral features can further improve the classification accuracy of LiDAR point clouds.

[0059] This disclosure primarily addresses the problems of low classification accuracy and poor generalization ability in existing classification algorithms for hyperspectral lidar point cloud classification. It provides a point cloud classification method that combines convolutional neural networks (CNNs) and spectral features. By combining CNNs and spectral features, it leverages the advantages of deep learning methods—their ability to learn low-level knowledge that is difficult to accurately represent and their high classification accuracy—while also incorporating the interpretability and reliability of spectral features. The CNN method extracts low-level features to complete the initial classification of the point cloud, while spectral features provide high-level classification knowledge to assist the deep learning-based classification, enhancing the distinguishability between ground feature categories. The entire process iterates and evolves continuously, further improving the accuracy of point cloud classification based on deep learning, and enhancing the interpretability and reliability of the classification.

[0060] Figure 1 The illustration shows an application scenario of a hyperspectral lidar point cloud data classification method provided according to an embodiment of the present disclosure.

[0061] like Figure 1As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0062] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0063] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0064] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0065] It should be noted that the hyperspectral lidar point cloud data classification method provided in this embodiment can generally be executed by server 105. Correspondingly, the hyperspectral lidar point cloud data classification device provided in this embodiment can generally be located in server 105. The hyperspectral lidar point cloud data classification method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the hyperspectral lidar point cloud data classification device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0066] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0067] The following will be based on Figure 1 The described scene, through Figures 2-7The classification method for hyperspectral lidar point cloud data according to the disclosed embodiments is described in detail.

[0068] Figure 2 The flowchart illustrates a method for classifying hyperspectral lidar point cloud data according to an embodiment of the present disclosure.

[0069] like Figure 2 As shown, a method for classifying hyperspectral lidar point cloud data in this embodiment may include operations S210 to S280.

[0070] In operation S210, the spectral reflectance of each detection point is determined based on the point cloud data of multiple detection points detected by the hyperspectral lidar.

[0071] In operation S220, a first image is determined based on the spectral reflectance of the detection points, wherein the first image includes the spectral reflectance of each detection point.

[0072] In operation S230, each detection point is traversed to determine the average spectral reflectance of the spectral neighborhood points of each detection point.

[0073] In operation S240, a second image is determined based on the average spectral reflectance, wherein the second image includes the average spectral reflectance information of the spectral neighborhood points of each detection point.

[0074] In operation S250, the first image is input into the first convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the first feature.

[0075] In operation S260, the second image is input into the second convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the second feature.

[0076] In operation S270, the first and second features are input into the stitching layer of the trained hyperspectral lidar point cloud data classification model to obtain stitched features.

[0077] In operation S280, the stitched features are input into the classifier of the trained hyperspectral lidar point cloud data classification model, and the classification result is output.

[0078] According to embodiments of this disclosure, the point cloud data in operation S210 includes the back reflection intensity of the probe point to the laser. Therefore, operation S210 includes obtaining the spectral reflectivity of the probe point based on the back reflection intensity of the probe point to the laser. Specifically, operation S210 includes: acquiring hyperspectral lidar point cloud data, processing the data, and calculating the spectral reflectivity of the point cloud data.

[0079] First, the hyperspectral lidar point cloud data acquisition method is as follows:

[0080] Hyperspectral lidar point cloud data is data obtained by hyperspectral lidar detecting the detection points of a target; each target contains multiple detection points. Hyperspectral lidar point cloud data acquisition is achieved using hyperspectral lidar point cloud data acquisition equipment. This equipment mainly includes a laser emitting unit, a laser receiving unit, and a scanning unit. The laser emitting unit uses a supercontinuum laser as the emission source, continuously emitting white light signals. The acousto-optic tunable filter in the laser emitting unit selects different wavelengths for emission through time-domain signal tuning. The laser receiving unit mainly consists of a receiving optical module, a photodetector, and a data acquisition module. Its function is to collect the laser echo signal through the receiving optical system, convert it into an electrical signal on the photodetector, and then record the full waveform echo signal of the hyperspectral lidar through the data acquisition module. The scanning unit mainly achieves high-density point cloud data acquisition through the control of a two-dimensional turntable.

[0081] The laser emitting unit of the hyperspectral lidar point cloud data acquisition equipment emits laser signals towards the ground objects. After atmospheric attenuation, the signals reach the target surface, interact with it, and are scattered before returning to the laser receiving unit after atmospheric scattering. A photodetector converts the backscattered echo signals from all detection points on the target into electrical signals. These signals are then amplified by a signal amplification circuit and processed by the data acquisition module to obtain the backscatter intensity of each detection point.

[0082] Secondly, the calculation of hyperspectral lidar point cloud data includes distance calculation, three-dimensional coordinate calculation, and incident angle calculation, and the specific methods are as follows:

[0083] (1) Distance calculation:

[0084] The ranging principle of hyperspectral lidar is similar to that of ordinary single-wavelength pulse lidar, primarily based on the time-of-flight ranging principle. It calculates the distance from the lidar to the target by measuring the time difference between the emitted laser signal and the backscattered echo signal. The formula for calculating the distance L between the lidar and the target is:

[0085]

[0086] Where Δt is the round-trip time of the laser pulse, and c is the speed of light.

[0087] (2) Three-dimensional coordinate calculation:

[0088] The laser emitting unit of the hyperspectral lidar point cloud data acquisition equipment needs to rotate in both azimuth and elevation directions with the help of the scanning unit to scan the target. While the hyperspectral lidar point cloud data acquisition equipment is working, the azimuth and elevation angles of the turntable are recorded each time for subsequent point cloud calculations.

[0089] Hyperspectral lidar records the time it takes for a laser to travel from the laser to the target and back. This time value needs to be converted into a distance value and then combined with the elevation and azimuth angles of the two-dimensional turntable at the moment of laser emission to calculate the three-dimensional coordinates of the laser point cloud (i.e., the detection point on the target) in the Cartesian coordinate system.

[0090] The origin is the transmission center of the hyperspectral lidar point cloud data acquisition equipment. The coordinates of any detection point are as follows: Using the turntable rotation axis as the coordinate axis, the pitch rotation axis as the X-axis (positive direction to the right), and the yaw rotation axis as the Y-axis (positive direction upwards), a right-handed coordinate system is established, with the direction pointing towards the target as the positive Z-axis. The formula for calculating the spatial coordinates of the detection point is as follows:

[0091]

[0092] Where α is the angle between the projection line of the laser pointer on the plane of the two-dimensional rotating stage and the Y-axis of the two-dimensional rotating stage, with clockwise rotation being positive, and β is the angle between the laser pointer and the projection of the laser pointer on the plane of the two-dimensional rotating stage.

[0093] (3) Calculation of the angle of incidence:

[0094] Let the normal vector of the detection point (X,Y,Z) be n(n1,n2,n3), then the incident laser vector is OS = (x-x0,y-y0,z-z0). The normal vector n is estimated from the best-fit plane of a small uniform region surrounding the scanning point. This is obtained by using the normal vector to establish a neighborhood set fitting plane of the point cloud using the K-Nearest Neighbor (KNN) algorithm, and then calculating the normal vector of the fitting plane. The incident angle of each detection point is calculated using the following formula:

[0095]

[0096] Furthermore, the specific method for calculating the spectral reflectance of point cloud data is as follows:

[0097] In addition to acquiring the three-dimensional point cloud geometric information of the target surface, the hyperspectral lidar also measures the power of the laser signal returned from the target surface, which is recorded as the hyperspectral lidar backscattering intensity. The hyperspectral lidar backscattering intensity is a continuous function of wavelength. To obtain the true reflectivity of the detection point, it is necessary to use a standard diffuse reflector with known reflectivity to obtain the received intensity of each emitted laser wavelength at the same distance. A standard diffuse reflector with fixed and known reflectivity is placed at the same distance and scanned at the same spatial position. The formula for calculating spectral reflectivity is as shown in equation (5):

[0098]

[0099] Among them, I target and I reference R represents the back reflection intensity of the hyperspectral lidar at the detection point of the target and the reference standard diffuse reflector, respectively. reference and R target These are the spectral reflectances of the detection point of the target and the reference standard diffuse reflector, respectively.

[0100] According to an embodiment of this disclosure, in operation S220, determining the first image based on the spectral reflectance of the detection point includes operations S221-S224.

[0101] In operation S221, the spectral reflectance of the detection point is corrected to obtain the corrected spectral reflectance of the detection point. The correction of the spectral reflectance of the detection point includes distance effect radiation correction and incident angle effect radiation correction.

[0102] The specific methods for distance effect radiation correction are as follows:

[0103] According to the radar equations, for the same diffuse reflection target, when the incident angle is constant, the backscattering intensity of the lidar is related to 1 / L 2 Linear correlation, where L is the transmission distance. To reduce the impact of distance on point cloud classification accuracy, all point clouds need to undergo distance-effect radiative correction to obtain a distance-independent backscattering intensity. Define the standard distance L. s The backscattering intensity of all detection points is corrected to the standard distance. The range-effect radiation correction model is expressed as:

[0104]

[0105] The method for incident angle effect radiation correction is as follows:

[0106] According to radar equations, for the same diffuse reflective target, at a constant distance, the backscattering intensity of lidar is linearly related to the cosine of the incident angle. In practical applications, perfect diffuse reflection is almost nonexistent; specular reflection and diffuse reflection are intertwined, and the degree of both varies depending on the terrain.

[0107] When the incident angle is greater than a certain threshold, although specular reflection exists, the lidar receiver cannot receive it; only diffuse reflection reaches the receiver. When the incident angle is less than a certain threshold, the receiver can simultaneously receive part of the specular reflection and diffuse reflection. An incident angle threshold θ0 is set, and a piecewise Lambertian-Beckmann model is used to correct for the incident angle effect of backscattering intensity. To obtain a backscattering intensity unaffected by the incident angle, the intensity values ​​at all points are corrected to the perpendicular incident angle. The incident angle effect radiation correction model is expressed as:

[0108]

[0109] Where I(θ,λ) is the backscattering intensity of the laser with wavelength λ at the incident angle θ; f0 is the backscattering intensity in the normal direction; k d is the coefficient of the diffuse reflection component, between 0 and 1, and is related to the smoothness of the object's surface. s It is the coefficient of the specular reflection component, and k d +k s =1

[0110] In operation S222, the spectral reflectance of the corrected detection point is extracted to obtain a one-dimensional vector of the detection point reflectance. The one-dimensional vector R of the spectral reflectance of the detection point of the target is then obtained. x =(R x1 ,R x2 ,…,R xn This one-dimensional vector includes the spectral reflectance of all detection points.

[0111] In operation S223, the one-dimensional vector of spectral reflectance of the detection point is converted into a two-dimensional matrix. Following the mapping order shown in Figure 2, the spectral vector and mean neighborhood feature of each point are converted into an a×b two-dimensional matrix. If a×b>N, zeros are padded to fill any missing positions in the matrix.

[0112] In operation S224, the two-dimensional matrix is ​​converted into the first image. This two-dimensional matrix is ​​then converted into an M×N dimensional uint8 image to obtain the first image.

[0113] Operations S221-S224 iterate through all the detection points in the point cloud, generating a sample set of grayscale images. Each pair of grayscale images in the sample set (spectral vector transformation map and mean neighborhood feature transformation map) represents a sample for classification. This sample contains rich texture information that describes the spectral characteristics of the detected target.

[0114] According to an embodiment of this disclosure, in operation S230, determining the second image based on the average spectral reflectance includes: operations S231-S232.

[0115] Operation S231 involves searching the spectral neighborhood of the detection point to determine its spectral neighbor points. Specifically, operation S231 includes:

[0116] (1) Spectral neighborhood selection

[0117] Utilizing the principle of spectral similarity, a neighborhood selection method based on spectral angle or Euclidean distance is constructed to calculate the spectral similarity between a detection point and its neighbors. The spectral angle refers to the angle between two spectral curves, which are treated as vectors in a two-dimensional space. The smaller the angle, the greater the similarity. The spectral reflectance of each detection point in the hyperspectral lidar point cloud is projected as a vector onto an N-dimensional space, where N is the number of real bands. In this N-dimensional space, each spectral curve is considered as a vector with direction and length, and the angle between the spectra is the spectral angle. Assume the spectral reflectances of two hyperspectral lidar detection points are R... x and R y R x =(R x1 ,R x2 ,…,R xn ), R y =(R y1 ,R y2 ,…,R yn ), R xn and R yn These represent the reflectance at different wavelengths of the two detection points. The spectral angles of these two detection points are expressed as:

[0118]

[0119] Among them, the smaller the spectral angle, the closer the cosine value is to 1, and the more similar the two spectral curves are. The spectral angle compares the similarity of the spectra in terms of the angle between the vectors.

[0120] Euclidean distance is the distance between two spectral curves. Let R be the spectral reflectance data of two hyperspectral lidar systems with N bands. x and R y The Euclidean distance between these two detection points is expressed as:

[0121]

[0122] Euclidean distance primarily describes the difference in reflectance between spectral vectors. A smaller Euclidean distance indicates a smaller difference in reflectance between two detection points. In contrast, the distance measure described by spectral angle focuses more on the difference in vector shape. Therefore, if the spectral differences of the detection targets in point cloud classification are large, spectral angle distance can be used; if the detection targets are different species with the same physical attribute (e.g., different types of plants), Euclidean distance can be used.

[0123] Based on the calculation of the spectral angular distance or Euclidean distance between neighboring points and the probe point, each probe point obtains its K nearest neighbors in the point cloud. K spectrally similar points within the neighborhood are selected to determine a suitable K value, assuming these probe points originate from the same target and have similar spectral characteristics. Based on the selected K spectra, neighborhood features are further calculated, allowing for the extraction of more precise spectral features of the target and improving classification accuracy. The choice of K value significantly impacts neighborhood feature extraction. A smaller K value makes neighborhood features more sensitive to the neighboring points of the probe point. Conversely, a larger K value may lead to the selection of dissimilar feature points. Generally, a smaller K value is initially chosen, and a suitable K value is selected through cross-validation to determine the spectral neighbors of the probe point.

[0124] Operation S232: Determine the mean reflectance of the spectral neighborhood points based on the corrected spectral reflectance.

[0125] The spectral neighborhood points of any detection point in the target are selected from all other detection points in the target except for that detection point. Therefore, the spectral reflectance correction of the detection point in operation S221 yields the corrected spectral reflectance of the spectral neighborhood points. Based on the assumption that points of the same class have similar spectra, the mean feature of the spectral neighborhood points is used as the neighborhood feature. Assume the spectral reflectance of detection point x is R. x =(R x1 ,R x2 ,…,R xn Mean of reflectance of the spectral neighborhood of detector point x x =(Mean x1 Mean x2 ,…,Mean xn ), where Mean xn The formula is expressed as follows:

[0126]

[0127] Among them, R kn This represents the reflectance of the nth band of the spectral neighborhood of the detector point x.

[0128] In operation S240, the process of determining the second image based on the average spectral reflectance is similar to that of determining the first image. Specifically, firstly, a one-dimensional vector of the average reflectance is obtained from the average spectral reflectance; secondly, this one-dimensional vector is converted into a two-dimensional matrix; finally, the second image is obtained from the two-dimensional matrix converted from the one-dimensional vector. Since the reflectance of each spectrum is in the range of 0-1, the average spectral reflectance is also in the range of 0-1. Each average spectral reflectance is multiplied by 255 for numerical conversion. Then, according to... Figure 3 The mapping order shown converts the spectral vector and mean neighborhood features of each point into an a×b 2D matrix. If a×b > N, zeros are padded to the last insufficient positions in the matrix. Finally, this matrix is ​​converted into an M×N dimensional uint8 image to obtain the second image.

[0129] After obtaining the first and second images, a hyperspectral lidar point cloud data classification model was constructed and trained. The trained model consisted of a cascaded first convolutional neural network branch, a second convolutional neural network branch, a concatenation layer, and a classifier. Both the first and second convolutional neural network branches included input layers, convolutional layers, pooling layers, and fully connected layers. Multiple convolutional and fully connected layers were embedded within the model to extract features layer by layer. The extracted features became increasingly abstract and generalized as the model progressed through the layers, moving from specific and detailed features at a shallow level to more abstract and generalized features at a deeper level.

[0130] The first convolutional neural network branch consists of a first input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, and a first fully connected layer, all cascaded together. The first layer is the first input layer, which takes the first image as input. In the first convolutional layer, num_output = 64, and the first image is filtered by 64 3×3 convolutional kernels with a stride of 1, resulting in 64 (a-2)×(b-2) dimensional feature maps. In the subsequent first pooling layer, max pooling is used for downsampling, generating 64 feature maps. The first layer contains 128 feature maps; the second convolutional layer, num_output = 128, uses 128 3×3 kernels for further convolution, with a stride of 1, and generates 128 features. The feature map is 128-dimensional; the second pooling layer also uses max pooling for downsampling, resulting in 128 features. The feature map is then processed; next, a first fully connected layer with num_output = 300 is used to generate 300 neurons, which are used as the first extracted feature F0.

[0131] The second convolutional neural network branch consists of a second input layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, and a second fully connected layer, all cascaded together. The first layer is the second input layer, which takes the second image as input. In the third convolutional layer, num_output = 64, and the first image is filtered by 64 3×3 convolutional kernels with a stride of 1, resulting in 64 (a-2)×(b-2) dimensional feature maps. In the subsequent third pooling layer, max pooling is used for downsampling, generating 64 feature maps. The feature map is 128-dimensional; the fourth convolutional layer, num_output=128, uses 128 3×3 convolutional kernels to continue convolution processing, with a stride of 1, and generates 128 features. The feature map is 128-dimensional; the fourth pooling layer also uses max pooling for downsampling, resulting in 128 features. The feature map is then processed; next, a second fully connected layer with num_output = 300 is used to generate 300 neurons, which serve as the extracted second feature F. mean .

[0132] The concatenation layer (i.e., the concat layer) extracts the first feature F0 from the first convolutional neural network branch and the second feature F from the second convolutional neural network branch. mean The data is concatenated into 600 neurons, and the resulting deep feature is represented as: F = [F0, F...]. mean ].

[0133] The classifier consists of a ReLU non-linear activation function, a third fully connected layer, and a Softmax layer. Concatenated features are input to the ReLU non-linear activation function of the classifier, the output is input to the third fully connected layer, passes through the ReLU non-linear activation function again, and finally passes through the Softmax layer to obtain the probability results for the specified number of classes (num_classes) and output them.

[0134] Figure 4 The diagram illustrates a convolutional neural network structure for classifying hyperspectral lidar point cloud data according to an embodiment of the present disclosure.

[0135] like Figure 4 As shown, hyperspectral lidar point cloud data is obtained based on the detection of ground features, and the detection points can be distributed on different ground features. By utilizing the hyperspectral lidar point cloud data classification model provided in the embodiments of this disclosure, the classification of different ground features can be achieved. The specific process is as follows:

[0136] Step A: Determine the spectral reflectance of each detection point based on the point cloud data detected by the hyperspectral lidar. The detection points are distributed on the target. For example, if the target is a plant, the detection points can be distributed on the plant leaves.

[0137] Step B: Determine the one-dimensional vector of reflectance of each detection point based on its spectral reflectance;

[0138] Step C: Determine the two-dimensional matrix based on the one-dimensional vector of reflectivity at the detection point;

[0139] Step D: Determine the first image based on the two-dimensional matrix;

[0140] Step E: Determine the mean spectral reflectance of the spectral neighborhood points of each detection point based on the point cloud data detected by the hyperspectral lidar;

[0141] Step F: Determine a one-dimensional vector of mean reflectance based on the mean spectral reflectance of the spectral neighborhood points of each detection point;

[0142] Step G: Determine the second image based on the one-dimensional vector of the mean reflectance;

[0143] Step 1: Input the first image into the first convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the first feature F0; input the second image into the second convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the second feature F0. mean ;

[0144] Step J: Design a concat layer to extract depth features F0 and F1 from the original spectral information and the mean neighborhood spectral information, respectively. mean The concatenated depth features are represented as: F = [F0, F...] mean ]

[0145] Step K: Pass through the ReLU non-linear activation function; pass through the second fully connected layer, pass through the ReLU non-linear activation function again; finally pass through the Softmax layer to obtain the probability results of the set classes (num_classes) and output them.

[0146] According to embodiments of this disclosure, the above classification method further includes: obtaining the spectral characteristics F of the detection point based on the spectral reflectance of the detection point. feature The spectral features, the first feature, and the second feature are input into the stitching layer of the trained hyperspectral lidar point cloud data classification model. The stitched depth features are represented as: F = [F0, F...] mean ,F featureThe spectral features are distinct from both the first and second features. Combining convolutional neural networks (CNNs) and spectral features leverages the advantages of deep learning—its ability to learn complex, low-level knowledge and its high classification accuracy—while also incorporating the interpretability and reliability of spectral features. CNNs extract low-level features for initial point cloud classification, while spectral features provide high-level classification knowledge to aid deep learning-based classification and enhance the distinguishability between land cover categories. This iterative process not only further improves point cloud classification accuracy based on deep learning but also enhances interpretability and reliability, achieving higher accuracy compared to methods that solely utilize CNNs. Specifically, the method for obtaining the spectral features of a detection point based on its spectral reflectance is as follows:

[0147] Spectral feature extraction is based on the recombination and optimization of various spectral bands. In the extracted spectral feature space, the new spectral vector reflects a certain characteristic of the target scanned by the hyperspectral lidar, or a spectral parameter that distinguishes it from other targets. Targets with significantly different spectral characteristics are relatively easy to distinguish during classification. However, more refined ground feature classification requires more spectral features.

[0148] This disclosure mainly provides spectral feature extraction schemes for two classification scenarios: classification of non-plants in common scenarios and fine classification of different plants.

[0149] For the classification of non-plant targets in common scenarios, each detection point in a hyperspectral lidar point cloud can be fitted with a spectral curve. Spectral features selectable for hyperspectral lidar point cloud classification include spectral absorption index, derivative spectral features, and spectral integrals. On the spectral curve of ground features, the typical feature characterizing the differences between ground features is the diagnostic spectral absorption peak caused by the different material compositions of the ground features. By extracting parameters such as wavelength position, absorption depth, absorption width, and absorption symmetry of the absorption bands of the absorption peaks on the curve, the purpose of identifying different ground features can be achieved. Spectral absorption feature parameters include absorption position, absorption depth, absorption width, and absorption symmetry. Absorption position refers to the wavelength corresponding to the minimum spectral value within the absorption valley; absorption depth refers to the distance from the point with the lowest reflectance within the absorption valley to the envelope of the spectral curve; absorption width refers to the wavelength interval between the start and end points of the absorption valley; absorption symmetry refers to the commonly used logarithm of the ratio of the area to the area to the left of the region, with the vertical line passing through the absorption position as the boundary. Derivative spectral characteristics mainly refer to calculating the first and second derivatives of the spectral curve, thereby further determining the locations of the curve's inflection points, maximum reflectance, and minimum reflectance. The spectral integral refers to the area covered by the spectral curve within a certain wavelength range.

[0150] Figure 5 A convolutional neural network structure diagram of a non-plant classification method provided according to embodiments of the present disclosure is illustrated schematically.

[0151] like Figure 5 As shown, for non-plant classification, the specific steps include:

[0152] Step O: Determine the spectral reflectance of each detection point based on the point cloud data detected by the hyperspectral lidar. The detection points are distributed on the detection target, which is not a plant. The spectral reflectance includes parameters such as wavelength position, absorption depth, absorption width, and absorption symmetry.

[0153] Step P: Determine the spectral characteristics based on the spectral reflectance of each detection point;

[0154] Step Q: Determine the one-dimensional vector of reflectance of each detection point based on its spectral reflectance;

[0155] Step R: Determine the two-dimensional matrix based on the one-dimensional vector of reflectivity at the detection point;

[0156] Step S: Determine the first image based on the two-dimensional matrix;

[0157] Step T: Determine the mean spectral reflectance of the spectral neighborhood points of each detection point based on the point cloud data detected by the hyperspectral lidar;

[0158] Step U: Determine a one-dimensional vector of mean reflectance based on the mean spectral reflectance of the spectral neighborhood points of each detection point;

[0159] Step V: Determine the second image based on the one-dimensional vector of the mean reflectance;

[0160] Step W: Input the first image into the first convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the first feature F0; input the second image into the second convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract the second feature F0. mean ;

[0161] Step X: Design a concat layer to extract depth features F0 and F1 from the original spectral information and the mean neighborhood spectral information, respectively. mean and spectral characteristics F feature The concatenated depth features are represented as: F = [F0, F...] mean ,F feature ].

[0162] Step Y: The process involves passing through a ReLU non-linear activation function; then through a second fully connected layer, followed by another ReLU non-linear activation function; finally, through a Softmax layer, the probability results for the specified number of classes (num_classes) are obtained and output.

[0163] For fine-grained classification of different plants, the spectral curves of different plants exhibit roughly similar waveforms. However, plants of different categories have different biological parameters due to differences in growth conditions, plant composition, leaf surface structure, etc. Plants of the same type under similar growth conditions and at similar growth stages typically have similar biological parameters. Therefore, biological parameters can be used for point cloud classification using hyperspectral lidar. Since the inversion process of vegetation biological parameters is usually quite complex, we select spectral indices sensitive to vegetation biological parameters as spectral features characterizing different plant characteristics. Considering the relative differences in chlorophyll, water, and nitrogen content among different plants, and combining this with the spectral range of hyperspectral lidar, this disclosure selects vegetation indices sensitive to changes in biological parameters as spectral features for auxiliary classification to improve recognition accuracy. Specific vegetation indices are shown in Table 1.

[0164] Table 1. Vegetation spectral indices used for extracting spectral features.

[0165]

[0166] Figure 6 The diagram schematically illustrates a convolutional neural network structure for a classification method in a plant classification scenario according to another embodiment of this disclosure.

[0167] like Figure 6 As shown, the process of this embodiment is similar to... Figure 5 The process is similar in the plant classification scenario shown, the difference being that the detection target is plants, and the spectral characteristics of the plants include the vegetation indices in Table 1. According to Figure 6 The classification method shown can be used to classify plants.

[0168] According to embodiments of this disclosure, a training method for a hyperspectral lidar point cloud data classification model is also provided, comprising:

[0169] Training samples are acquired, including a first sample image and a second sample image. The first sample image is obtained by measuring the spectral reflectance of the detection points using a hyperspectral lidar, and the second sample image is obtained by measuring the average spectral reflectance of the spectral neighborhood points of the detection points. For example, taking root-type land cover as an example, root-type land cover object classes are defined, and object class annotations are performed on the point cloud data through visual interpretation and manual delineation. This results in a large and high-quality set of training, validation, and test data to meet the large data requirements of convolutional neural networks, thereby enhancing the model's generalization ability and classification performance.

[0170] The first sample image is input into the first convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the first prediction feature.

[0171] The second sample image is input into the second convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the second prediction feature.

[0172] The first and second predicted features are input into the stitching layer of the hyperspectral lidar point cloud data classification model to obtain the predicted stitched features; the predicted stitched features are input into the classifier of the hyperspectral lidar point cloud data classification model to output the predicted classification result.

[0173] Based on the predicted classification results, the parameters of the hyperspectral lidar point cloud data classification model are continuously adjusted through backpropagation, and finally a trained hyperspectral lidar point cloud data classification model is obtained.

[0174] According to an embodiment of this disclosure, inputting the first predicted feature and the second predicted feature into the stitching layer of the hyperspectral lidar point cloud data classification model includes: inputting spectral features, the first predicted feature, and the second predicted feature into the stitching layer of the hyperspectral lidar point cloud data classification model; wherein, the spectral features are obtained based on the spectral reflectance of the detection point.

[0175] According to embodiments of this disclosure, after obtaining a trained hyperspectral lidar point cloud data classification model, the method further includes classifying and predicting the test set in the hyperspectral lidar point cloud data. The point cloud classification accuracy for each target is evaluated using a confusion matrix and Kappa coefficients.

[0176] Figure 7 A schematic block diagram of a hyperspectral lidar point cloud data classification device according to an embodiment of the present disclosure is shown.

[0177] like Figure 7 As shown, the hyperspectral lidar point cloud data classification device 700 of this embodiment includes a determination module 710 and a classification module 720.

[0178] The determination module 710 is used to determine the spectral reflectance of multiple detection points based on point cloud data from the hyperspectral lidar, determine the first image based on the spectral reflectance of the detection points, and traverse the detection points to determine the average spectral reflectance of the spectral neighborhood points of each detection point, and determine the second image based on the average spectral reflectance. The classification module 720 is used to classify the first image and the second image using a trained hyperspectral lidar point cloud data classification model to obtain the classification result.

[0179] According to embodiments of this disclosure, the determining module 710 is further configured to obtain the spectral characteristics of the detection point based on the spectral reflectance information of the detection point.

[0180] Figure 8 The diagram illustrates the structure of a training apparatus for a hyperspectral lidar point cloud data classification model according to an embodiment of the present disclosure.

[0181] like Figure 8 As shown, the training device 800 for the hyperspectral lidar point cloud data classification model in this embodiment includes a sample acquisition module 810 and a training module 820.

[0182] The sample acquisition module 810 is used to acquire training samples. The first training sample includes a first sample image and a second sample image. The first sample image is obtained by the spectral reflectance of the detection point detected by the hyperspectral lidar, and the second sample image is obtained by the average spectral reflectance of the spectral neighborhood points of the detection point.

[0183] The training module 820 is used to input the first sample image into the first convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the first predicted feature, input the second sample image into the second convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract the second predicted feature, input the first and second predicted features into the stitching layer of the hyperspectral lidar point cloud data classification model to stitch together the predicted stitched features, input the predicted stitched features into the classifier of the hyperspectral lidar point cloud data classification model, continuously adjust the model parameters based on the predicted classification results, and obtain the trained hyperspectral lidar point cloud data classification model when the accuracy of the predicted classification results meets the preset threshold.

[0184] According to embodiments of this disclosure, any and multiple modules among the determining module 710, classification module 720, sample acquisition module 810, and training module 820 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in one module.

[0185] According to embodiments of this disclosure, at least one of the determining module 710, the classification module 720, the sample acquisition module 810, and the training module 820 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the determining module 710, the classification module 720, the sample acquisition module 810, and the training module 820 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0186] Figure 9 A block diagram of an electronic device provided according to an embodiment of the present disclosure is shown schematically.

[0187] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0188] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0189] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0190] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0191] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0192] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the hyperspectral lidar point cloud data classification method provided in embodiments of this disclosure.

[0193] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0194] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0195] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0196] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0198] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0199] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for classifying hyperspectral lidar point cloud data, comprising: determining spectral reflectance of each of a plurality of detected points according to point cloud data of the plurality of detected points detected by a hyperspectral lidar; determining a first image according to the spectral reflectance of the detected points, wherein the first image comprises information of the spectral reflectance of each of the detected points; determining a mean value of spectral reflectance of spectral neighborhood points of each of the detected points by traversing each of the detected points; determining a second image according to the mean value of spectral reflectance, wherein the second image comprises information of the mean value of spectral reflectance of the spectral neighborhood points of each of the detected points; inputting the first image into a first convolutional neural network branch of a trained hyperspectral lidar point cloud data classification model to extract a first feature; inputting the second image into a second convolutional neural network branch of the trained hyperspectral lidar point cloud data classification model to extract a second feature; inputting the first feature and the second feature into a concatenation layer of the trained hyperspectral lidar point cloud data classification model to concatenate a concatenated feature; and inputting the concatenated feature into a classifier of the trained hyperspectral lidar point cloud data classification model to output a classification result.

2. The classification method of claim 1, wherein, Further comprising: obtaining a spectral feature of the detected points according to the spectral reflectance of the detected points, wherein the spectral feature is different from the first feature and the second feature; inputting the first feature and the second feature into the concatenation layer of the trained hyperspectral lidar point cloud data classification model comprises: inputting the spectral feature, the first feature and the second feature into the concatenation layer of the trained hyperspectral lidar point cloud data classification model.

3. The classification method of claim 1, wherein, The point cloud data comprises hyperspectral lidar backscattering intensity of the detected points, and the spectral reflectance of the detected points is obtained according to the point cloud data of the plurality of detected points detected by the hyperspectral lidar, comprising: obtaining the spectral reflectance of the detected points according to the hyperspectral lidar backscattering intensity of the detected points.

4. The classification method of claim 1, wherein, The determination of the first image according to the spectral reflectance of the detected points comprises: correcting the spectral reflectance of the detected points to obtain corrected spectral reflectance of the detected points; extracting the corrected spectral reflectance of the detected points to obtain a one-dimensional vector of the spectral reflectance of the detected points; converting the one-dimensional vector of the spectral reflectance of the detected points into a two-dimensional matrix; converting the two-dimensional matrix into the first image.

5. The classification method of claim 4, wherein, The determination of the mean value of spectral reflectance of the spectral neighborhood points of the detected points comprises: searching for spectral neighborhood points of the detected points to determine the spectral neighborhood points of the detected points; determining a mean value of reflectance of the spectral neighborhood points according to the corrected spectral reflectance of the spectral neighborhood points. 6.A method for training a hyperspectral lidar point cloud data classification model, comprising: obtaining training samples, wherein the training samples comprise a first sample image and a second sample image, the first sample image is obtained by spectral reflectance of detected points detected by a hyperspectral lidar, and the second sample image is obtained by a mean value of spectral reflectance of spectral neighborhood points of the detected points; inputting the first sample image into a first convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract first predicted features; inputting the second sample image into a second convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract second predicted features; inputting the first predicted features and the second predicted features into a concatenation layer of the hyperspectral lidar point cloud data classification model to obtain predicted concatenated features by concatenation; inputting the predicted concatenated features into a classifier of the hyperspectral lidar point cloud data classification model to output a predicted classification result; continuously adjusting parameters of the hyperspectral lidar point cloud data classification model through back propagation according to the predicted classification result to obtain a trained hyperspectral lidar point cloud data classification model.

7. The training method of claim 6, wherein, inputting the first predicted features and the second predicted features into a concatenation layer of the hyperspectral lidar point cloud data classification model includes: inputting spectral features, the first predicted features and the second predicted features into the concatenation layer of the hyperspectral lidar point cloud data classification model, wherein the spectral features are obtained according to spectral reflectivity of the detection points.

8. A hyperspectral lidar point cloud data classification apparatus, comprising: a determination module configured to determine spectral reflectivity of detection points according to point cloud data of the detection points detected by a hyperspectral lidar, determine a first image according to the spectral reflectivity of the detection points, and traverse the detection points to determine a spectral reflectivity mean value of spectral neighborhood points of the detection points, and determine a second image according to the spectral reflectivity mean value; a classification module configured to classify the first image and the second image by using a trained hyperspectral lidar point cloud data classification model to obtain a classification result.

9. A hyperspectral lidar point cloud data classification model training apparatus, comprising: a sample acquisition module configured to acquire training samples, the training samples including a first sample image and a second sample image, wherein the first sample image is obtained by spectral reflectivity of detection points detected by a hyperspectral lidar, and the second sample image is obtained by a spectral reflectivity mean value of spectral neighborhood points of the detection points; and a training module configured to input the first sample image into a first convolutional neural network branch of a hyperspectral lidar point cloud data classification model to extract first predicted features, input the second sample image into a second convolutional neural network branch of the hyperspectral lidar point cloud data classification model to extract second predicted features, input the first predicted features and the second predicted features into a concatenation layer of the hyperspectral lidar point cloud data classification model to obtain predicted concatenated features by concatenation, input the predicted concatenated features into a classifier of the hyperspectral lidar point cloud data classification model, continuously adjust model parameters based on a predicted classification result, and obtain a trained hyperspectral lidar point cloud data classification model in a case where accuracy of the predicted classification result meets a preset threshold.

10. An electronic device, comprising: one or more processors; a memory configured to store one or more instructions, wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

Images