Hyperspectral and point cloud data integrated tree identification system based on attention mechanism convolutional neural network

Abstract

The invention provides a hyperspectral and point cloud data integrated tree identification system based on an attention mechanism convolutional neural network, relates to the technical field of tree identification, and realizes efficient tree classification through a series of fine processing and deep learning methods. The system firstly collects hyperspectral data and point cloud data, and corrects and de-noising the collected data. And then, performing feature extraction on the hyperspectral data and the point cloud data by using a convolutional neural network and a graph convolutional network, and introducing an attention mechanism to analyze features and obtain an attention weight matrix, thereby realizing effective fusion of the two data features. And finally, inputting the fusion features into a classifier for probability calculation of tree categories, and outputting a final category vector. The system improves the accuracy and efficiency of tree identification, and is expected to play an important role in the fields of ecological monitoring, resource management and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of tree recognition, and in particular to a tree recognition system based on the integration of hyperspectral and point cloud data of an attention mechanism convolutional neural network. Background Art

[0002] Tree identification is of great significance in environmental protection, forest management, and ecological monitoring. Traditional tree identification methods mainly rely on optical images or lidar data, which usually require manual annotation, are complex to process, and are time-consuming. With the development of deep learning technology, convolutional neural networks have made significant progress in image classification, target detection, and other fields. However, tree identification from a single data source still faces many challenges in complex natural environments, such as lighting changes, occlusion, and tree species diversity.

[0003] The prior art has the following deficiencies:

[0004] Traditional optical image tree recognition methods mainly rely on optical images for tree recognition. Such methods work well in good lighting conditions and simple backgrounds, but they perform poorly in complex environments and are greatly affected by factors such as weather and lighting, and have poor robustness. The lidar tree recognition method lacks spectral information through a single point cloud data, making it difficult to distinguish between tree species with similar spectra but different morphologies. In recent years, deep learning methods have been applied in tree recognition, such as CNN-based image classification and PointNet-based point cloud classification. However, single-modality data is difficult to fully characterize the complex characteristics of trees, and recognition accuracy and robustness are limited. Summary of the invention

[0005] The purpose of the present invention is to provide a tree recognition system based on the integration of hyperspectral and point cloud data of attention mechanism convolutional neural network to solve the problems raised in the above background technology.

[0006] The tree recognition system based on the hyperspectral and point cloud data integration with attention mechanism convolutional neural network includes:

[0007] The data acquisition module is used to collect the hyperspectral data and point cloud data of trees. The hyperspectral data records the spectral value of each pixel, and the point cloud data records the three-dimensional coordinates and intensity value of each point.

[0008] A data preprocessing module is used to preprocess the collected spectral data and point cloud data, including correction processing of spectral data and denoising and registration processing of point cloud data;

[0009] Multimodal feature extraction module, used to extract features from hyperspectral data and point cloud data based on convolutional neural networks and graph convolutional networks, and obtain hyperspectral data features and point cloud data features;

[0010] The attention mechanism fusion module is used to analyze hyperspectral data and point cloud data based on the attention mechanism, obtain the attention weight matrix, calculate the fusion features of hyperspectral data features and point cloud data respectively, and then splice them to obtain the final fusion features;

[0011] The tree recognition module is used to input the fused features into the classifier, calculate the probability of identifying the tree as each category, and output the category vector.

[0012] Furthermore, the specific logic for collecting hyperspectral data and point cloud data is as follows:

[0013] Hyperspectral data of trees are collected using hyperspectral equipment, and the spectral information of each pixel is calibrated as S(x, y, λ), which represents the reflectivity of the wavelength λ of the pixel at the (x, y) position. LiDAR equipment is used to collect point cloud data at high density. The point cloud data records the three-dimensional coordinates and intensity value of each point. The calibration point cloud data set is q = {(x i ,y i ,z i ,I i )|i=1,2,…,N}, N is the total number of points in the point cloud.

[0014] Furthermore, the specific logic for preprocessing hyperspectral data and point cloud data is as follows:

[0015] Each spectral value in the spectral data is firstly calibrated by radiation, and then the hyperspectral data is aligned to the actual geographic coordinate system using ground control points through geometric correction. Back projection is used for geometric transformation, and finally wavelet transform is used to denoise the hyperspectral data to obtain preprocessed hyperspectral data. The conditional filtering method is used to remove outliers, and an outlier threshold is set. If the point q i If the distance to the mean of its neighborhood is greater than the outlier threshold, the point is marked as an outlier, all points are traversed, and the unmarked points are retained to form a new point cloud data set. The transformation matrix is ​​calculated through the ICP algorithm, and the new point cloud data set is registered to obtain the registered point cloud data.

[0016] Furthermore, the specific logic for obtaining the hyperspectral data features and point cloud data features is as follows:

[0017] Based on the 3D convolutional neural network, the features of hyperspectral data are extracted. The preprocessed hyperspectral data with the shape of (H, W, D) is input. After multiple 3D convolutional layers, spatial and spectral information is extracted. A pooling layer is connected after each convolutional layer. The size of the feature map is reduced by downsampling operation, and the main features are retained. The hyperspectral data features are output through the output layer, where H and W are the height and width of the image, and D is the number of spectral bands. The preprocessed point cloud data is regarded as a graph structure, and each point is used as a node to construct an adjacency matrix. The preprocessed point cloud data is input into the graph neural network, and features of each point are extracted by sharing a multi-layer perceptron. After the output of the last layer of the graph neural network, the global features are obtained by maximum pooling, and the point cloud data features are output.

[0018] Furthermore, the specific logic for obtaining fusion features is as follows:

[0019] Input the hyperspectral data features and point cloud data features, and obtain Q, K, and V through linear changes. The formula is:

[0020] Q R =W Q R

[0021] K C =W K C

[0022] V C =W V C

[0023] Among them, Q, K, and V correspond to Query, Key, and Value in the self-attention mechanism, respectively, and W Q , W K , W V is the corresponding linear transformation weight obtained through learning, R is the hyperspectral feature data, and C is the point cloud feature data; by calculating the dot product of Q and K, a similarity score is obtained, and the similarity score is converted into attention weight through the softmax function. Finally, the attention weight matrix is ​​obtained by weighted summation of each V according to the attention weight, and the fusion features of hyperspectral data and point cloud data are obtained respectively according to the attention weight matrix, and then spliced ​​to obtain the final fusion feature F.

[0024] Furthermore, the specific logic for obtaining the tree category probability is as follows:

[0025] Use the expert scoring method to assign a class score z to each tree class j j , the fusion feature F and all class scores z are input into the softmax classifier, and the formula for calculating the probability of each category being y is:

[0026]

[0027] Where P(y=j|F) is the probability that the sample belongs to category j given the fusion feature F, and M is the number of tree species categories;

[0028] According to the category vector output by the model, the category index with the maximum probability is selected as the predicted category. In the above technical solution, the technical effects and advantages provided by the present invention are:

[0029] The present invention combines hyperspectral data with point cloud data for tree identification. Hyperspectral data provides rich spectral information, and point cloud data provides accurate three-dimensional structural information. The two complement each other and can significantly improve the recognition accuracy. The attention mechanism is introduced to achieve effective fusion of multimodal data by learning the correlation between hyperspectral data and point cloud data. The attention mechanism can dynamically adjust the importance of each feature and improve the model's adaptability to complex environments. Convolutional neural networks are used to extract feature representations of hyperspectral data and point cloud data respectively to ensure the effectiveness and robustness of feature extraction. Compared with traditional methods, convolutional neural networks can automatically learn features, reduce manual intervention, and improve the degree of system automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0032] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0033] Example:

[0034] See also Figure 1 The present invention provides a tree recognition system based on the integration of hyperspectral and point cloud data with an attention mechanism convolutional neural network, comprising:

[0035] The data acquisition module is used to collect the hyperspectral data and point cloud data of trees. The hyperspectral data records the spectral value of each pixel, and the point cloud data records the three-dimensional coordinates and intensity value of each point.

[0036] In this embodiment, the specific logic for collecting hyperspectral data and point cloud data is as follows:

[0037] Hyperspectral data of trees are collected using hyperspectral equipment, and the spectral information of each pixel is calibrated as S(x, y, λ), which represents the reflectivity of the wavelength λ of the pixel at the (x, y) position. LiDAR equipment is used to collect point cloud data at high density. The point cloud data records the three-dimensional coordinates and intensity value of each point. The calibration point cloud data set is q = {(x i ,y i ,z i ,I i )|i=1,2,…,N}, N is the total number of points in the point cloud.

[0038] Hyperspectral imaging technology can capture the reflectance information of each pixel at multiple wavelengths. These spectral information provides the spectral characteristics of the object, which can effectively distinguish different tree species and their physiological states, especially in vegetation monitoring and health assessment. Through the hyperspectral imaging equipment, multiple bands ranging from 400nm to 2500nm can be collected, and the LiDAR equipment can obtain the three-dimensional structural information of the collected trees, including the height, crown width and other spatial characteristics of the trees. The combination of hyperspectral data and point cloud data can improve the accuracy of tree identification. The spectral characteristics provided by hyperspectral data can help identify the types of trees, while point cloud data supplements the spatial and shape information to make the classification model more comprehensive. In complex environments (such as occlusion, interlacing of multiple vegetation, etc.), a single data source may lead to recognition errors. By fusing hyperspectral and point cloud data, the system can better adapt to various environmental conditions and improve the robustness of recognition.

[0039] A data preprocessing module is used to preprocess the collected spectral data and point cloud data, including correction processing of spectral data and denoising and registration processing of point cloud data;

[0040] In this embodiment, the specific logic for preprocessing the hyperspectral data and point cloud data is as follows:

[0041] Each spectral value in the spectral data is firstly calibrated by radiation, and then the hyperspectral data is aligned to the actual geographic coordinate system using ground control points through geometric correction. Back projection is used for geometric transformation, and finally wavelet transform is used to denoise the hyperspectral data to obtain preprocessed hyperspectral data. The conditional filtering method is used to remove outliers, and an outlier threshold is set. If the point qi If the distance to the mean of its neighborhood is greater than the outlier threshold, the point is marked as an outlier, all points are traversed, and the unmarked points are retained to form a new point cloud data set. The transformation matrix is ​​calculated through the ICP algorithm, and the new point cloud data set is registered to obtain the registered point cloud data.

[0042] Spectral data and point cloud data may be affected by various factors during the collection process (such as illumination, sensor noise, etc.), which requires correction and denoising to ensure the accuracy and consistency of the data. Aligning the spectral data to the actual geographic coordinate system is to ensure the spatial consistency between different data sources so that subsequent analysis can be performed within the same reference frame.

[0043] Radiation correction and geometric correction can significantly reduce data deviations caused by equipment errors and environmental changes, thereby improving the quality of spectral data and making it more reliable. Regarding denoising and outliers, noise and outliers can be effectively removed through methods such as wavelet transform and conditional filtering, which can improve the signal-to-noise ratio of the data and ensure more accurate feature input during analysis. The outlier threshold value can be three times the standard deviation of the neighborhood points. Point cloud data is processed by registration to ensure the spatial consistency of hyperspectral data and point cloud data, making subsequent feature extraction and fusion analysis more practical and effective. Preprocessed data can better reflect the actual situation in the subsequent feature extraction and classification process, thereby improving the robustness of the model and reducing classification errors caused by data quality issues. In addition, high-quality preprocessed data can reduce the complexity of subsequent analysis, thereby saving computing resources and time and improving the efficiency of the overall system. For tree recognition, background noise and outliers may cause misclassification. Preprocessing to eliminate these interferences can capture the characteristics of trees more accurately.

[0044] Multimodal feature extraction module, used to extract features from hyperspectral data and point cloud data based on convolutional neural networks and graph convolutional networks, and obtain hyperspectral data features and point cloud data features;

[0045] In this embodiment, the specific logic for obtaining the hyperspectral data features and point cloud data features is as follows:

[0046] Based on the 3D convolutional neural network, the features of hyperspectral data are extracted. The preprocessed hyperspectral data with the shape of (H, W, D) is input. After multiple 3D convolutional layers, spatial and spectral information is extracted. A pooling layer is connected after each convolution layer. The size of the feature map is reduced by downsampling operation, and the main features are retained. The hyperspectral data features are output through the output layer, where H and W are the height and width of the image, and S is the number of spectral bands. The preprocessed point cloud data is regarded as a graph structure, and each point is used as a node to construct an adjacency matrix. The preprocessed point cloud data is input into the graph neural network, and features of each point are extracted by sharing a multi-layer perceptron. After the output of the last layer of the graph neural network, the global features are obtained by maximum pooling, and the point cloud data features are output.

[0047] Hyperspectral data and point cloud data have different characteristics and information. Hyperspectral data emphasizes the spectral characteristics of objects, while point cloud data provides spatial geometric information. Through multimodal feature extraction, the information of these two types of data can be fully utilized to enhance the expressiveness of the model. Convolutional neural networks and graph convolutional networks have powerful capabilities in feature extraction and can automatically learn potential patterns and features in data, which is particularly important for high-dimensional data. Extracting hyperspectral data features through 3D convolutional neural networks can effectively capture the correlation between spatial structure and spectral information, ensuring that important information is not lost during feature extraction. Point cloud data is processed under a graph structure, so that the features of each point not only rely on its own information, but also integrate neighborhood information. This method can fully explore local and global features in point cloud data. Multimodal feature extraction can generate richer feature representations, thereby improving the accuracy and reliability of tree classification. By combining features from different data sources, the model can better cope with complex environments and diversity. Pooling operations reduce the size of feature maps, which can effectively reduce computational complexity and avoid overfitting, while retaining the main features, thereby improving the generalization ability of the model.

[0048] The attention mechanism fusion module is used to analyze hyperspectral data and point cloud data based on the attention mechanism, obtain the attention weight matrix, calculate the fusion features of hyperspectral data features and point cloud data respectively, and then splice them to obtain the final fusion features;

[0049] In this embodiment, the specific logic for obtaining the fusion features is:

[0050] Input the hyperspectral data features and point cloud data features, and obtain Q, K, and V through linear changes. The formula is:

[0051] Q R =W Q R

[0052] K C =W K C

[0053] V C =W V C

[0054] Among them, Q, K, and V correspond to Query, Key, and Value in the self-attention mechanism, respectively, and W Q , W K , W r is the corresponding linear transformation weight obtained through learning, R is the hyperspectral feature data, and C is the point cloud feature data; by calculating the dot product of Q and K, a similarity score is obtained, and the similarity score is converted into attention weight through the softmax function. Finally, the attention weight matrix is ​​obtained by weighted summation of each V according to the attention weight, and the fusion features of hyperspectral data and point cloud data are obtained respectively according to the attention weight matrix, and then spliced ​​to obtain the final fusion feature F.

[0055] The attention mechanism can automatically assign different weights to input features, highlighting important features and suppressing irrelevant or redundant information. The weighting mechanism enables the model to focus more on key information that helps with identification. The attention mechanism can better capture the relationship between features and improve the expressiveness of features in classification tasks, especially when dealing with complex vegetation and environments. In the process of feature fusion, it can adaptively adjust according to the specific situation of the input data without manually setting weights, which can effectively cope with the diversity and complexity of input features. By focusing on important features and suppressing unimportant features, the model's robustness to noise and abnormal data is significantly enhanced, which helps to improve the accuracy of classification.

[0056] The tree recognition module is used to input the fused features into the classifier, calculate the probability of identifying the tree as each category, and output the category vector.

[0057] In this embodiment, the specific logic for obtaining the tree category probability is as follows:

[0058] The expert scoring method is used to assign a class score z to each tree class j. j , the fusion feature F and all class scores z are input into the softmax classifier, and the formula for calculating the probability of each category being y is:

[0059]

[0060] Where P(y=j|F) is the probability that the sample belongs to category j given the fusion feature F, and M is the number of tree species categories;

[0061] According to the category vector output by the model, the category index with the maximum probability is selected as the predicted category.

[0062] Through the softmax classifier, the output of multiple categories can be converted into probability distributions to facilitate category selection. This process ensures that the output of the model is more interpretable and makes it easier to understand the relative importance of each category. The output of the softmax function provides a probability value for each category, so that the classification result is not only a category label, but also reflects the model's confidence in each category, which is helpful for decision analysis. Combined with the expert scoring method, it provides an intuitive understanding of the output probability of each category, helping users and researchers to better understand the decision-making process of the model. This model framework can flexibly adapt to different numbers and types of tree categories. It can be used for new tree species classification tasks by simply adjusting the class scores. By selecting the category index with the maximum probability as the predicted category, the occurrence of misclassification can be reduced, especially in complex environments or in distinguishing between similar tree species.

[0063] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0064] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A tree recognition system based on the integration of hyperspectral and point cloud data with attention mechanism convolutional neural network, characterized by: include: The data acquisition module is used to collect the hyperspectral data and point cloud data of trees. The hyperspectral data records the spectral value of each pixel, and the point cloud data records the three-dimensional coordinates and intensity value of each point. A data preprocessing module is used to preprocess the collected spectral data and point cloud data, including correction processing of spectral data and denoising and registration processing of point cloud data; Multimodal feature extraction module, used to extract features from hyperspectral data and point cloud data based on convolutional neural networks and graph convolutional networks, and obtain hyperspectral data features and point cloud data features; The attention mechanism fusion module is used to analyze hyperspectral data and point cloud data based on the attention mechanism, obtain the attention weight matrix, calculate the fusion features of hyperspectral data features and point cloud data respectively, and then splice them to obtain the final fusion features; The tree recognition module is used to input the fused features into the classifier, calculate the probability of identifying the tree as each category, and output the category vector.

2. The tree recognition system based on hyperspectral and point cloud data integration based on attention mechanism convolutional neural network according to claim 1 is characterized in that: The specific logic for collecting hyperspectral data and point cloud data is as follows: Hyperspectral data of trees are collected using hyperspectral equipment, and the spectral information of each pixel is calibrated as S(x, y, λ), which represents the reflectivity of the wavelength λ of the pixel at the (x, y) position. LiDAR equipment is used to collect point cloud data at high density. The point cloud data records the three-dimensional coordinates and intensity value of each point. The calibration point cloud data set is q = {(x i ,y i ,z i ,I i )|i=1,2,…,N}, N is the total number of points in the point cloud.

3. The tree recognition system based on hyperspectral and point cloud data integration based on attention mechanism convolutional neural network according to claim 2 is characterized in that: The specific logic for preprocessing hyperspectral data and point cloud data is as follows: Each spectral value in the spectral data is firstly calibrated by radiation, and then the hyperspectral data is aligned to the actual geographic coordinate system using ground control points through geometric correction, and geometric transformation is performed using back projection. Finally, wavelet transform is used to denoise the hyperspectral data to obtain pre-processed hyperspectral data. The conditional filtering method is used to remove outliers and an outlier threshold is set. If the point q i If the distance to the mean of its neighborhood is greater than the outlier threshold, the point is marked as an outlier, all points are traversed, and the unmarked points are retained to form a new point cloud data set. The transformation matrix is ​​calculated through the ICP algorithm, and the new point cloud data set is registered to obtain the registered point cloud data.

4. The tree recognition system based on hyperspectral and point cloud data integration based on attention mechanism convolutional neural network according to claim 3 is characterized in that: The specific logic for obtaining the features of hyperspectral data and point cloud data is as follows: Based on the 3D convolutional neural network, the features of hyperspectral data are extracted. The preprocessed hyperspectral data with the shape of (H, W, D) is input. After multiple 3D convolutional layers, spatial and spectral information is extracted. A pooling layer is connected after each convolutional layer. The size of the feature map is reduced by downsampling operation, and the main features are retained. The hyperspectral data features are output through the output layer, where H and W are the height and width of the image, and D is the number of spectral bands. The preprocessed point cloud data is regarded as a graph structure, and each point is used as a node to construct an adjacency matrix. The preprocessed point cloud data is input into the graph neural network, and features of each point are extracted by sharing a multi-layer perceptron. After the output of the last layer of the graph neural network, the global features are obtained by maximum pooling, and the point cloud data features are output.

5. The tree recognition system based on hyperspectral and point cloud data integration based on attention mechanism convolutional neural network according to claim 4 is characterized in that: The specific logic for obtaining fusion features is: Input the hyperspectral data features and point cloud data features, and obtain Q, K, and V through linear changes. The formula is: Q R =W Q R K C =W K C V C =W V C Among them, Q, K, and V correspond to Query, Key, and Value in the self-attention mechanism, respectively, and W Q , W K , W V is the corresponding linear transformation weight obtained through learning, R is the hyperspectral feature data, and C is the point cloud feature data; by calculating the dot product of Q and K, a similarity score is obtained, and the similarity score is converted into attention weight through the softmax function, and finally the attention weight matrix is ​​obtained by weighted summation of each V according to the attention weight, and the fusion features of hyperspectral data and point cloud data are obtained respectively according to the attention weight matrix, and then spliced ​​to obtain the final fusion feature F.

6. The tree recognition system based on hyperspectral and point cloud data integration based on attention mechanism convolutional neural network according to claim 5 is characterized in that: The specific logic for obtaining tree category probabilities is: Use the expert scoring method to assign a class score z to each tree class j j , the fusion feature F and all class scores z are input into the softmax classifier, and the formula for calculating the probability of each category being y is: Where P(y=j|F) is the probability that the sample belongs to category j given the fusion feature F, and M is the number of tree species categories; According to the category vector output by the model, the category index with the maximum probability is selected as the predicted category.

Images