Single tree instance segmentation method based on scale pyramid graph network

Abstract

This invention discloses a single-tree instance segmentation method based on a scale pyramid graph network. The method includes: acquiring a hyperspectral image to be processed; obtaining multiple superpixel images based on the hyperspectral image; obtaining corresponding pixel scale features for each superpixel image; obtaining corresponding superpixel scale features based on each pixel scale feature, and constructing a global graph based on all superpixel scale features; performing a classification operation on each superpixel image in the global graph to obtain a superpixel classification result; performing a filtering operation on the global graph based on the superpixel classification result to obtain a tree sub-image; obtaining edge prediction features corresponding to each superpixel image based on the tree sub-image; performing an edge prediction operation based on all edge prediction features to obtain a superpixel aggregation result; and integrating the superpixel classification result and the superpixel aggregation result to obtain a single-tree instance segmentation result. This invention improves the feature extraction capability of hyperspectral images and enhances the accuracy of single-tree segmentation tasks.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, terminal, and computer-readable storage medium for single-tree instance segmentation based on a scale pyramid graph network. Background Technology

[0002] Combining image acquisition with Unmanned Aerial Vehicles (UAVs) for individual tree segmentation has become a research hotspot in recent years. UAVs, also known as unmanned aerial vehicles, can capture hyperspectral images at millimeter spatial resolution. Compared to traditional aircraft-based remote sensing methods, they are a cost-effective alternative, making tree mapping more affordable and easier to implement, even for small projects or resource-constrained areas. The spatial resolution of images acquired by UAVs can be higher than that obtained through satellite remote sensing. UAVs equipped with hyperspectral sensors can more accurately identify and label individual trees, achieving a higher spatial resolution than traditional satellite remote sensing.

[0003] Existing methods typically use RGB image information or point cloud data for single tree instance segmentation. The ultra-high spatial resolution of hyperspectral images has not been fully and effectively utilized. Due to the irregular shape of the tree crown, although hyperspectral images provide more useful information, they lead to data redundancy in feature extraction, resulting in greater difficulty and lower accuracy in single tree instance segmentation.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a single-tree instance segmentation method based on a scale pyramid graph network, which aims to solve the problem that the existing technology has poor feature extraction capabilities for hyperspectral images, resulting in high difficulty and low accuracy in single-tree instance segmentation.

[0006] To achieve the above objectives, this invention provides a method for single-tree instance segmentation based on a scale-based pyramid graph network. The method includes the following steps: A hyperspectral image to be processed is acquired, and a feature map construction operation is performed based on the hyperspectral image to obtain multiple superpixel images; Perform pixel-scale feature extraction on each superpixel image to obtain the pixel-scale features corresponding to each superpixel image; Based on each pixel-scale feature, a superpixel-scale feature extraction operation is performed to obtain the superpixel-scale features corresponding to each superpixel map, and a global map is constructed based on all the superpixel-scale features. Based on the global graph, a classification feature extraction operation is performed to obtain global classification features. Based on the global classification features, a classification operation is performed on each superpixel in the global graph to obtain superpixel classification results. Based on the superpixel classification results, a filtering operation is performed on the global graph to obtain tree sub-graphs. Based on the tree subgraph, edge prediction features corresponding to each superpixel graph are calculated, and edge prediction operations are performed based on all the edge prediction features to obtain superpixel aggregation results. The superpixel classification results and the superpixel aggregation results are integrated to obtain single tree instance segmentation results.

[0007] Optionally, the single-tree instance segmentation method based on scale pyramid graph network, wherein acquiring the hyperspectral image to be processed and performing feature map construction based on the hyperspectral image to obtain multiple superpixel images specifically includes: Receive hyperspectral images to be processed sent by the drone; The hyperspectral image is subjected to superpixel segmentation to obtain multiple superpixel blocks; The pixel edge corresponding to each superpixel block is obtained based on the pixel adjacency relationship of each superpixel block; A feature map construction operation is performed on each superpixel block and its corresponding pixel edge to obtain a superpixel map corresponding to each superpixel block.

[0008] Optionally, the single-tree instance segmentation method based on scale pyramid graph network, wherein performing pixel-scale feature extraction on each superpixel image to obtain the pixel-scale features corresponding to each superpixel image specifically includes: Obtain the superpixel block corresponding to each superpixel image, and perform global average pooling and one-dimensional convolution operation on each superpixel block in sequence to obtain the first intermediate feature corresponding to each superpixel block; Each of the first intermediate features is processed by an activation function to obtain the weight matrix corresponding to each superpixel block; Perform unpooling on each weight matrix, and then multiply the weight matrix after each unpooling operation with the corresponding superpixel block element by element to obtain the pixel scale features corresponding to each superpixel image.

[0009] Optionally, the single-tree instance segmentation method based on scale pyramid graph network, wherein the superpixel scale feature extraction operation is performed based on each pixel scale feature to obtain the superpixel scale features corresponding to each superpixel graph, and a global graph is constructed based on all the superpixel scale features, specifically includes: Based on the pixel scale features corresponding to each superpixel image, obtain the first node feature matrix and the first neighbor matrix corresponding to each superpixel image; The first closed-loop adjacency matrix corresponding to each superpixel map is obtained based on each of the first adjacency matrices; The second intermediate feature corresponding to each superpixel image is calculated based on the first node feature matrix and the first closed-loop adjacency matrix corresponding to each superpixel image. Perform global average pooling on each of the second intermediate features to obtain the superpixel scale features corresponding to each superpixel map; The edge set corresponding to each superpixel image is obtained based on the superpixel scale feature corresponding to each superpixel image; Construct a global graph based on all the superpixel graphs and all the edge sets.

[0010] Optionally, the single-tree instance segmentation method based on scale pyramid graph network, wherein the step of performing classification feature extraction based on the global graph to obtain global classification features specifically includes: The second node feature matrix and the second adjacent matrix are obtained based on the global graph. Obtain the second closed-loop adjacency matrix based on the second adjacency matrix; A nonlinear enhancement operation is performed on the second closed-loop adjacency matrix, and a batch normalization operation is performed based on the result of the nonlinear enhancement operation and the second node feature matrix to obtain the third intermediate feature. The third intermediate feature and the second node feature matrix are fused to obtain the global classification feature corresponding to the global graph.

[0011] Optionally, in the single-tree instance segmentation method based on scale pyramid graph network, the global classification features include the classification features corresponding to each superpixel graph; The step of classifying each superpixel in the global graph based on the global classification features to obtain superpixel classification results, and then filtering the global graph based on the superpixel classification results to obtain tree subgraphs, specifically includes: A classification operation is performed based on each of the classification features to obtain the classification result corresponding to each superpixel image; By integrating all the classification results, the superpixel classification result is obtained. The global graph is filtered based on the superpixel classification results, and the superpixel graphs with the classification result of tree are retained to obtain the tree subgraph.

[0012] Optionally, the single-tree instance segmentation method based on scale pyramid graph network, wherein the step of calculating edge prediction features corresponding to each superpixel graph based on the tree subgraph, performing edge prediction operation based on all edge prediction features to obtain superpixel aggregation results, and integrating the superpixel classification results and the superpixel aggregation results to obtain single-tree instance segmentation results, specifically includes: Generate a subgraph mask based on the tree subgraph; Based on the sub-image mask, the sub-image position encoding is calculated to obtain the tree sub-image position encoding corresponding to each superpixel image in the tree sub-image; Perform global average pooling on all the tree subgraph position codes to obtain the subgraph position code corresponding to each superpixel in the global graph; The tree node feature matrix is ​​obtained based on the tree subgraph, and the position codes of all the subgraphs are combined with the tree node feature matrix to obtain the tree edge prediction features, wherein the tree edge prediction features include the edge prediction features corresponding to each superpixel in the tree subgraph; Perform edge prediction operations based on each of the edge prediction features to obtain the superpixel aggregation result; The superpixel aggregation result and the superpixel classification result are fused to obtain the final single-tree instance segmentation result.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a single-tree instance segmentation system based on a scale-based pyramid graph network, wherein the single-tree instance segmentation system based on the scale-based pyramid graph network includes: The superpixel image construction module is used to acquire the hyperspectral image to be processed, and perform feature map construction operation based on the hyperspectral image to obtain multiple superpixel images; The pixel feature extraction module is used to perform pixel scale feature extraction operations on each of the superpixel images to obtain the pixel scale features corresponding to each of the superpixel images. The superpixel feature extraction module is used to perform superpixel scale feature extraction operations based on each of the pixel scale features to obtain the superpixel scale features corresponding to each superpixel map, and to construct a global map based on all the superpixel scale features. The superpixel classification module is used to perform classification feature extraction based on the global graph to obtain global classification features, perform classification operation on each superpixel in the global graph based on the global classification features to obtain superpixel classification results, and perform filtering operation on the global graph based on the superpixel classification results to obtain tree sub-graphs. The single-tree instance segmentation module is used to calculate based on the tree subgraph to obtain the edge prediction features corresponding to each superpixel graph, and to perform edge prediction operation based on all the edge prediction features to obtain the superpixel aggregation result. The superpixel classification result and the superpixel aggregation result are integrated to obtain the single-tree instance segmentation result.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a single-tree instance segmentation program based on a scale-based pyramid graph network stored in the memory and executable on the processor, wherein when the single-tree instance segmentation program based on a scale-based pyramid graph network is executed by the processor, it implements the steps of the single-tree instance segmentation method based on a scale-based pyramid graph network as described above.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a single-tree instance segmentation program based on a scale-based pyramid graph network, and when the single-tree instance segmentation program based on the scale-based pyramid graph network is executed by a processor, it implements the steps of the single-tree instance segmentation method based on the scale-based pyramid graph network as described above.

[0016] In this invention, a hyperspectral image to be processed is acquired, and a feature map construction operation is performed based on the hyperspectral image to obtain multiple superpixel images. Pixel-scale feature extraction is performed on each superpixel image to obtain pixel-scale features corresponding to each superpixel image. Superpixel-scale feature extraction is performed based on each pixel-scale feature to obtain superpixel-scale features corresponding to each superpixel image, and a global map is constructed based on all superpixel-scale features. Classification feature extraction is performed on the global map to obtain global classification features. Classification is performed on each superpixel image in the global map based on the global classification features to obtain superpixel classification results. The global map is then filtered based on the superpixel classification results to obtain tree sub-images. Edge prediction features are calculated based on the tree sub-images to obtain edge prediction features corresponding to each superpixel image, and edge prediction is performed based on all edge prediction features to obtain superpixel aggregation results. The superpixel classification results and the superpixel aggregation results are integrated to obtain single-tree instance segmentation results. This invention designs a scale pyramid module, from pixel, superpixel, sub-scale... Figure 3 The feature extraction process is carried out at each level, which fully explores the spatial-spectral features of hyperspectral images and further improves the feature extraction capability of graph structures. In addition, a subgraph position encoding is designed to make up for the deficiency that graph structures only have adjacency information, and further enhance the natural spatial position information of graph structures, so that the final single-tree segmentation results are more accurate. Attached Figure Description

[0017] Figure 1This is a flowchart of a preferred embodiment of the single-tree instance segmentation method based on a scale-based pyramid graph network of the present invention; Figure 2 This is a detailed flowchart of a preferred embodiment of the single-tree instance segmentation method based on a scale-based pyramid graph network of the present invention; Figure 3 This is a network model structure diagram of a preferred embodiment of the single-tree instance segmentation method based on the scale pyramid graph network of the present invention; Figure 4 This is a scale pyramid module diagram of a preferred embodiment of the single-tree instance segmentation method based on scale pyramid graph network of the present invention; Figure 5 This is a flowchart of pixel scale feature extraction of a preferred embodiment of the single-tree instance segmentation method based on scale pyramid graph network of the present invention; Figure 6 This is a flowchart of subgraph scale feature extraction, which is a preferred embodiment of the single-tree instance segmentation method based on scale pyramid graph network of the present invention. Figure 7 This is a schematic diagram of the subgraph mask of a preferred embodiment of the single-tree instance segmentation method based on the scale pyramid graph network of the present invention; Figure 8 This is a schematic diagram illustrating the principle of a preferred embodiment of the single-tree instance segmentation system based on a scale pyramid graph network of the present invention. Figure 9 This is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] The preferred embodiment of the present invention describes a single-tree instance segmentation method based on a scale-based pyramid graph network, such as... Figure 1 and Figure 2 As shown, the single-tree instance segmentation method based on scale pyramid graph network includes the following steps: Step S10: Obtain the hyperspectral image to be processed, and perform feature map construction operation based on the hyperspectral image to obtain multiple superpixel images.

[0020] Specifically, in this invention, such as Figure 3 As shown, a Scale Pyramid Graph Network (SPGN) is proposed, which consists of three key modules: (a) a scale pyramid module for feature extraction at three different scales; (b) a GCN decoder for obtaining superpixel aggregation; and (c) an MLP classifier for superpixel classification.

[0021] In a preferred embodiment of the present invention, a hyperspectral image that needs to be segmented into a single tree instance is first obtained, and then a feature map construction operation is performed based on the hyperspectral image, thereby obtaining multiple superpixel images corresponding to the hyperspectral image.

[0022] Furthermore, the step of acquiring the hyperspectral image to be processed and performing a feature map construction operation based on the hyperspectral image to obtain multiple superpixel maps specifically includes: The system receives a hyperspectral image to be processed sent by a drone; performs superpixel segmentation on the hyperspectral image to obtain multiple superpixel blocks; obtains the pixel edge corresponding to each superpixel block based on the pixel adjacency relationship of each superpixel block; and performs feature map construction based on each superpixel block and its corresponding pixel edge to obtain a superpixel map corresponding to each superpixel block.

[0023] Specifically, the system first receives hyperspectral images acquired by a drone equipped with a hyperspectral sensor. Then, it uses the SLIC algorithm (Simple Linear Iterative Clustering) to perform superpixel segmentation on the hyperspectral images, thereby dividing them into multiple superpixel blocks. A hyperspectral image can be defined as a superpixel block composed of... A pixel set consisting of pixels is represented as ,in, For hyperspectral images, and These represent the height and width of the hyperspectral image, respectively, and each pixel in the hyperspectral image. It is 3D spectral vector, for 3D real space; a hyperspectral image through superpixel segmentation. Divided into a containing A set of superpixel blocks of superpixel blocks . No. indivual( Superpixel blocks Include 100 pixels, of which 100 pixels are used. .

[0024] Then, superpixel blocks Obtain a set of pixel edges based on the adjacency relationship of its pixels. , indicating superpixel block Each pixel in the image is connected to its four neighboring pixels (top, bottom, left, and right). and Representing superpixel blocks The first in The and the first Each pixel; finally, the superpixel blocks are divided according to the superpixel blocks and their corresponding pixel edges. Converted into superpixel image After processing all superpixel blocks, the superpixel map corresponding to each superpixel block is obtained.

[0025] Step S20: Perform pixel scale feature extraction operation on each superpixel image to obtain the pixel scale features corresponding to each superpixel image.

[0026] Specifically, in a preferred embodiment of the present invention, a Scale Pyramid Module (SPM) is proposed. This module further extracts information from different levels of the hyperspectral image, such as... Figure 3 (a) and Figure 4 As shown, the scale pyramid module is divided into pixels, superpixels, and sub-pixels. Figure 3 Feature extraction at each scale level: First, pixel-scale feature extraction is performed on each superpixel image to capture the pixel-scale features corresponding to each superpixel image.

[0027] Further, the step of performing pixel-scale feature extraction on each of the superpixel images to obtain the pixel-scale features corresponding to each superpixel image specifically includes: Obtain the superpixel block corresponding to each superpixel image, and perform global average pooling and one-dimensional convolution operations on each superpixel block in sequence to obtain the first intermediate feature corresponding to each superpixel block; process each first intermediate feature through an activation function to obtain the weight matrix corresponding to each superpixel block; perform unpooling operation on each weight matrix, and perform element-wise multiplication operation between the weight matrix after each unpooling operation and the corresponding superpixel block to obtain the pixel scale feature corresponding to each superpixel image.

[0028] Specifically, such as Figure 5 As shown, the scale pyramid module in this invention includes an efficient channel attention module for graph convolution, namely... Figure 5 The pixel-scale feature extraction (PSFE) module shown in the diagram recognizes significant variations in spectral information between different tree species. However, as spectral resolution increases, some redundancy exists in the spectral information, making it crucial to capture spectral information with representative differences. Furthermore, although two trees belonging to the same category may exhibit similar spectral information, they may differ in specific bands. The ability to capture these subtle differences directly impacts the quality of the extracted features. Therefore, this invention designs a pixel-scale feature extraction module that uses channel attention to capture information from different bands.

[0029] Tree species exhibit unique spectral characteristics across different wavelength bands. Therefore, pixel-scale feature extraction modules can help achieve weighted allocation for specific wavelength bands in species classification, such as... Figure 5 As shown, a global average pooling operation is first performed on the superpixel blocks corresponding to each superpixel image (equivalent to...). Figure 5 Global mean pooling is used to convert superpixel blocks into one-dimensional vectors, and then a predefined kernel size is used. Perform a one-dimensional convolution operation to obtain the first intermediate feature corresponding to each superpixel block; then use the superpixel block... For example, superpixel blocks The first intermediate feature is represented as ;in, This is a channel-level global average pooling operation. For a kernel size of One-dimensional convolution.

[0030] Then, by inputting each first intermediate feature into the Sigmoid activation function, the weight matrix corresponding to each superpixel block can be obtained, specifically as follows: ;in, It is the Sigmoid activation function. Represents superpixel blocks The corresponding weight matrix.

[0031] Finally, each calculated weight matrix is ​​first unpooled. After unpooling all weight matrices, they are then multiplied element-wise with their corresponding superpixel blocks. This can be represented as follows: ;in, This is an unpooling operation used to... Extended to Same size; It is an element-wise multiplication operation; For superpixel blocks The corresponding pixel-scale features, also known as superpixel blocks The corresponding superpixel image's pixel-scale features; the pixel-scale feature extraction module can learn and capture representative differences in spectral information with fewer parameters, thereby improving the network's efficiency.

[0032] Step S30: Perform superpixel scale feature extraction operation based on each pixel scale feature to obtain the superpixel scale feature corresponding to each superpixel map, and construct a global map based on all the superpixel scale features.

[0033] Specifically, in a preferred embodiment of the present invention, the scale pyramid module further includes a Superpixel Scale Feature Extraction (SPSFE) module, which aims to learn feature representations at the superpixel scale. In the superpixel scale feature extraction module, such as... Figure 3 As shown, during the process of inputting each superpixel block into Simple Graph Convolution (SGC) and Global Average Pooling (GAP), SGC obtains the representation of each node in the superpixel graph through multi-layer graph convolution, and then Global Average Pooling aggregates the information of all nodes in the superpixel graph to obtain the superpixel scale features corresponding to the superpixel graph.

[0034] Further, the step of performing superpixel scale feature extraction based on each pixel scale feature to obtain the superpixel scale features corresponding to each superpixel map, and constructing a global map based on all the superpixel scale features, specifically includes: Based on the pixel scale features corresponding to each superpixel image, obtain the first node feature matrix and the first adjacency matrix corresponding to each superpixel image; based on each first adjacency matrix, obtain the first closed-loop adjacency matrix corresponding to each superpixel image; calculate the second intermediate feature corresponding to each superpixel image based on the first node feature matrix and the first closed-loop adjacency matrix corresponding to each superpixel image; perform global average pooling operation on each second intermediate feature to obtain the superpixel scale feature corresponding to each superpixel image; obtain the edge set corresponding to each superpixel image based on the superpixel scale feature corresponding to each superpixel image; construct a global graph based on all superpixel images and all edge sets.

[0035] Specifically, firstly, based on the pixel-scale features corresponding to each superpixel image, the first node feature matrix and the first neighbor matrix corresponding to each superpixel image are obtained respectively. Then, based on each first neighbor matrix, the corresponding first closed-loop adjacency matrix is ​​obtained respectively. Using the superpixel image... For example, superpixel images The corresponding pixel scale features are ,according to The feature matrix of the first node is obtained as follows The first adjacent matrix is For the first adjacent matrix After adding self-loops, the first closed-loop adjacency matrix is ​​obtained. .

[0036] Then, based on the first node feature matrix and the first closed-loop adjacency matrix corresponding to each superpixel image, the second intermediate feature corresponding to each superpixel image is calculated; based on the first node feature matrix... Adjacency matrix of the first closed loop Calculations were performed to obtain the second intermediate feature. The specific calculation formula is as follows: ;in, This is a trainable matrix in the superpixel scale feature extraction module; K In a preferred embodiment of the invention, the neighbor hop count of the SGC is represented. K Set to 3; It is the feature matrix of the first node Diagonal matrices of the same dimension.

[0037] Finally, a global average pooling operation is performed on the second intermediate features to obtain the superpixel scale features corresponding to each superpixel image; for the second intermediate features The global average pooling operation is performed, and the specific calculation formula is as follows: ;in, This indicates a global average pooling operation; Represents superpixel image The corresponding superpixel scale features; after the superpixel scale feature extraction operation is completed for each superpixel image, a superpixel scale feature set is obtained. , of which indivual( Superpixel image The corresponding superpixel scale features can also be represented as After superpixel-scale feature extraction, low-frequency information, such as texture features in uniform regions, is preserved, while high-frequency information, such as noise, is greatly reduced.

[0038] Considering that superpixel segmentation often over-segments hyperspectral images and strives for superpixel uniformity, this typically results in information about a single instance being scattered across multiple superpixel maps. Furthermore, the spatial relationships between different instances play a crucial role in instance recognition. To address this issue, it is necessary to obtain an edge set based on the adjacency relationships between superpixel maps. This involves acquiring the edge set corresponding to each superpixel map based on its superpixel scale features, and then integrating all edge sets to obtain a global edge set. Finally, based on the superpixel scale feature set and global edge set Building a global graph .

[0039] Step S40: Perform classification feature extraction operation on the global graph to obtain global classification features, perform classification operation on each superpixel image in the global graph according to the global classification features to obtain superpixel classification results, and perform filtering operation on the global graph according to the superpixel classification results to obtain tree sub-images.

[0040] Specifically, in a preferred embodiment of the present invention, such as Figure 6 As shown, the scale pyramid module also includes a subgraph scale feature extraction module. After extracting superpixel features from all superpixel graphs, a global graph is obtained. In the context of the global graph, effectively utilizing the neighborhood information of each node can enhance the understanding of the node environment and its surrounding context. Including neighborhood information during feature extraction plays a crucial role in node classification and edge prediction tasks. Considering the layering and signal propagation characteristics of graph convolutional networks, this invention designs an N-layer graph convolutional network for extracting neighborhood information and enhancing contextual understanding. This module is the subgraph scale feature extraction module. The subgraph scale feature extraction module performs classification feature extraction operations on the global graph to obtain global classification features. Then, the global graph is classified according to the global classification features to obtain the superpixel classification results. Finally, the global graph is filtered according to the superpixel classification results to obtain the corresponding tree subgraphs.

[0041] Furthermore, the step of performing classification feature extraction based on the global graph to obtain global classification features specifically includes: The second node feature matrix and the second adjacency matrix are obtained from the global graph; the second closed-loop adjacency matrix is ​​obtained from the second adjacency matrix; a nonlinear enhancement operation is performed on the second closed-loop adjacency matrix, and a batch normalization operation is performed based on the result of the nonlinear enhancement operation and the second node feature matrix to obtain the third intermediate feature; the third intermediate feature and the second node feature matrix are fused to obtain the global classification feature corresponding to the global graph.

[0042] Specifically, this invention introduces a nonlinear operation, ReLU(), after each graph convolutional network layer to enhance the nonlinear expressive power of features. However, iteratively applying linear and nonlinear operations during propagation may distort the feature distribution, thus posing a challenge to the network's convergence. To alleviate this challenge, a batch normalization layer is introduced to normalize the output, making it closer to a distribution with a mean of 0 and a variance of 1.

[0043] First, based on the global graph Obtain the feature matrix of the second node Second adjacent matrix Then, for the second adjacent matrix Adding self-loops yields the second closed-loop adjacency matrix. Then, for the second closed-loop adjacency matrix A nonlinear enhancement operation is performed, followed by batch normalization based on the result of the nonlinear enhancement operation and the feature matrix of the second node, thus obtaining the third intermediate feature. The specific formula is as follows: ;in, Indicates the number of floors; Indicates the first The trainable matrix of the layer; when hour, ; It is the second adjacent matrix Diagonal matrices of the same dimension; Indicates the first Layer input, Indicates the first The output of the layer; considering that the relevant features of the nodes are mainly concentrated in low-order regions, and that as the number of layers in the graph convolutional network increases, the receptive field of the nodes expands, thus fusing more heterogeneous information. This may lead to a smoothing effect between features, ultimately resulting in a decrease in model performance. Therefore, deeper network layers may not necessarily lead to better results. Considering the above, in the preferred embodiment of the present invention, It is set to 2; therefore, after processing, a third intermediate feature can be obtained. .

[0044] To further improve network performance and effectively utilize shallow network features, this invention incorporates residual blocks into the design, resulting in a final output global classification feature (subgraph scale feature). This involves fusing the third intermediate feature and the second node feature matrix to obtain the global classification feature, which is specifically represented as follows: ,in, Indicates the first The classification features corresponding to each superpixel image.

[0045] Further, the global classification features include classification features corresponding to each superpixel image; the step of performing a classification operation on each superpixel image in the global image based on the global classification features to obtain a superpixel classification result, and then performing a filtering operation on the global image based on the superpixel classification result to obtain a tree sub-image, specifically includes: Classification operations are performed based on each of the classification features to obtain the classification result corresponding to each superpixel image; all the classification results are integrated to obtain the superpixel classification result; the global image is filtered according to the superpixel classification result, and the superpixel images whose classification result is a tree are retained to obtain the tree sub-image.

[0046] Specifically, such as Figure 2 and Figure 3 In (c) of the preferred embodiment of the invention, a multilayer perceptron (MLP) classifier is used to classify each superpixel in the global graph, thereby classifying the superpixel based on the classification features corresponding to each superpixel. ( Perform classification operations on each superpixel image to obtain the classification result for each superpixel image. Specifically, it can be expressed as: It should be noted that the classification results include whether it is a tree and the type of tree.

[0047] Based on the classification results of all superpixel images, a filtering operation is performed on the global image to remove superpixel images whose classification result is not tree, retaining only the superpixel images whose classification result is tree, thus obtaining the tree subgraph. .

[0048] Step S50: Calculate the edge prediction features corresponding to each superpixel image based on the tree subgraph, and perform edge prediction operation based on all the edge prediction features to obtain the superpixel aggregation result. Integrate the superpixel classification result and the superpixel aggregation result to obtain the single tree instance segmentation result.

[0049] Specifically, in a preferred embodiment of the present invention, since the constructed global graph, superpixel graph, and tree subgraph only have adjacency information and cannot capture spatial information well, the present invention proposes Subgraph Position Encoding (SPE) to further enhance the spatial information of the graph. Based on the tree subgraph, the subgraph position encoding of each superpixel graph in the global graph can be obtained. Then, the edge prediction features corresponding to each superpixel graph are obtained according to all the subgraph position encodings. The edge prediction operation is performed on each superpixel graph according to the edge prediction features to obtain the superpixel aggregation result. The superpixel aggregation result reveals whether two superpixel graphs are the same tree. Finally, the superpixel aggregation result and the superpixel classification result are fused to obtain the final single tree instance segmentation result.

[0050] Further, the calculation based on the tree subgraph to obtain the edge prediction features corresponding to each superpixel graph, and the edge prediction operation based on all the edge prediction features to obtain the superpixel aggregation result, and the integration of the superpixel classification result and the superpixel aggregation result to obtain the single tree instance segmentation result, specifically includes: A subgraph mask is generated based on the tree subgraph; subgraph position encoding is calculated based on the subgraph mask to obtain the tree subgraph position encoding corresponding to each superpixel image in the tree subgraph; a global average pooling operation is performed on all the tree subgraph position encodings to obtain the subgraph position encoding corresponding to each superpixel image in the global image; a tree node feature matrix is ​​obtained based on the tree subgraph, and all the subgraph position encodings are combined with the tree node feature matrix to obtain tree edge prediction features, wherein the tree edge prediction features include the edge prediction features corresponding to each superpixel image in the tree subgraph; an edge prediction operation is performed based on each edge prediction feature to obtain a superpixel aggregation result; the superpixel aggregation result and the superpixel classification result are fused to obtain the final single tree instance segmentation result.

[0051] Specifically, the initial design purpose of subgraph positional encoding is to enhance the natural spatial positional information lacking in graph structures. Subgraph positional encoding aims to explore the spatial relationships between nodes by combining the use of subgraph masks and positional encoding. Since the constructed global graph, superpixel graph, and tree subgraph are built based on neighborhood relationships, they only have adjacency information and lack natural spatial positional information between superpixel graphs. Natural spatial positional information includes the distribution relationship between trees, which has certain guiding significance for the identification of tree canopies. Furthermore, natural spatial positional relationships can also guide the similarity matching between spatial texture details from different superpixel graphs, and therefore can be used to assist in determining whether two superpixel graphs belong to the same tree.

[0052] First, based on the tree subgraph Generate a subgraph mask, specifically as follows: Figure 7 As shown in (a), the values ​​1 and 0 represent the presence and absence of a tree, respectively. A subgraph refers to a region composed of adjacent pixels with the same attributes (i.e., pixel values). Therefore, for each connected subgraph, the corresponding bounding rectangle mask can be obtained based on the subgraph mask. Then, the relative position index of the bounding rectangle mask is calculated, and the cumulative sum is calculated along specific dimensions (such as the x-axis and y-axis). Finally, the subgraph mask and the relative position index are multiplied element-wise to obtain the relative position index of the subgraph. The calculation of the cumulative sum is as follows: Figure 7 As shown in (b) and (c) in the figure.

[0053] Subsequently, the sinusoidal positional encoding technique used in the Transformer is calculated with its relative position to obtain the positional encoding matrix corresponding to the tree subgraph. The positional encoding matrix contains the positional encoding of the tree subgraph corresponding to each superpixel in the tree subgraph. The specific formula for calculating the value of each element in the positional encoding matrix is ​​as follows: ; ;in, pos Indicates the relative position index of the subgraph; i Index representing the dimension; In a preferred embodiment of the invention, the dimension output by the superpixel scale feature extraction module is set to 128. Then, a global flat pooling operation is performed on the position codes of all tree subgraphs in the position coding matrix to obtain the position code corresponding to each superpixel in the global graph.

[0054] According to the tree subgraph Obtain the feature matrix of tree nodes ,in, The first in the tree subgraph n Each superpixel image; encode all locations and combine them with the tree node feature matrix. Combined, tree-based prediction features are obtained for edge prediction. ;in, Represents the th subgraph of the tree. n Edge prediction features corresponding to each superpixel image; such as Figure 2 and Figure 3 As shown in (b), in a preferred embodiment of the present invention, edge prediction is performed using a GCN decoder to achieve superpixel aggregation and obtain superpixel aggregation results. First, the obtained tree edge prediction features are passed through a graph convolutional layer, then through a nonlinear activation function ReLU() and batch normalization. Finally, the cosine similarity between adjacent superpixel images is calculated. The formula for calculating the cosine similarity is: ;in, and Tree edge prediction features The i Each edge prediction feature and the first j Each edge prediction feature ( i Not equal to j Finally, the calculated similarities are batch normalized and activated by the sigmoid function to generate a representation. and Whether the corresponding superpixel images belong to the aggregation result of the same tree After completing the superpixel aggregation task for any two superpixel images in the tree subgraph, the superpixel aggregation result is obtained. The superpixel aggregation result shows whether any two superpixel images in the tree subgraph belong to the same tree.

[0055] Finally, the superpixel aggregation results and superpixel classification results are fused to obtain the final single-tree instance segmentation results.

[0056] Furthermore, the superpixel classification loss is measured using cross-entropy loss: ;in, It is the first The classification results of each superpixel image. The total number of superpixel images is denoted as . After classifying all superpixel images, the classification result for each superpixel image is obtained. All classification results are then integrated to obtain the superpixel classification result. In the superpixel aggregation task, the edge prediction loss is measured using the binary cross-entropy loss. Therefore, the final loss function can be expressed as: .

[0057] Furthermore, such as Figure 8 As shown, based on the above-described single-tree instance segmentation method based on scale-based pyramid graph networks, this invention also provides a single-tree instance segmentation system based on scale-based pyramid graph networks, wherein the single-tree instance segmentation system based on scale-based pyramid graph networks includes: The superpixel image construction module 51 is used to acquire the hyperspectral image to be processed, and perform feature map construction operation based on the hyperspectral image to obtain multiple superpixel images. The pixel feature extraction module 52 is used to perform pixel scale feature extraction operations on each of the superpixel images to obtain the pixel scale features corresponding to each of the superpixel images. The superpixel feature extraction module 53 is used to perform superpixel scale feature extraction operation based on each pixel scale feature to obtain the superpixel scale feature corresponding to each superpixel map, and construct a global map based on all the superpixel scale features. The superpixel classification module 54 is used to perform classification feature extraction operation based on the global graph to obtain global classification features, perform classification operation on each superpixel graph in the global graph based on the global classification features to obtain superpixel classification results, and perform filtering operation on the global graph based on the superpixel classification results to obtain tree sub-graphs. The single-tree instance segmentation module 55 is used to calculate based on the tree subgraph to obtain the edge prediction features corresponding to each superpixel graph, and to perform edge prediction operation based on all the edge prediction features to obtain the superpixel aggregation result. The superpixel classification result and the superpixel aggregation result are integrated to obtain the single-tree instance segmentation result.

[0058] Furthermore, such as Figure 9 As shown, based on the above-mentioned single-tree instance segmentation method and system based on scale pyramid graph network, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 9 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0059] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a single-tree instance segmentation program 40 based on a scale-based pyramid graph network, which can be executed by the processor 10 to implement the single-tree instance segmentation method based on a scale-based pyramid graph network in this application.

[0060] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the single-tree instance segmentation method based on the scale pyramid graph network.

[0061] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0062] In one embodiment, when the processor 10 executes the single-tree instance segmentation program 40 based on the scale pyramid graph network in the memory 20, the steps of the single-tree instance segmentation method based on the scale pyramid graph network described above are implemented.

[0063] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a single-tree instance segmentation program based on a scale-based pyramid graph network, and the single-tree instance segmentation program based on a scale-based pyramid graph network implements the steps of the single-tree instance segmentation method based on a scale-based pyramid graph network as described above when executed by a processor.

[0064] In summary, this invention provides a single-tree instance segmentation method based on a scale pyramid graph network. The method includes: acquiring a hyperspectral image to be processed; constructing a feature map based on the hyperspectral image to obtain multiple superpixel images; extracting pixel-scale features from each superpixel image to obtain pixel-scale features corresponding to each superpixel image; extracting superpixel-scale features from each pixel-scale feature to obtain superpixel-scale features corresponding to each superpixel image, and constructing a global graph based on all superpixel-scale features; extracting classification features from the global graph to obtain global classification features; classifying each superpixel image in the global graph based on the global classification features to obtain superpixel classification results; filtering the global graph based on the superpixel classification results to obtain tree subgraphs; calculating edge prediction features corresponding to each superpixel image based on the tree subgraphs; performing edge prediction based on all edge prediction features to obtain superpixel aggregation results; and integrating the superpixel classification results and the superpixel aggregation results to obtain a single-tree instance segmentation result. This invention designs a scale pyramid module, which integrates pixels, superpixels, and subgraphs... Figure 3 The feature extraction process is carried out at each level, which fully explores the spatial-spectral features of hyperspectral images and further improves the feature extraction capability of graph structures. In addition, a subgraph position encoding is designed to make up for the deficiency that graph structures only have adjacency information, and further enhance the natural spatial position information of graph structures, so that the final single-tree segmentation results are more accurate.

[0065] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0066] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0067] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for segmenting single-tree instances based on a scale-based pyramid graph network, characterized in that, The single-tree instance segmentation method based on scale pyramid graph network includes: A hyperspectral image to be processed is acquired, and a feature map construction operation is performed based on the hyperspectral image to obtain multiple superpixel images; Perform pixel-scale feature extraction on each superpixel image to obtain the pixel-scale features corresponding to each superpixel image; Based on each pixel-scale feature, a superpixel-scale feature extraction operation is performed to obtain the superpixel-scale features corresponding to each superpixel map, and a global map is constructed based on all the superpixel-scale features. Based on the global graph, a classification feature extraction operation is performed to obtain global classification features. Based on the global classification features, a classification operation is performed on each superpixel in the global graph to obtain superpixel classification results. Based on the superpixel classification results, a filtering operation is performed on the global graph to obtain tree sub-graphs. Based on the tree subgraph, the edge prediction features corresponding to each superpixel graph are calculated, and the edge prediction operation is performed based on all the edge prediction features to obtain the superpixel aggregation result. The superpixel classification result and the superpixel aggregation result are integrated to obtain the single tree instance segmentation result. The calculation based on the tree subgraph yields the edge prediction features corresponding to each superpixel graph. An edge prediction operation is then performed based on all the edge prediction features to obtain the superpixel aggregation result. Finally, the superpixel classification result and the superpixel aggregation result are integrated to obtain the single-tree instance segmentation result, specifically including: Generate a subgraph mask based on the tree subgraph; Based on the sub-image mask, the sub-image position encoding is calculated to obtain the tree sub-image position encoding corresponding to each superpixel image in the tree sub-image; Perform global average pooling on all the tree subgraph position codes to obtain the subgraph position code corresponding to each superpixel in the global graph; The tree node feature matrix is ​​obtained based on the tree subgraph, and the position codes of all the subgraphs are combined with the tree node feature matrix to obtain the tree edge prediction features, wherein the tree edge prediction features include the edge prediction features corresponding to each superpixel in the tree subgraph; Perform edge prediction operations based on each of the edge prediction features to obtain the superpixel aggregation result; The superpixel aggregation result and the superpixel classification result are fused to obtain the final single-tree instance segmentation result.

2. The single-tree instance segmentation method based on scale-based pyramid graph networks according to claim 1, characterized in that, The process of acquiring the hyperspectral image to be processed and performing a feature map construction operation based on the hyperspectral image to obtain multiple superpixel images specifically includes: Receive hyperspectral images to be processed sent by the drone; The hyperspectral image is subjected to superpixel segmentation to obtain multiple superpixel blocks; The pixel edge corresponding to each superpixel block is obtained based on the pixel adjacency relationship of each superpixel block; A feature map construction operation is performed on each superpixel block and its corresponding pixel edge to obtain a superpixel map corresponding to each superpixel block.

3. The single-tree instance segmentation method based on scale-based pyramid graph networks according to claim 1, characterized in that, The step of performing pixel-scale feature extraction on each of the superpixel images to obtain the pixel-scale features corresponding to each superpixel image specifically includes: Obtain the superpixel block corresponding to each superpixel image, and perform global average pooling and one-dimensional convolution operation on each superpixel block in sequence to obtain the first intermediate feature corresponding to each superpixel block; Each of the first intermediate features is processed by an activation function to obtain the weight matrix corresponding to each superpixel block; Perform unpooling on each weight matrix, and then multiply the weight matrix after each unpooling operation with the corresponding superpixel block element by element to obtain the pixel scale features corresponding to each superpixel image.

4. The single-tree instance segmentation method based on scale-based pyramid graph networks according to claim 1, characterized in that, The step of performing superpixel scale feature extraction based on each pixel scale feature to obtain the superpixel scale features corresponding to each superpixel map, and constructing a global map based on all the superpixel scale features, specifically includes: Based on the pixel scale features corresponding to each superpixel image, obtain the first node feature matrix and the first neighbor matrix corresponding to each superpixel image; The first closed-loop adjacency matrix corresponding to each superpixel map is obtained based on each of the first adjacency matrices; The second intermediate feature corresponding to each superpixel image is calculated based on the first node feature matrix and the first closed-loop adjacency matrix corresponding to each superpixel image. Perform global average pooling on each of the second intermediate features to obtain the superpixel scale features corresponding to each superpixel map; The edge set corresponding to each superpixel image is obtained based on the superpixel scale feature corresponding to each superpixel image; Construct a global graph based on all the superpixel graphs and all the edge sets.

5. The single-tree instance segmentation method based on scale-based pyramid graph networks according to claim 1, characterized in that, The step of extracting classification features based on the global graph to obtain global classification features specifically includes: The second node feature matrix and the second adjacent matrix are obtained based on the global graph. Obtain the second closed-loop adjacency matrix based on the second adjacency matrix; A nonlinear enhancement operation is performed on the second closed-loop adjacency matrix, and a batch normalization operation is performed based on the result of the nonlinear enhancement operation and the second node feature matrix to obtain the third intermediate feature. The third intermediate feature and the second node feature matrix are fused to obtain the global classification feature corresponding to the global graph.

6. The single-tree instance segmentation method based on scale-based pyramid graph networks according to claim 1, characterized in that, The global classification features include the classification features corresponding to each superpixel image; The step of classifying each superpixel in the global graph based on the global classification features to obtain superpixel classification results, and then filtering the global graph based on the superpixel classification results to obtain tree subgraphs, specifically includes: A classification operation is performed based on each of the classification features to obtain the classification result corresponding to each superpixel image; By integrating all the classification results, the superpixel classification result is obtained. The global graph is filtered based on the superpixel classification results, and the superpixel graphs with the classification result of tree are retained to obtain the tree subgraph.

7. A single-tree instance segmentation system based on a scale-based pyramid graph network, characterized in that, The single-tree instance segmentation system based on scale-based pyramid graph networks is used to implement the single-tree instance segmentation method based on scale-based pyramid graph networks as described in any one of claims 1-6. The single-tree instance segmentation system based on scale-based pyramid graph networks includes: The superpixel image construction module is used to acquire the hyperspectral image to be processed, and perform feature map construction operation based on the hyperspectral image to obtain multiple superpixel images; The pixel feature extraction module is used to perform pixel scale feature extraction operations on each of the superpixel images to obtain the pixel scale features corresponding to each of the superpixel images. The superpixel feature extraction module is used to perform superpixel scale feature extraction operations based on each of the pixel scale features to obtain the superpixel scale features corresponding to each superpixel map, and to construct a global map based on all the superpixel scale features. The superpixel classification module is used to perform classification feature extraction based on the global graph to obtain global classification features, perform classification operation on each superpixel in the global graph based on the global classification features to obtain superpixel classification results, and perform filtering operation on the global graph based on the superpixel classification results to obtain tree sub-graphs. The single-tree instance segmentation module is used to calculate based on the tree subgraph to obtain the edge prediction features corresponding to each superpixel graph, and to perform edge prediction operation based on all the edge prediction features to obtain the superpixel aggregation result. The superpixel classification result and the superpixel aggregation result are integrated to obtain the single-tree instance segmentation result.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a single-tree instance segmentation program based on a scale-based pyramid graph network stored in the memory and executable on the processor. When the single-tree instance segmentation program based on a scale-based pyramid graph network is executed by the processor, it implements the steps of the single-tree instance segmentation method based on a scale-based pyramid graph network as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a single-tree instance segmentation program based on a scale-based pyramid graph network. When the single-tree instance segmentation program based on the scale-based pyramid graph network is executed by a processor, it implements the steps of the single-tree instance segmentation method based on a scale-based pyramid graph network as described in any one of claims 1-6.

Images