A vegetation remote sensing classification method constrained by knowledge graph ontology

Abstract

The present invention discloses a method for remote sensing classification of vegetation constrained by a knowledge graph ontology, including data acquisition and data preprocessing, deep learning model selection, and the fusion method and steps of the knowledge graph ontology and the deep learning model. This application fuses the knowledge graph ontology and the FCN8s-ResNet 50 model in two stages: model training and post-classification processing. The results show that: 1) Fusing the mountain vegetation knowledge graph ontology in the two stages of deep learning model training and post-classification processing can significantly improve the classification accuracy of the deep learning model: the accuracy of the deep learning model constrained by the knowledge graph ontology has increased by 10%, and the overall classification accuracy of vegetation types has reached 89.2%. 2) The deep learning model constrained by the mountain vegetation knowledge graph ontology not only achieves excellent results in the fine classification of vegetation types, but also improves the universality and interpretability of the model, which will provide strong support for large-scale remote sensing vegetation classification and the application of deep learning models.

Description

Technical Field

[0001] The present invention relates to the technical field of vegetation remote sensing classification, and particularly to a remote sensing classification method for vegetation formation level constrained by a knowledge graph ontology. Background Art

[0002] Vegetation, as an important part of the terrestrial ecosystem, has an important impact on the global material flow, energy flow, carbon balance and climate stability at different spatio-temporal scales. High-precision classification and rapid monitoring of mountain vegetation are important bases for systematically studying the structure and ecological functions of mountain ecosystems, and provide necessary reference information for ecological restoration, climate change and carbon balance. Traditional vegetation classification and mapping mainly rely on field surveys, which are time-consuming and laborious. The development of remote sensing technology not only solves the obstacles of field surveys caused by inconvenient transportation in mountainous environments, but also makes the monitoring of mountain vegetation more real-time and effective.

[0003] At present, using high-resolution remote sensing images for intelligent interpretation has become an important way to study vegetation cover, vegetation structure composition and its dynamic changes. Although the amount of remote sensing data obtained every day has reached the PB level, there are more and more sensors, the acquisition time is getting faster, and there are more and more algorithms for remote sensing image target recognition and classification. However, due to the spectral similarity between vegetation canopies, vegetation classification research is still one of the most challenging problems in remote sensing science. Although deep learning methods are widely used in vegetation classification tasks and have good performance in vegetation cover recognition and dynamic monitoring, they perform poorly in fine vegetation classification, especially in vegetation formation level classification. At the same time, the existing deep learning models for vegetation classification are difficult to be applied to large-scale tasks due to their low interpretability and universality, which limits the application and service capabilities of remote sensing big data. Summary of the Invention

[0004] Aiming at the above existing problems, the present invention aims to provide a remote sensing classification method for vegetation constrained by a knowledge graph ontology. Integrating the mountain vegetation knowledge graph ontology in the model training and classification post-processing stages can significantly improve the classification accuracy of the deep learning model. The deep learning model constrained by the mountain vegetation knowledge graph ontology not only achieves excellent results in the fine classification of vegetation types, but also greatly improves the universality and interpretability of the model, which will provide strong support for remote sensing vegetation classification and application in a wider range.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A remote sensing classification method for vegetation constrained by a knowledge graph ontology, comprising the following steps:

[0006] S1. Data acquisition and data preprocessing

[0007] 1) Obtain the digital surface model data of the study area, and combine with the spatial resampling technology to generate a digital elevation model with the same resolution.

[0008] 2) Obtain the multi-temporal remote sensing and digital elevation model data of the study area, unify the projection coordinate system and perform corresponding cropping according to the resolution, and preprocess the multi-temporal remote sensing and digital elevation model data to be consistent with the input data spatial range.

[0009] 3) Obtain the vegetation vertical zonation spectrum of the study area, and preprocess and transform the vegetation vertical zonation spectrum into a structured knowledge graph ontology that reflects the geographical distribution law of vegetation.

[0010] 4) Obtain the vegetation type map data and remote sensing images of the study area, and obtain samples of each vegetation type in the study area according to the vegetation type map data and remote sensing images to form a sample data set of the vegetation types in the study area.

[0011] S2. Select a deep learning model

[0012] Select a deep learning model that can simultaneously learn the features of remote sensing images with different sources and resolutions and has multiple channels.

[0013] S3. Integrate the knowledge graph ontology and the deep learning model in the training stage

[0014] Input the data preprocessed in steps 1)-2) and 4) in step S1 into the deep learning model for training, and at the same time integrate the structured knowledge graph ontology and the deep learning model in the model training stage to obtain a training model for vegetation classification.

[0015] S4. Integrate the knowledge graph ontology and the deep learning model in the post-classification processing stage

[0016] Integrate the structured knowledge graph ontology and the deep learning model in the post-processing stage of vegetation classification to finally obtain the classification result of vegetation.

[0017] Preferably, divide the sample data set preprocessed in step S1 into a training set and a validation set, and use the deep learning model for training, and at the same time integrate the structured knowledge graph to finally obtain the classification result of vegetation.

[0018] Preferably, during the training process of the deep learning model, integrate the structured knowledge graph information, predict a vegetation type with the deep learning model, and compare the difference between the altitude range of the corresponding vertical zone in the knowledge graph ontology and the true altitude.

[0019] Preferably, based on the loss function of interval overlap, quantify the matching degree between the altitude distribution range of the vegetation type in the knowledge graph ontology and the true altitude. The specific process is as follows:

[0020] and represent the lower and upper limits of the altitude range of the th sample in the knowledge graph ontology respectively. represents the true altitude of the th sample. Then the form of the interval overlap loss is Equation (5-1):

[0021]

[0022] where is a small positive number used to prevent the denominator from being zero;

[0023] Adding this constraint loss to the original cross-entropy loss forms a new composite loss function, as shown in Equation (5-2):

[0024] (5-2)

[0025] where is the cross-entropy loss of the classification task, is the weight coefficient for balancing the classification loss and the interval overlap loss.

[0026] Preferably, use the deep learning model to first perform similarity classification on the prediction results of the deep learning model in step S3 to obtain the similarity values of each category , and then fuse the structured knowledge graph ontology to calculate the similarity to obtain a new predicted similarity distribution , and obtain the final vegetation classification result according to the new similarity.

[0027] Preferably, for the prediction results with similarity less than the threshold, use DEM to obtain the altitude at the corresponding location , and obtain the vertical distribution information of all vegetation types in the knowledge graph ontology, and then calculate the degree of closeness to the vertical zone distribution of each vegetation type, and convert it into a similarity distribution according to Equations (5-3) and (5-4) where represents the possible value range, represents the distribution range of the th vegetation type in the knowledge graph ontology, represents the number of vegetation types:

[0028]

[0029] (5-4)

[0030] The predicted similarity of the model The similarity calculated based on altitude are weighted and fused to generate a new predicted similarity distribution , as shown in formula (5-5):

[0031]

[0032] wherein and are the weights of the model predicted similarity and the similarity calculated based on altitude respectively, and the sum of the two is 1;

[0033] Finally, based on the new similarity distribution , select the category with the highest similarity as the final model classification result.

[0034] The beneficial effects of the present invention are: The present invention combines the knowledge graph and the FCN8s-ResNet50 model to conduct four classification experiments, and the results show that:

[0035] 1) Fusing the mountain vegetation knowledge graph ontology in the model training and classification post-processing stages can significantly improve the classification accuracy of the deep learning model: The accuracy of the deep learning model constrained by the knowledge graph ontology has increased by 10%, and the overall classification accuracy of the vegetation type at the formation level has reached 89.2%.

[0036] 2) The deep learning model constrained by the mountain vegetation knowledge graph not only achieves excellent results in the fine classification of vegetation types, but also greatly improves the universality and interpretability of the model, which will provide strong support for a wider range of remote sensing vegetation classification and applications.

[0037] Fusing the knowledge graph with the deep learning model is of great significance for realizing the rapid extraction of vegetation information, the rapid update of vegetation maps, and the real-time monitoring of vegetation coverage. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is the general situation of the research area of the present invention.

[0039] Figure 2 is the technical flow chart of Experiment 2 of the present invention (the deep learning model fuses the knowledge graph ontology in the training stage).

[0040] Figure 3 is the technical flow chart of Experiment 3 of the present invention (the deep learning model fuses the knowledge graph ontology in the classification post-processing stage).

[0041] Figure 4 is the technical flow chart of Experiment 4 of the present invention (fusing the knowledge graph ontology in the training stage and the classification post-processing stage).

[0042] Figure 5 This is the confusion matrix of the classification model for the four comparative experiments of the present invention.

[0043] Figure 6 This is the training curve of the deep learning classification model in the four comparative experiments of the present invention.

[0044] Figure 7 This is the vegetation remote sensing classification result integrating the knowledge graph of the present invention: (a) is the 1:250,000 vegetation type map interpreted manually by visual inspection, (b) is the vegetation remote sensing classification result integrating the knowledge graph ontology, and (c) is the remote sensing image map of the study area. Detailed implementation manners

[0045] In order to enable those of ordinary skill in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0046] Referring to Figures 1 to 7 A vegetation remote sensing classification method constrained by a knowledge graph ontology shown below, comprising the following steps:

[0047] S1. Data acquisition and data preprocessing

[0048] 1) Obtain the digital surface model data of the study area, and generate a digital elevation model with the same resolution in combination with the spatial resampling technology;

[0049] 2) Obtain the multi-temporal remote sensing and digital elevation model data of the study area, unify the projection coordinate system and perform corresponding cropping according to the resolution, and preprocess the multi-temporal remote sensing and digital elevation model data to be consistent with the input data spatial range;

[0050] 3) Obtain the vegetation vertical zonation spectrum of the study area, and preprocess and transform the vegetation vertical zonation spectrum into a structured knowledge graph ontology reflecting the vegetation geographical distribution law; if there is already a vegetation vertical zone knowledge graph in the study area, it can be directly applied;

[0051] 4) Obtain the vegetation type map data and remote sensing images of the study area, and obtain the samples of each vegetation type in the study area according to the vegetation type map data and remote sensing images to form a sample data set of the vegetation types in the study area.

[0052] S2. Select a deep learning model

[0053] Select a deep learning model that can simultaneously learn the features of remote sensing images with different sources and different resolutions and has multiple channels.

[0054] S3. Fusion of the knowledge graph ontology and the deep learning model in the training stage

[0055] Input the data preprocessed in 1)-2) and 4) in step S1 into the deep learning model for training, and at the same time fuse the structured knowledge graph ontology with the deep learning model during the model training stage to obtain a training model for vegetation classification.

[0056] S4. Fuse the knowledge graph ontology with the deep learning model in the post-classification processing stage

[0057] Fuse the structured knowledge graph ontology with the deep learning model in the post-processing stage of vegetation classification to finally obtain the classification result of the vegetation.

[0058] The classification method of fusing the knowledge graph in the training stage of the present invention is as follows:

[0059] Divide the sample data set preprocessed in step S1 into a training set and a validation set, and use the deep learning model for training. At the same time, fuse the structured knowledge graph to finally obtain the classification result of the vegetation. During the training process of the deep learning model, incorporate the structured knowledge graph information. Use the deep learning model to predict a vegetation type, and compare the difference between the altitude range of the corresponding vertical zone in the knowledge graph ontology and the true altitude. Based on the loss function of interval overlap, quantify the matching degree between the altitude distribution range of the vegetation type in the knowledge graph ontology and the true altitude. The specific process is as follows:

[0060] and respectively represent the lower limit and the upper limit of the altitude interval of the th sample in the knowledge graph ontology, represents the true altitude of the th sample, then the form of the interval overlap loss is formula (5-1):

[0061]

[0062] where, is a small positive number used to prevent the denominator from being zero;

[0063] Add this constraint loss to the original cross-entropy loss to form a new composite loss function, as shown in formula (5-2):

[0064] (5-2)

[0065] where, is the cross-entropy loss of the classification task, is the weight coefficient for balancing the classification loss and the interval overlap loss.

[0066] The classification method of fusing the knowledge graph in the post-classification processing stage of the present invention is as follows:

[0067] First, use a deep learning model to classify the similarity of the prediction results of the deep learning model in step S3 to obtain the similarity values of each category , and then fuse the structured knowledge graph ontology to calculate the similarity to obtain a new predicted similarity distribution . According to the new similarity, obtain the final vegetation classification result. For the prediction results with similarity less than the threshold (this threshold can be adjusted according to the actual situation), use DEM to obtain the altitude of the corresponding location , and obtain the vertical distribution information of all vegetation types in the knowledge graph ontology, and then calculate the degree of proximity to the vertical zone distribution of each vegetation type, and convert it into a similarity distribution according to formulas (5-3) and (5-4) , where represents the possible value range, represents the th vegetation type's distribution range in the knowledge graph ontology, represents the number of vegetation types:

[0068]

[0069] (5-4)

[0070] Weightedly fuse the model-predicted similarity and the similarity calculated based on altitude to generate a new predicted similarity distribution , as shown in formula (5-5):

[0071]

[0072] Among them, and are the weights of the model-predicted similarity and the similarity calculated based on altitude respectively, and the sum of the two is 1;

[0073] Finally, based on the new similarity distribution , select the category with the highest similarity as the final model classification result.

[0074] Embodiment

[0075] 1. General situation of the study area

[0076] The study area is located in Taibai Mountain in the middle of the Qinling Mountains. Taibai Mountain is one of the national nature reserves and also the main peak of the Qinling-Daba Mountains ( Figure 1). Taibai Mountain, with an altitude of 3,767.2 meters, is a typical mountain in Central China. It has rich and diverse vegetation types and landscapes, making it an ideal research area for vegetation classification and mapping. To evaluate the universality of the model and avoid overfitting, the present invention selected training areas and test areas outside the research area (about 20 kilometers away from the research area) to construct a deep learning model ( Figure 1 as shown).

[0077] 2. Vegetation Vertical Zones in Taibai Mountain - Distribution Patterns of Mountain Vegetation

[0078] The climate in Taibai Mountain is affected by the Mongolian cold air mass in winter and the Pacific subtropical high pressure belt in summer, forming a transitional climate. As the altitude increases, the hydrothermal conditions change regularly, and five different climate zones are formed from bottom to top: warm temperate zone, temperate zone, cold temperate zone, frigid zone, and alpine frigid zone. The vegetation landscape also shows an obvious vertical zonation pattern, that is, the mountain vegetation vertical zones (MABs) corresponding to the climate zones. Due to the climate differences between the north and south slopes of Taibai Mountain, there are also obvious differences in the distribution of vegetation on the north and south slopes. The north slope is a typical warm temperate vertical zone spectrum with Quercus variabilis as the base belt (1,350 m), above which are the subzones of Quercus aliena var. acuteserrata (1,350 - 2,100 m), Quercus mongolica (2,100 - 2,400 m), Betula albo-sinensis (2,400 - 2,700 m), and Betula utilis var. jacquemontii (2,700 - 2,900 m). The base belt on the south slope is a deciduous broad-leaved oak forest dominated by Quercus acutissima, containing evergreen broad-leaved tree species such as Cyclobalanopsis myrsinifolia and Lithocarpus saxicola (800 m), above which are the subzones of Quercus variabilis (800 - 1,400 m), Quercus aliena var. acuteserrata (1,400 - 2,000 m), Betula albo-sinensis (2,000 - 2,600 m), and Betula utilis var. jacquemontii (2,600 - 2,800 m). Above the deciduous broad-leaved forest belt are the coniferous forest belt (2,800 - 3,450 m) and the shrub and meadow belt (above 3,450 m), and the coniferous forest belt is divided into the subzone of Abies fargesii (2,800 - 3,100 m) and the subzone of Larix chinensis (above 3,100 m) from bottom to top.

[0079] 3. Data and Data Preprocessing

[0080] (1)Remote sensing data: The multi-source and multi-temporal remote sensing data used in the present invention include ZY3 and GF2 satellite data (winter, 2m resolution), and GF1 satellite data (summer and winter, 16m resolution). These multi-temporal and multi-spatial resolution remote sensing data enable the deep learning model to capture the seasonal variation characteristics of vegetation. The digital elevation model (DEM) data with a resolution of 10 meters is derived from the digital surface model (DSM) product of ZY3 satellite data, providing elevation information for vegetation types. For the convenience of data input and standardized processing, the 2m resolution image is cropped into 224×224 pixel data, and the 16m resolution remote sensing image and DEM are cropped into 1 / 8 of the size of the 2m image (28×28 pixel data) to ensure the consistency of the spatial range of input data with different resolutions.

[0081] (2)Taibai Mountain mountain vegetation vertical zone knowledge graph. As mentioned above, the vertical zone distribution law of vegetation on the north and south slopes of Taibai Mountain is important information for vegetation remote sensing classification, which helps to improve the accuracy and interpretability of classification. However, the traditional vegetation vertical zone spectrum exists in the form of unstructured text or charts, making it difficult to be used by structured models or processed by computers. Therefore, this application adopts the conversion into a structured knowledge graph. The Taibai Mountain vegetation vertical zone knowledge graph is constructed with the support of ArcGIS and deep learning methods, combining vegetation distribution laws, maps, and remote sensing images, etc., to define and describe the composition, characteristics, and relationships of Taibai Mountain vegetation, reflecting the geographical distribution pattern of vegetation in this area, so as to promote the application of knowledge such as vegetation distribution laws in various disciplines. The ontology of this vegetation knowledge graph has a four-level classification system: L0 level, natural or cultivated vegetation, non-vegetation; L1 level, vegetation type group; L2 level, vegetation type or subtype; L3 level, vegetation formation group, formation, and sub-formation. The Taibai Mountain vegetation vertical zone knowledge graph is constructed and stored using the Neo4j ontology editing tool and described using RDF and Web Ontology Language (OWL).

[0082] (3)Vegetation type map data. The vegetation type map data used in the present invention includes the data of "1:250,000 Vegetation Type Map of Qinling-Bashan Mountains" and "1:1,000,000 Vegetation Type Map of China". Supported by the project of "Comprehensive Scientific Investigation of the North-South Transition Zone in China", the present invention uses multi-source and multi-temporal high-resolution image data such as GF-1 and GF-2, as well as relevant field investigation sample points and literature materials. Based on the classification system of the original 1:1,000,000 vegetation type map, following the process of vegetation major category → vegetation type group interpretation → vegetation formation group, formation, and sub-formation interpretation → vegetation type and vegetation subtype interpretation, a 1:250,000 vegetation type map covering the Qinling-Bashan Mountains is compiled by visual interpretation using a classification method combining top-down and bottom-up. The "1:1,000,000 Vegetation Type Map of China" is obtained through scanning and digitization. It is the most detailed and accurate national vegetation type map to date and can also provide assistance for the research of the present invention.

[0083] (4) Digital Surface Model (DSM) data. The 1:10,000 DSM data (resolution 10m) of Taibai Mountain was generated based on ZY-3 satellite images, and was used to extract fine-scale slope, aspect, elevation, and land cover data. In addition, spatial resampling technology was also used to make the DEM data consistent with the 16-meter resolution remote sensing data, and the DEM data was also cropped into 28×28 pixel data.

[0084] 4. Classification method

[0085] (1) Deep learning model

[0086] The present invention adopts an improved multi-channel FCN8s-ResNet50 deep learning method. The advantage of this method is that the model is modified into a four-channel structure to learn and extract features from multi-source remote sensing data: the 2-meter resolution GF2 or ZY3 images in winter, the 16-meter resolution GF1 images in winter and summer, and the resampled 16-meter resolution DEM data are respectively input into the above 4 channels. Since the above four-channel data have different resolutions and contain different image features, the ResNet50 model is used to train ImageNet to separately extract features from each channel, rather than simply stacking different data bands together. In addition, compared with other deep network models, ResNet50 has the advantages of fewer required parameters and less computing resources while ensuring accuracy, so it is more suitable for the present invention.

[0087] The present invention selects 24,000 samples from the Taibai Mountain area, 3,000 for each of the 8 pure forest vegetation types. The preprocessed dataset is divided into a training set and a validation set, with 15,000 samples for training and 9,000 samples for validating the model performance.

[0088] (2) Triple transformation of the ontology of the Taibai Mountain vegetation vertical zone knowledge graph

[0089] Use Python to export the ontology of the Taibai Mountain vegetation vertical zone knowledge graph from Neo4j, represent it as a relational triple table and an attribute triple table (Table 1), and store it as CSV data. The relational triple is represented as <head node - relationship - tail node>, such as <923 - belongs to - 922>; the attribute triple is represented as <node - attribute - value>, such as <923 - upper limit - 1300 meters> (922 and 923 are the IDs of the ontology nodes). By structuring the representation of the triples of the vegetation knowledge graph ontology, the vertical distribution patterns of various vegetation types in Taibai Mountain can be introduced into the deep learning model.

[0090] Table 1 Example of triples of the mountain vegetation knowledge graph ontology

[0091]

[0092] (3)Integration of Knowledge Graph Ontology and Deep Learning Model

[0093] In order to study the impact of the integration of knowledge graph ontology and deep learning model at different stages of the remote sensing classification process on the classification results, four groups of comparative experiments were set up in this invention for analysis, as shown in Table 2. The Adam optimizer was used for model training, the loss function was set as cross-entropy loss, the batch size was set as 64, the initial learning rate was 1e-5, and the learning rate decayed exponentially with the number of training rounds. The number of training rounds was 150 to balance the convergence speed and generalization ability of the model.

[0094] Table 2 Comparative Experiments of Pure Forest Deep Learning Model Integrating Knowledge Graph Ontology

[0095]

[0096] Experiment 1: Classification Model without Knowledge Graph Integration

[0097] In this experiment, a vegetation classification method based on traditional remote sensing image features was adopted, and its core algorithm framework was a ResNet50 deep neural network model pre-trained on the large-scale ImageNet dataset.

[0098] Experiment 2: Classification Model Integrating Knowledge Graph during Model Training Phase

[0099] This experiment adopted the method of integrating knowledge graph ontology during the training phase, aiming to integrate the advantages of deep learning models and knowledge graphs to improve the accuracy of the model in mountain vegetation classification. The specific implementation process is shown in Figure 2 .

[0100] The detailed implementation steps are as follows:

[0101] First, construct a knowledge graph of mountain vegetation vertical zones, and represent and export it in a triple form (entity-relationship-entity) for structured representation. Use programming languages such as Python to parse and integrate this knowledge information. During the training process of the deep learning model (such as the improved ResNet50), integrate the knowledge graph information into it. For each prediction result of the model, query the knowledge graph in real time to obtain the vertical distribution band range corresponding to the predicted vegetation type. At the same time, according to the actual sample labels, retrieve the altitude range where the corresponding vegetation type is located from the knowledge graph. Then design and calculate the loss under constraints, that is, compare the difference between the vertical distribution range of the vegetation type predicted by the model and the true altitude label.

[0102] The model prediction will obtain a vegetation type, and compare the altitude range of its corresponding vertical zone in the ontology With its true elevation . To quantify the matching degree between the prediction interval and the true elevation, a loss function based on interval overlap is designed as follows:

[0103] and represent the lower and upper bounds of the elevation interval of the -th sample in the knowledge graph ontology, respectively. represents the true elevation of the -th sample. Then a simple form of the interval overlap loss can be designed as formula (5-1):

[0104]

[0105] where is a small positive number used to prevent the denominator from being zero. This loss function measures the proportion of the non-overlapping part between the ontology distribution interval of the prediction type and the true elevation in the ontology distribution interval of the prediction type. When the true elevation is within the distribution range interval of the ontology type, the loss is 0, otherwise the loss increases as it moves away from the interval.

[0106] Adding this constraint loss to the original cross-entropy loss forms a new composite loss function. The model parameters are updated using the backpropagation algorithm to optimize the model so that it can follow the prior knowledge in the knowledge graph while learning image features, achieving more accurate and reasonable mountain vegetation classification.

[0107] The composite loss function is calculated as formula (5-2):

[0108] (5-2)

[0109] where is the cross-entropy loss of the classification task, and is the weight coefficient that balances the classification loss and the interval overlap loss.

[0110] Experiment 3: Classification Model Incorporating Knowledge Graph Ontology in the Post-Processing Stage

[0111] In Experiment 2, a neural network classification model with ontology constraints in the training stage was established for vegetation classification. However, different embedding methods and positions of the knowledge graph may have different effects and optimization results on the neural network model. Therefore, in this experiment, the knowledge graph ontology ( Figure 3 shown) is incorporated in the post-processing of the classification results.

[0112] The specific steps for incorporating the knowledge graph ontology in the post-processing of the classification are as follows:

[0113] First, use a deep learning model to classify remote sensing images and output the similarity of each category, denoted as . Then, set a threshold (e.g., 0.5) in the model. For all categories with predicted similarity lower than this threshold, it is considered that the model's prediction is not certain enough.

[0114] For the prediction results with similarity lower than the threshold, obtain the corresponding altitude at this location , and find all the vertical distribution information of vegetation types in the knowledge graph. Then calculate the degree of closeness to the vertical zone distribution of each vegetation type, and convert it into a similarity distribution according to formulas (5-3) and (5-4) , where represents the possible value range, represents the th vegetation type's distribution range in the knowledge graph ontology, represents the number of vegetation types.

[0115]

[0116] (5-4)

[0117] Weightedly fuse the model predicted similarity and the similarity calculated based on altitude to generate a new predicted similarity distribution , as shown in formula (5-5), where and are the weights of the model predicted similarity and the similarity calculated based on altitude respectively, and the sum of the two is 1;

[0118]

[0119] Finally, based on the new similarity , select the category with the highest similarity as the final model prediction result. This method also utilizes the domain expert knowledge of the knowledge graph, which helps to improve the accuracy of vegetation type classification in complex environments.

[0120] Experiment 4: Classification model that fuses the knowledge graph in both the model training stage and the post-classification processing stage

[0121] Fuse the optimization methods for the neural network model in Experiment 2 and Experiment 3, and fuse the knowledge graph ontology in both the model training stage and the post-classification processing stage to verify the final optimization effect of the knowledge graph ontology on the model. The technical process implemented in Experiment 4 is as Figure 4 shown.

[0122] Accuracy Evaluation

[0123] When comparing and analyzing the model accuracies of 4 experiments, according to the confusion matrix, the accuracy rate, recall rate, F1-Score, and overall accuracy are calculated to evaluate the performance of the model on the validation set. The final accuracy evaluation of the vegetation remote sensing classification uses random point sampling verification. The specific method is as follows: Conduct 10 random samplings in the vegetation classification results, with 100 points sampled each time, and remove the invalid points taken on the boundary of the study area; obtain the vegetation types of the valid points in the "1:250,000 Vegetation Type Map of the Qinling-Bashan Mountains" and the predicted vegetation type map respectively to obtain the sampling results; then calculate the accuracy statistical indicators. In this invention, the overall accuracy is used to evaluate the quality of the classification results. The overall accuracy is the proportion of the number of correctly classified samples to the total number of samples. Its calculation formula is:

[0124] Overall accuracy = (Number of correctly classified samples / Total number of samples) × 100%.

[0125] Research Results

[0126] 1. Prediction Results and Accuracy of Experiment 1

[0127] Table 3 Accuracy Evaluation of the Training Model without Fusing the Knowledge Graph Ontology

[0128]

[0129] From Figure 5 As can be seen from (a) in the figure and Table 3, Experiment 1, as the benchmark experiment, has an overall accuracy of approximately 70.6%. The accuracy rates of different categories are approximately between 0.63 and 0.83. Among them, the accuracy rates of shrubs and meadows are relatively high, both being 0.83, while the accuracy rate of Larix chinensis forest is the lowest, being 0.63; the recall rates of different categories are between 0.59 and 0.82. Among them, the recall rate of Pinus tabuliformis forest is the highest, being 0.82, while the recall rate of shrubs is the lowest, being 0.67; the F1 scores of different categories are between 0.61 and 0.79. Among them, the F1 score of meadows is the highest, being 0.79, while the F1 score of Pinus armandii forest is the lowest, being 0.61.

[0130] 2. Prediction Results and Accuracy of Experiment 2

[0131] As Figure 5 As shown in (b) in the figure and Table 4, the overall accuracy of Experiment 2 is 75.3%, which is about 4.7% higher than that of the benchmark experiment. Generally speaking, the performance of this group of models has been improved in most categories, and the overall average accuracy rate, recall rate, and F1 score have all increased by about 4% - 5%. The accuracy rates of Quercus aliena var. acuteserrata, Larix chinensis forest, and Pinus armandii forest, which had relatively poor classification results in Experiment 1, have all increased significantly. The accuracy rates and recall rates of Quercus aliena var. acuteserrata forest and Pinus armandii forest have both increased by nearly 10%, and the F1 score has also increased significantly.

[0132] Table 4 Classification Model Accuracy Evaluation of Integrating Knowledge Graph in the Training Stage

[0133]

[0134] 3. Prediction Results and Accuracy of Experiment 3

[0135] As Figure 5 shown in (c) and Table 5, the overall accuracy of Experiment 3 is approximately 76.1%, which is about 5.5% higher than that of the baseline experiment. As can be seen from Table 5, the accuracy of Quercus variabilis forest has increased significantly, by about 10%. Vegetation types with poor baseline results such as Larix chinensis forest and Abies fargesii forest have also improved significantly. In addition, the recall rate of shrubs has increased significantly, but the accuracy has decreased slightly, probably because the model sacrifices a certain classification accuracy while ensuring that more shrub samples are detected.

[0136] Table 5 Classification Model Accuracy Evaluation of Integrating Knowledge Graph in the Classification Post-Processing Stage

[0137]

[0138] 4. Prediction Results and Accuracy of Experiment 4

[0139] Table 6 Classification Model Accuracy Evaluation of Integrating Knowledge Graph Ontology in Both the Training and Classification Post-Processing Stages

[0140]

[0141] As Figure 5 shown in (d) and Table 6, the overall accuracy of Experiment 4 is approximately 81.5%, which is about 10.9% higher than that of the baseline experiment, showing a significant improvement. Except for shrubs and meadows, the accuracy and recall rate of the remaining vegetation types have all increased to varying degrees, showing stronger classification ability overall. And from the confusion matrix, it can be seen that the confusion phenomenon has significantly decreased. Although the accuracy of the two types of shrubs and meadows has decreased slightly, the recall rate has increased significantly, from 0.67 and 0.75 to 0.88 respectively, indicating that the model also has better performance in the recognition of this class.

[0142] Comparative Analysis of Prediction Results and Accuracy of Four Groups of Experiments

[0143] The training processes of the four groups of experiments are as Figure 6 shown. By comparing the results of the four experiments (Table 7), the following conclusions can be drawn:

[0144] Experiment 1 showed significant differences in vegetation type recognition. The classification results of shrubland and meadow were better due to their unique remote sensing image features. However, the classification results of Quercus aliena var. acuteserrata forest, Larix chinensis forest, and Pinus armandii forest were not good because their distribution areas were close to those of other vegetation types and their image features were similar. The overall accuracy of this model was 70.64%.

[0145] After integrating the knowledge graph ontology in the model training stage of Experiment 2, the overall accuracy was significantly improved. Especially for types with poor classification results such as Quercus aliena var. acuteserrata forest, Larix chinensis forest, and Pinus armandii forest, introducing ontology knowledge helped the model understand and distinguish these vegetation types with similar image features and close distributions. When the remote sensing image features were similar, more accurate judgments could be made by relying on the different altitudes at which different vegetation types were distributed in the vertical zonation spectrum. The overall accuracy of this model was 75.25%.

[0146] After integrating the knowledge graph ontology in the classification post - processing stage of Experiment 3, the accuracy of some categories such as Quercus variabilis forest was significantly improved. This might be because Quercus variabilis forest was at the bottom layer of the vertical zonation spectrum. When classification errors occurred and there were obvious differences from the distribution rules of the vertical zonation spectrum, the knowledge graph ontology could be used to identify and correct them in the post - processing stage. The overall accuracy of this model was 76.08%.

[0147] After integrating the knowledge graph ontology in both the training stage and the classification post - processing stage of Experiment 4, the overall accuracy was greatly improved, and the recognition effects of almost all vegetation types were significantly improved. This proved that integrating the knowledge graph ontology with the deep learning model in the model training and decision - making processes could greatly reduce misclassifications caused by similar remote sensing image features, effectively utilize geographical knowledge such as the vertical zonation spectrum to solve the problem of vegetation type recognition in complex environments, and improve the interpretability and universality of the model. The overall accuracy of this model was 81.51%.

[0148] Generally speaking, the application of the knowledge graph ontology in different stages (model training and classification post - processing) of the vegetation type classification task could significantly improve performance, especially in solving the confusion problem between vegetation types with similar image features. This was because the knowledge graph could provide rich semantic and context information, making up for the deficiency of relying solely on remote sensing image features for classification, thus helping the model better understand and distinguish different vegetation types.

[0149] Table 7 Prediction Results and Accuracy Evaluation of Four Groups of Experiments

[0150]

[0151] Remote Sensing Classification Results of Vegetation Types in Taibai Mountain Based on Experiment 4

[0152] To further verify the effectiveness of the deep learning model with knowledge graph ontology constraints in vegetation classification, the deep learning model in Experiment 4 was used for remote sensing classification of vegetation in the Taibai Mountain area. The classification results are as follows Figure 7 shown in (b) of Figure 7 ((a) in

[0153] is the 1:250,000 vegetation type map interpreted manually; (b) is the remote sensing classification result of vegetation integrating the knowledge graph ontology; (c) is the remote sensing image map of the study area). Through random point verification of 10 random samplings (100 points each time), 22 invalid points taken on the boundary of the study area were removed, and the valid results were 978. Among them, 872 results were predicted correctly, and the overall accuracy of the final vegetation classification result in the Taibai Mountain study area was 89.2%.

[0154] The principle of the present invention is that the present invention integrates the mountain vegetation knowledge graph ontology with the FCN8s-ResNet50 model to achieve high-precision classification at the vegetation formation level. Under the constraints of the knowledge graph ontology, the accuracy of the deep learning model has increased by 10%, and the overall classification accuracy of vegetation types has reached 89.2%. At the same time, the model established in the training area also achieved high accuracy in the classification of the experimental area, indicating that the deep learning model with knowledge graph ontology constraints has good universality, and further verifying the strong potential and practical value of the knowledge graph in the classification of vegetation remote sensing images.

[0155] The above shows and describes the basic principles, main features and advantages of the present invention. The present invention will also have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed.

Claims

1. A vegetation remote sensing classification method constrained by knowledge graph ontology, characterized in that: The following steps are involved: S1. Data acquisition and data preprocessing 1) Obtain digital surface model data of the study area and generate a digital elevation model with the same resolution by combining spatial resampling technology; 2) Obtain multi-temporal remote sensing and digital elevation model data of the study area, unify the projection coordinate system and crop accordingly according to the resolution, and pre-process the multi-temporal remote sensing and digital elevation model data to be consistent with the spatial range of the input data; 3) Obtain the vertical band spectrum of vegetation in the study area, and pre-process the vertical band spectrum of vegetation into a structured knowledge graph ontology that reflects the geographical distribution law of vegetation; 4) Obtain vegetation type map data and remote sensing images of the study area, obtain samples of each vegetation type in the study area based on the vegetation type map data and remote sensing images, and form a sample data set of the study vegetation type; S2. Select a deep learning model Select a deep learning model that can simultaneously learn the features of remote sensing images from different sources and resolutions and has multiple channels; S3. Integration of knowledge graph ontology and deep learning model in the training phase The data pre-processed in steps 1)-2) and 4) in step S1 are input into the deep learning model for training, and the structured knowledge graph ontology and the deep learning model are integrated in the model training stage to obtain a training model for vegetation classification; In the process of deep learning model training, structured knowledge graph information is integrated, a vegetation type is predicted by the deep learning model, and the difference between the altitude range of the corresponding vertical zone in the knowledge graph ontology and the actual altitude is compared; Based on the interval overlap loss function, the matching degree between the altitude distribution range of vegetation types in the knowledge graph ontology and the actual altitude is quantified. The specific process is as follows: and Respectively represent The lower and upper limits of the altitude range of the samples in the knowledge graph ontology, Indicates The true altitude of samples, then the interval overlap loss is in the form of formula (5-1): ; in, is a small positive number used to prevent the denominator from being zero; Add this constraint loss to the original cross entropy loss to form a new composite loss function, as shown in formula (5-2): （5-2） in, is the cross entropy loss for classification tasks, is the weight coefficient that balances the classification loss and interval overlap loss; S4. Fusion of knowledge graph ontology and deep learning model in post-classification processing The structured knowledge graph ontology and the deep learning model are integrated in the post-processing stage of vegetation classification to finally obtain the vegetation classification results.

2. According to claim 1, a vegetation remote sensing classification method constrained by knowledge graph ontology is characterized by: The sample data set preprocessed in step S1 is divided into a training set and a validation set, and trained using a deep learning model while integrating a structured knowledge graph to finally obtain the vegetation classification result.

3. According to claim 2, a vegetation remote sensing classification method constrained by knowledge graph ontology is characterized by: Use the deep learning model to first perform similarity classification on the prediction results of the deep learning model in step S3 to obtain the similarity value of each category , and then integrate the structured knowledge graph ontology to calculate the similarity and obtain a new predicted similarity distribution , and obtain the final vegetation classification result based on the new similarity.

4. According to claim 3, a vegetation remote sensing classification method constrained by knowledge graph ontology is characterized by: For similarity For prediction results that are less than the threshold, use DEM to obtain the altitude of the corresponding location , and obtain the vertical distribution information of all vegetation types in the knowledge graph ontology, and then calculate The closeness to the vertical distribution of each vegetation type is converted into a similarity distribution according to formulas (5-3) and (5-4): ,in represent The possible range of values ​​is Representative The distribution range of vegetation types in the knowledge graph ontology, Number of representative vegetation types: ; (5-4) The model predicts similarity Similarity to that calculated based on altitude Perform weighted fusion to generate a new predicted similarity distribution , as shown in formula (5-5): ; in, and The model predicts similarity Similarity to that calculated based on altitude The weight of , the sum of the two is 1; Finally, based on the new similarity distribution , select the category with the highest similarity as the final model classification result.

Images