Remote sensing image land cover classification method and system based on spatiotemporal knowledge graph

Abstract

This invention discloses a remote sensing image land cover classification method and system based on a spatiotemporal knowledge graph. The method first constructs a spatiotemporal knowledge graph. Then, based on the geographic coordinates and imaging time metadata of the remote sensing image, the constructed spatiotemporal knowledge graph is queried to obtain the corresponding natural prior knowledge. The remote sensing image and natural prior knowledge are combined as input data for a deep semantic segmentation network to obtain the remote sensing image land cover category. Finally, the expert prior inference rules and scene prior inference rules in the spatiotemporal knowledge graph are used to correct the classification results output by the deep semantic segmentation network. This method can effectively improve the accuracy and interpretability of remote sensing image land cover classification.

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary technical field of remote sensing interpretation and artificial intelligence, and specifically relates to a remote sensing image land cover classification method and system based on spatiotemporal knowledge graph. Background Art

[0002] With the continuous advancement of remote sensing earth observation technology, the spatial resolution of remote sensing images has continued to improve. Optical high-resolution remote sensing images, due to their rich texture and spatial structure information and increasingly convenient acquisition methods, have gradually become an important data support for the interpretation of remote sensing ground object information. As a fundamental work in remote sensing information interpretation, high-resolution remote sensing image ground object classification has received widespread attention from scientific researchers and technicians, and has important application value in transportation, agricultural and forestry protection, environmental assessment, disaster relief, national defense and military fields. Although high-resolution remote sensing images help to display more details of objects, the "same spectrum, different objects, same object, different spectrum" phenomenon of high-resolution remote sensing image data is significant, which poses a huge challenge to remote sensing image classification.

[0003] Methods for classifying objects in high-resolution remote sensing images can be broadly divided into two categories: traditional methods based on manually designed features and shallow classifiers, and deep semantic segmentation network methods that integrate features and classification. Traditional classification methods generally first extract spectral, textural, and structural features from remote sensing images using expert-defined feature descriptors, and then perform feature classification. (Feature classification methods include supervised classification methods based on shallow feature discrimination, such as maximum likelihood, decision trees, and support vector machines.) Deep semantic segmentation network methods that integrate features and classification include fully convolutional networks (FCNs), segmentation networks (SegNets), U-nets (U-Nets), deep segmentation networks (DeepLabs), and mask region convolutional networks (Mask R-CNNs). Traditional methods generally require manually designed feature descriptors, making it difficult to bridge the semantic gap between low-level image data and high-level logical information, resulting in poor classification accuracy. Methods based on deep semantic segmentation networks automatically learn feature extraction and classification models from data through an end-to-end learning mechanism, thereby adaptively extracting features from remote sensing images and performing classification. Given sufficient labeled training samples, deep semantic segmentation networks often achieve relatively good classification accuracy. However, deep semantic segmentation networks are data-driven methods that rely heavily on data learning and lack human-like cognitive capabilities. They are unable to fully utilize the rich logical information between objects (such as spatial relationships) for self-correction. Consequently, their results are poorly interpretable (the black-box nature of deep networks remains prominent) and their reliability needs to be improved (weak noise interference often causes deep networks to produce inexplicable prediction errors). Furthermore, deep semantic segmentation networks fail to fully utilize the rich spatiotemporal prior knowledge in remote sensing, including data-based knowledge such as slope and aspect, land cover products, quantitative remote sensing products, and multi-source geographic data, as well as regularity-based knowledge such as interpretation common sense and object distribution. To effectively improve the accuracy of remote sensing image interpretation, a spatiotemporal knowledge graph is constructed to uniformly manage, access, and analyze multiple spatiotemporal prior knowledge. This approach extracts data-based knowledge from multi-dimensional remote sensing data organized in space and time, while also building rich regularity-based knowledge through expert experience. By coupling this graphed spatiotemporal data and knowledge with deep learning methods, a remote sensing image land cover classification method based on a spatiotemporal knowledge graph is developed.

[0004] Based on the above analysis, the remote sensing image land cover classification method based on spatiotemporal knowledge graph is the only way to achieve truly intelligent remote sensing land feature information interpretation. Summary of the Invention

[0005] In response to the problems of low classification accuracy and poor interpretability in the existing technology of land cover classification of high-resolution remote sensing images, the present invention provides a remote sensing image land cover classification method and system based on spatiotemporal knowledge graph, which can effectively improve the accuracy and interpretability of high-resolution remote sensing image land cover classification.

[0006] To achieve the above objectives, the present invention provides a method for land cover classification based on remote sensing images using a spatiotemporal knowledge graph, comprising the following steps:

[0007] Step 1: Construct a spatiotemporal knowledge graph, including offline storage of natural prior knowledge and scenario prior knowledge in the spatiotemporal knowledge graph, and establishing rules for applying natural prior knowledge, inference rules for expert prior knowledge, and inference rules for scenario prior knowledge.

[0008] Step 1.1: Store natural prior knowledge offline in the spatiotemporal knowledge graph and establish rules for applying natural prior knowledge to facilitate query and call in subsequent steps;

[0009] Step 1.2: Establish inference rules based on expert prior knowledge offline in the spatiotemporal knowledge graph.

[0010] Step 1.3: Store the scene prior knowledge offline in the spatiotemporal knowledge graph and establish scene prior reasoning rules to facilitate query calls in subsequent steps;

[0011] Step 2: Based on the geographic coordinates and imaging time metadata of the remote sensing image, the spatiotemporal knowledge graph constructed in step 1 is queried to obtain the corresponding natural prior knowledge. The remote sensing image and natural prior knowledge are combined as input data for the deep semantic segmentation network. Based on the loss function, the deep semantic segmentation network is optimized through the backpropagation algorithm. The trained deep semantic segmentation network is used to classify the land cover of the remote sensing image, and the classification result map as well as the predicted category and classification confidence of each pixel are obtained.

[0012] Step 3: Use the connected domain segmentation method to segment the classification result graph obtained in step 2 to obtain a segmentation graph, and use the connected domain in the segmentation graph as the expert prior reasoning unit;

[0013] Step 4: Referring to the land cover categories defined in the inference rules of the expert prior knowledge, each expert prior reasoning unit is instantiated according to its corresponding geographical scope and land cover category, and all the generated expert prior reasoning unit instances are constructed into the spatiotemporal knowledge graph;

[0014] Step 5: Using the expert prior reasoning unit instance as the reasoning unit, according to the expert prior reasoning rules, find the misclassified categories in the classification result graph in step 2 and correct them, thereby obtaining the expert prior reasoning result graph;

[0015] Step 6: Based on the metadata information such as the geographic coordinates and imaging time of the remote sensing image, query the spatiotemporal knowledge graph to obtain the corresponding scene prior knowledge, and further correct the expert prior reasoning result graph according to the scene prior reasoning rules constructed in the spatiotemporal knowledge graph to obtain the final comprehensive reasoning result graph.

[0016] Furthermore, the natural prior knowledge in step 1.1 includes slope and aspect. A mapping relationship between the digital elevation model file path for a certain imaging time, geographic range, and projection system is established in the spatiotemporal knowledge graph, and an algorithm for calculating slope and aspect from the digital elevation model is established. Slope is defined as the angle between the plumb line and the normal. The slope α at pixel (x, y) is calculated as follows:

[0017]

[0018] Where, f x and f y are the gradients in the x and y directions at pixel (x,y), respectively.

[0019] The slope aspect is defined as the angle between the projection of the normal on the horizontal plane and the north direction, measured clockwise. The slope aspect A at the pixel (x, y) is calculated as follows:

[0020]

[0021] Where, f x and f y are the gradients in the x and y directions at pixel (x,y), respectively.

[0022] The rule of applying natural prior knowledge refers to querying the spatiotemporal knowledge graph based on the metadata of remote sensing images to obtain slope and aspect.

[0023] Furthermore, in step 1.2, a remote sensing interpretation ontology is established and described using the semantic web rule language (SWRL). A root land cover category is defined, whose subcategories include inference unit classes, buildings, infrastructure, industrial and mining land, urban green space, cultivated land, gardens, pastures, forests, shrubs, bare land, and water bodies. The inference unit class includes correctly classified units and incorrectly classified units. Core attributes are defined, including the spatial relationship attributes of adjacency and encirclement, and the statistical attribute of adjacent edge length ratio. Inference rules based on expert prior knowledge are created offline and symbolically represented using the semantic web rule language. These inference rules are used to correct land cover classification results. These include: Rule 1: Forests whose adjacent length to buildings, industrial and mining land, or urban green space accounts for more than half of the total adjacent edge length are corrected to urban green space; Rule 2: Bare land adjacent to industrial and mining land is corrected to industrial and mining land; Rule 3: Water bodies surrounded by buildings, infrastructure, or industrial and mining land are corrected to buildings, infrastructure, or industrial and mining land of the corresponding category; and Rule 4: Bare land surrounded by water is corrected to water bodies.

[0024] Moreover, in step 1.3, the Open Street Map (OSM) land cover data is stored in the spatiotemporal knowledge graph as scene prior knowledge, and scene prior inference rules are created offline to correct the land cover classification results based on the scene prior knowledge, including: Rule 1 modifies pixels belonging to industrial and mining land in the OSM data to industrial and mining land, Rule 2 modifies pixels belonging to industrial land, commercial land, and retail land in the OSM data to infrastructure, Rule 3 modifies pixels belonging to roads in the OSM data to infrastructure, Rule 4 modifies pixels belonging to transportation facilities in the OSM data to infrastructure, and Rule 5 modifies wetlands and gardens belonging to water bodies in the OSM data to water bodies.

[0025] Moreover, in step 2, the spatiotemporal knowledge graph is queried according to the geographic coordinates and imaging time of the remote sensing image I to obtain the corresponding natural prior knowledge E, and the remote sensing image I and the natural prior knowledge E are combined as the input data of the deep semantic segmentation network HRNet. Based on the classic cross-entropy loss function, the HRNet network is optimized through the backpropagation algorithm adaptive moment estimation. The HRNet network contains 4 types of resolution feature map information. Each row of feature map sequence corresponds to a branch, and the feature map is divided into 4 stages in the column direction: stage 1 consists of 1 layer, which is composed of 4 bottleneck modules in series; stage 2 contains two branches, each branch contains 4 basic modules; stage 3 contains 3 branches, each branch contains 4 basic modules, and this stage is repeated 4 times in the entire network; stage 4 contains 4 branches, each branch contains 4 basic modules, and this stage is repeated 3 times in the entire network; the bottleneck module compresses the feature channel through 1×1 convolution blocks to reduce computational costs, and uses skip connections in the residual network to ensure that the resolution of the feature map remains unchanged during feature transfer; the basic module uses 3×3 convolution blocks to extract features and transfers feature information through skip connections, so that the resolution remains unchanged during feature transfer. The HRNet network is trained using the remote sensing image land cover classification training set data. The training set data categories include buildings, infrastructure, industrial and mining land, urban green space, cultivated land, gardens, pastures, forests, shrubs, bare land, wetlands and water bodies; the remote sensing images are classified into land cover using the trained deep semantic segmentation network to obtain the classification result map R as well as the predicted category C and classification confidence D of each pixel point P.

[0026] Moreover, in step 3, the connected domain segmentation method is used to segment the classification result graph R into connected domains to obtain a segmentation graph G. The segmentation graph G contains K connected domains S, and the connected domain S is used as the expert prior reasoning unit S′, that is:

[0027] G={S1,S2,…,S i ,…|S i =ConnectedComponents(R),1≤i <K} (3)

[0028] Where S i represents the i-th connected component, and ConnectedComponents(·) represents the division of the image according to the connected components.

[0029] The present invention also provides a remote sensing image land cover classification system based on a spatiotemporal knowledge graph, which is used to implement the remote sensing image land cover classification method based on a spatiotemporal knowledge graph as described above.

[0030] Moreover, it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the above-mentioned remote sensing image land cover classification method based on spatiotemporal knowledge graph.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] Existing remote sensing image land cover classification methods based on deep learning have poor interpretability, and the model performance is heavily dependent on the number of trained samples. This invention, by constructing a spatiotemporal knowledge graph and introducing semantic relationship information and reasoning capabilities, can further enhance the interpretability of the model and improve the credibility of the classification results based on the deep learning model remote sensing image land cover classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an overall flow chart of an embodiment of the present invention (it should be noted that due to the limitation of display space, the knowledge reasoning results are only displayed using expert prior reasoning results).

[0034] Figure 2 This is a structural diagram of the deep semantic segmentation network according to an embodiment of the present invention.

[0035] Figure 3 Schematic diagram of land cover classification results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention provides a remote sensing image land cover classification method and system based on spatiotemporal knowledge graph. The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0037] Example 1

[0038] like Figure 1 As shown, the present invention provides a remote sensing image land cover classification method based on spatiotemporal knowledge graph, comprising the following steps:

[0039] Step 1: Construct a spatiotemporal knowledge graph, including offline storage of natural prior knowledge and scenario prior knowledge in the spatiotemporal knowledge graph, and establish rules for the application of natural prior knowledge, reasoning rules for expert prior knowledge, and scenario prior reasoning rules.

[0040] Step 1.1: Store natural prior knowledge offline in the spatiotemporal knowledge graph and establish rules for applying natural prior knowledge to facilitate query and call in subsequent steps.

[0041] Natural prior knowledge includes slope and aspect. A mapping relationship between the digital elevation model file path for a certain imaging time, geographic range, and projection system is established in the spatiotemporal knowledge graph, and an algorithm for calculating slope and aspect from the digital elevation model is established. Slope is defined as the angle between the plumb line and the normal. The slope α at a pixel (x, y) is calculated as follows:

[0042]

[0043] Where, f x and f y are the gradients in the x and y directions at pixel (x,y), respectively.

[0044] The slope aspect is defined as the angle between the projection of the normal on the horizontal plane and the north direction (measured clockwise). The slope aspect A at the pixel (x, y) is calculated as follows:

[0045]

[0046] Where, f x and f y are the gradients in the x and y directions at pixel (x,y), respectively.

[0047] The rules for applying natural prior knowledge refer to obtaining natural prior knowledge (slope and aspect) by querying the spatiotemporal knowledge graph based on the metadata of remote sensing images.

[0048] Step 1.2: Establish inference rules based on expert prior knowledge offline in the spatiotemporal knowledge graph.

[0049] A remote sensing interpretation ontology was established, using the Semantic Web Rule Language (SWRL) to describe the ontology. The root class, land cover category (rs:geoClass), was defined, and its subclasses included the inference unit class (rs:Segment), building (rs:Building), infrastructure (rs:Infrastructure), industrial and mining land (rs:Building), urban green space (rs:Urban green space), cultivated land (rs:Farmland), garden (rs:Garden), pasture (rs:Pasture), forest (rs:Forest), shrub (rs:Shrub), bare land (rs:Bareland), wetland (rs:Wetland), and water (rs:Water). The inference unit class (rs:Segment) contained subclasses: correctly classified units (rs:ClassifiedSegment) and incorrectly classified units (rs:MisClassifiedSegment). rs:geoClass is the classification category ontology class, that is, the land cover category. For example, if A is a building, its broad category is rs:geoClass, then rs:geoClass rs:Building. Defines core properties, including the adjacent (geo:adjacentTo) and surrounded (geo:surroundedBy) spatial relationship properties and the adjacent edge length ratio (geo:adjacentRatio) statistical property.

[0050] We created inference rules based on expert prior knowledge offline and used a semantic web rule language to symbolize the ontology inference rules. These rules were then used to correct the land cover classification results. For example, in Table 1, rs:ClassifiedSegment(?entity) represents a correctly classified inference unit ontology instance, while rs:ClassifiedSegment(?misEntity) represents an incorrectly classified inference unit ontology instance.

[0051] Table 1 Inference rules of expert prior knowledge

[0052]

[0053] Step 1.3: Store the scene prior knowledge offline in the spatiotemporal knowledge graph and establish scene prior reasoning rules to facilitate query calls in subsequent steps.

[0054] Open Street Map (OSM) land cover data is stored in the spatiotemporal knowledge graph as contextual prior knowledge. Contextual prior inference rules are created offline and symbolized using a semantic web rule language to represent the ontology inference rules. These rules are used to correct land cover classification results based on contextual prior knowledge. The contextual prior inference rules are shown in Table 2:

[0055] Table 2 Scenario prior reasoning rules

[0056]

[0057] Step 2: Based on the geographic coordinates and imaging time metadata information of the remote sensing image, the spatiotemporal knowledge graph is queried to obtain the corresponding natural prior knowledge. The remote sensing image and natural prior knowledge are combined as the input data of the deep semantic segmentation network. Based on the loss function, the deep semantic segmentation network is optimized through the backpropagation algorithm. The trained deep semantic segmentation network is used to classify the land cover of the remote sensing image to obtain the classification result map as well as the predicted category and classification confidence of each pixel point.

[0058] According to the metadata information of the remote sensing image I, such as the geographic coordinates and imaging time, the spatiotemporal knowledge graph is queried to obtain the corresponding natural prior knowledge E. The remote sensing image I and the natural prior knowledge E are combined as the input data of the deep semantic segmentation network. Based on the classic cross entropy loss function, the deep semantic segmentation network is optimized through the backpropagation algorithm Adaptive Moment Estimation (Adam).

[0059] In this embodiment, the deep semantic segmentation network uses the HRNet network. Other deep semantic segmentation networks can also be selected according to actual conditions. The HRNet network is different from the conventional encoder-decoder framework. Instead, it retains high-resolution information as much as possible from beginning to end. The HRNet network contains 4 types of resolution feature map information. Each row of the feature map sequence corresponds to a branch, and the feature map is divided into 4 stages in the column direction: Stage 1 consists of 1 layer, which is composed of 4 bottleneck modules in series; Stage 2 contains two branches, each branch contains 4 basic modules; Stage 3 contains 3 branches, each branch contains 4 basic modules, and this stage is repeated 4 times in the entire network; Stage 4 contains 4 branches, each branch contains 4 basic modules, and this stage is repeated 3 times in the entire network; The bottleneck module compresses the feature channel through a 1×1 convolution block to reduce the computational cost, and uses the skip connection in the residual network to ensure that the resolution of the feature map remains unchanged during the feature transfer process; The basic module uses a 3×3 convolution block to extract features and transfers feature information through a skip connection. The resolution remains unchanged during the feature transfer process.

[0060] This embodiment uses a self-constructed remote sensing image land cover classification dataset to train a deep semantic segmentation network. Other remote sensing image land cover classification datasets can also be selected according to actual conditions. The dataset used in this embodiment includes 12 categories of land feature categories, namely building, infrastructure, industrial and mining land, urban green space, cultivated land, garden, pasture, forest, shrub, bare land, wetland and water. The ratio of the number of images in the training set, validation set and test set is 6:2:2. The trained deep semantic segmentation network is used to classify the land cover of the remote sensing image to obtain the classification result map R and the predicted category C and classification confidence D of each pixel point P.

[0061] Step 3: Use the connected domain segmentation method to segment the classification result graph to obtain a segmentation graph, and use the connected domain in the segmentation graph as the expert prior reasoning unit.

[0062] Use the connected domain segmentation method to segment the classification result graph R into connected domains, and obtain the segmentation graph G, which contains K connected domains S. The connected domain S is used as the expert prior reasoning unit S', that is:

[0063] G={S1,S2,…,S i ,…|S i =ConnectedComponents(R),1≤i <K} (3)

[0064] Where ConnectedComponents(·) represents the division of the image into connected components.

[0065] Step 4: Referring to the land cover categories defined in the inference rules of expert prior knowledge, each expert prior reasoning unit is instantiated according to its corresponding geographical scope and land cover category, and all generated expert prior reasoning unit instances are constructed into the spatiotemporal knowledge graph.

[0066] "Land cover category class" is an abstract concept class. For example, "arable land" is the abstract general term for the land cover category of arable land. Instantiation is the process of obtaining all expert prior reasoning units of the arable land category in the image and their corresponding geographical ranges, generating instances of expert prior reasoning units, and finally constructing the instantiated expert prior reasoning units into the spatiotemporal knowledge graph for subsequent reasoning.

[0067] In step 5, the expert prior reasoning unit instance is used as the reasoning unit. According to the expert prior reasoning rules, the misclassified categories in the classification result graph R in step 2 are searched and corrected to obtain the expert prior reasoning result graph R′.

[0068] Taking the expert prior reasoning unit S′ as the reasoning unit, the following expert prior reasoning rules are used to correct the misclassified categories in the classification result graph R, and the expert prior reasoning result graph R′ is obtained:

[0069] Expert a priori reasoning rule 1: forests whose adjacent length to buildings, industrial and mining land, and urban green land accounts for more than half of the total length of the adjacent edges are corrected to urban green land;

[0070] Expert a priori reasoning rule 2: bare land adjacent to industrial and mining land is corrected to industrial and mining land;

[0071] Expert a priori reasoning rule 3: water bodies surrounded by buildings, infrastructure, and industrial and mining land are corrected into corresponding categories of buildings, infrastructure, and industrial and mining land;

[0072] Expert prior reasoning rule 4 corrects the bare land surrounded by water bodies into water bodies.

[0073] Step 6: Based on the metadata information such as the geographic coordinates and imaging time of the remote sensing image, query the spatiotemporal knowledge graph to obtain the corresponding scene prior knowledge, and further correct the expert prior reasoning result graph according to the scene prior reasoning rules constructed in the spatiotemporal knowledge graph to obtain the final comprehensive reasoning result graph.

[0074] Based on the metadata information such as the geographic coordinates and imaging time of the remote sensing image I, the spatiotemporal knowledge graph is queried to obtain the corresponding scene prior knowledge M, namely the OSM land cover data. The expert prior reasoning result graph R′ is further corrected according to the scene prior reasoning rules constructed in the spatiotemporal knowledge graph, and the comprehensive reasoning result graph R″ is obtained as the output.

[0075] The expert prior reasoning result graph R′ is further corrected through the following scenario prior reasoning rules:

[0076] Scene prior inference rule 1: modify the pixels belonging to industrial and mining land in OSM data to industrial and mining land;

[0077] Scene prior inference rule 2 changes the pixels belonging to industrial land, commercial land, and retail land in OSM data to infrastructure;

[0078] Scene prior inference rule 3 modifies pixels belonging to roads in OSM data into infrastructure;

[0079] Scene prior reasoning rule 4 modifies the pixels belonging to transportation facilities in OSM data into infrastructure;

[0080] Scene prior reasoning rule 5 is changing wetlands and gardens that belong to water bodies in OSM data into water bodies.

[0081] The classification results finally obtained by the present invention are as follows Figure 3 shown.

[0082] Example 2

[0083] Based on the unified inventive concept, the present invention also provides a remote sensing image land cover classification system based on a spatiotemporal knowledge graph, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a remote sensing image land cover classification method based on a spatiotemporal knowledge graph as described above.

[0084] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0085] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A remote sensing image land cover classification method based on spatiotemporal knowledge graph, characterized in that: The following steps are involved: Step 1: Construct a spatiotemporal knowledge graph, including offline storage of natural prior knowledge and scenario prior knowledge in the spatiotemporal knowledge graph, and establishing rules for applying natural prior knowledge, inference rules for expert prior knowledge, and inference rules for scenario prior knowledge. Step 1.1: Store natural prior knowledge offline in the spatiotemporal knowledge graph and establish rules for applying natural prior knowledge to facilitate query and call in subsequent steps; Step 1.2: Establish inference rules based on expert prior knowledge offline in the spatiotemporal knowledge graph; Step 1.3: Store the scene prior knowledge offline in the spatiotemporal knowledge graph and establish scene prior reasoning rules to facilitate query calls in subsequent steps; Step 2: Based on the geographic coordinates and imaging time metadata of the remote sensing image, the spatiotemporal knowledge graph constructed in step 1 is queried to obtain the corresponding natural prior knowledge. The remote sensing image and natural prior knowledge are combined as input data for the deep semantic segmentation network. Based on the loss function, the deep semantic segmentation network is optimized through the backpropagation algorithm. The trained deep semantic segmentation network is used to classify the land cover of the remote sensing image, and the classification result map as well as the predicted category and classification confidence of each pixel are obtained. Step 3: Use the connected domain segmentation method to segment the classification result graph obtained in step 2 to obtain a segmentation graph, and use the connected domain in the segmentation graph as the expert prior reasoning unit; Step 4: Referring to the land cover categories defined in the inference rules of the expert prior knowledge, each expert prior reasoning unit is instantiated according to its corresponding geographical scope and land cover category, and all generated expert prior reasoning unit instances are constructed into the spatiotemporal knowledge graph; Step 5: Using the expert prior reasoning unit instance as the reasoning unit, according to the expert prior reasoning rules, find the misclassified categories in the classification result graph in step 2 and correct them, thereby obtaining the expert prior reasoning result graph; In step 6, based on the geographic coordinates and imaging time of the remote sensing image, the spatiotemporal knowledge graph is queried to obtain the corresponding scene prior knowledge, and the expert prior reasoning result graph obtained in step 5 is further corrected according to the scene prior reasoning rules constructed in the spatiotemporal knowledge graph to obtain the final comprehensive reasoning result graph.

2. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 1, characterized in that: The natural prior knowledge in step 1.1 includes slope and aspect. A mapping relationship between the digital elevation model file paths for a certain imaging time, geographic range, and projection system is established in the spatiotemporal knowledge graph. Furthermore, an operation rule for calculating the slope and aspect from the digital elevation model is established. The slope is defined as the angle between the plumb line and the normal line. The slope α at the pixel (x, y) is calculated as follows: Where, f x and f y are the gradients in the x and y directions at the pixel (x, y); The slope aspect is defined as the angle between the projection of the normal on the horizontal plane and the north direction, measured clockwise. The slope aspect A at the pixel (x, y) is calculated as follows: Where, f x and f y are the gradients in the x and y directions at the pixel (x, y); The rule of applying natural prior knowledge refers to querying the spatiotemporal knowledge graph based on the metadata of remote sensing images to obtain slope and aspect.

3. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 1, characterized in that: In step 1.2, a remote sensing interpretation ontology was established and described using the semantic web rule language (SWRL). A root land cover category was defined, with subcategories including inference unit, buildings, infrastructure, industrial and mining land, urban green space, cultivated land, gardens, pastures, forests, shrubs, bare land, and water bodies. The inference unit category included both correctly classified units and incorrectly classified units. Core attributes were defined, including the spatial relationship attributes of contiguity and encirclement, and the statistical attribute of adjacent edge length ratio. Inference rules based on expert prior knowledge are created offline, and the semantic web rule language is used to symbolically represent the ontology inference rules, which are used to correct the land cover classification results. These rules include: Rule 1: Forests whose adjacent length to buildings, industrial and mining land, and urban green land accounts for more than half of the total length of the adjacent edges are corrected to urban green land; Rule 2: Bare land adjacent to industrial and mining land is corrected to industrial and mining land; Rule 3: Water bodies surrounded by buildings, infrastructure, and industrial and mining land are corrected to buildings, infrastructure, and industrial and mining land of the corresponding categories; Rule 4: Bare land surrounded by water bodies is corrected to water bodies.

4. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 1, characterized in that: In step 1.3, the Open Street Map (OSM) land cover data is stored in the spatiotemporal knowledge graph as scene prior knowledge, and scene prior inference rules are created offline to correct the land cover classification results based on the scene prior knowledge. The rules include: Rule 1 modifies pixels belonging to industrial and mining land in the OSM data to industrial and mining land; Rule 2 modifies pixels belonging to industrial land, commercial land, and retail land in the OSM data to infrastructure; Rule 3 modifies pixels belonging to roads in the OSM data to infrastructure; Rule 4 modifies pixels belonging to transportation facilities in the OSM data to infrastructure; and Rule 5 modifies wetlands and gardens belonging to water bodies in the OSM data to water bodies.

5. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 1, characterized in that: In step 2, the spatiotemporal knowledge graph is queried according to the geographic coordinates and imaging time of the remote sensing image I to obtain the corresponding natural prior knowledge E. The remote sensing image I and the natural prior knowledge E are combined as the input data of the deep semantic segmentation network HRNet. Based on the classic cross-entropy loss function, the HRNet network is optimized through the backpropagation algorithm adaptive moment estimation.

6. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 5, characterized in that: In step 2, the HRNet network contains 4 types of resolution feature map information, each row of feature map sequence corresponds to a branch, and the feature map is divided into 4 stages in the column direction: stage 1 consists of 1 layer, which is composed of 4 bottleneck modules in series; stage 2 contains two branches, each branch contains 4 basic modules; stage 3 contains 3 branches, each branch contains 4 basic modules, and this stage is repeated 4 times in the entire network; stage 4 contains 4 branches, each branch contains 4 basic modules, and this stage is repeated 3 times in the entire network; the bottleneck module compresses the feature channel through a 1×1 convolution block to reduce the computational cost, and uses the skip connection in the residual network to ensure that the resolution of the feature map remains unchanged during the feature transfer process; the basic module uses a 3×3 convolution block to extract features and transfers feature information through skip connections, so that the resolution remains unchanged during the feature transfer process.

7. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 6, characterized in that: In step 2, the HRNet network is trained using the remote sensing image land cover classification training set data. The training set data categories include buildings, infrastructure, industrial and mining land, urban green space, cultivated land, gardens, pastures, forests, shrubs, bare land, wetlands and water bodies; the remote sensing image land cover is classified using the trained deep semantic segmentation network to obtain the classification result map R and the predicted category C and classification confidence D of each pixel point P.

8. The remote sensing image land cover classification method based on spatiotemporal knowledge graph according to claim 1, characterized in that: In step 3, the connected domain segmentation method is used to segment the classification result graph R into connected domains to obtain a segmentation graph G. The segmentation graph G contains K connected domains S. The connected domain S is used as the expert prior reasoning unit S′, that is: G={S1,S2,…,S i ,…|S i =ConnectedComponents(R),1≤i<K} (3) Where S i represents the i-th connected component, and ConnectedComponents(·) represents the division of the image according to the connected components.

9. A remote sensing image land cover classification system based on spatiotemporal knowledge graph, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a remote sensing image land cover classification method based on a spatiotemporal knowledge graph as described in any one of claims 1 to 8.

Images