A multi-modal remote sensing big data intelligent semantic analysis system based on geographic entities

Abstract

This invention discloses a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities, comprising: a knowledge graph construction module, used to extract triple knowledge of geographic entities from remote sensing images to construct a knowledge graph, and convert the triple knowledge of geographic entities in the knowledge graph into natural language; a geographic entity segmentation module, used to segment geographic entities with obvious boundary contours in remote sensing images using natural language and remote sensing images as input, and obtain segmentation results through a first codec model; a geographic entity classification module, used to classify geographic entities without obvious boundary contours in remote sensing images using natural language and remote sensing images as input, and obtain classification results through a second codec model; and a result merging and output module, used to merge the segmentation results and classification results to obtain and output a land cover map with geographic entity scale corresponding to the remote sensing image, thereby achieving accurate land cover mapping.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and geographic information systems, and in particular to a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities. Background Technology

[0002] Remote sensing is the most important macroscopic monitoring method for the Earth's surface. By extracting land cover information from remote sensing data through intelligent interpretation technology, and monitoring continuous changes in land cover and its environmental impact, it forms a crucial information foundation supporting national strategies such as land surveys and dual-carbon targets. Over the years, intelligent interpretation technology for land cover in remote sensing imagery has expanded from the pixel scale to the object scale and then to the scene scale. For example, traditional intelligent interpretation technologies are mostly based on machine learning algorithms such as support vector machines and random forests, using pixels as the basic unit to identify land cover types. Even cutting-edge deep learning models, such as semantic segmentation neural network models, also use pixels as the basic unit for land cover identification. However, due to issues such as spatial-spectral heterogeneity in pixels on remote sensing images, the classification results contain a lot of noise, making it difficult to meet the needs of practical applications. Subsequently, some scholars developed intelligent land cover interpretation technologies based on objects as the basic unit. These technologies delineate land cover boundaries using techniques such as superpixel segmentation, ensuring the homogeneity of land cover types within the boundaries, thus obtaining smooth land cover mapping. However, because object-level segmentation techniques only divide boundaries based on spatial-spectral features and lack the ability to consider spatial contextual semantic information, the ability to recognize and understand land cover is limited. Therefore, some scholars have developed intelligent interpretation technologies based on scenes as the basic unit. Through plot-level scene segmentation and semantic understanding of the contextual relationships of land cover within a scene, more robust land use and cover mapping can be achieved.

[0003] Traditional land cover mapping or land use mapping techniques are discrete abstractions of geographic space. This abstraction narrowly regards "pixel-object-scene" as isolated objects in space and time, and cannot directly or effectively represent the spatial and temporal correlations between geographic objects. At the same time, intelligent interpretation algorithms such as machine learning or deep learning mostly achieve high-level semantic understanding and recognition through bottom-up, layer-by-layer abstraction. They are difficult to intuitively reflect the morphological structure or electromagnetic wave reflection characteristics on which geographic features are based, and are difficult to explain the causal laws of geographic feature cognition. This results in poor interpretability and weak self-cognitive reasoning ability of traditional land cover mapping systems, which also contradicts the process of human cognition and understanding of geographic objects in the real world.

[0004] Current research has attempted to embed semantic knowledge of land features into deep network models to enhance cognitive capabilities. For example, Chinese invention patents with application numbers 2023111400740 and 2023100828961 disclose an urban land use function identification system based on visual and semantic information and an urban land use distribution assessment system based on a knowledge embedding model, respectively. The core technologies of the two are similar, both utilizing language representation networks to extract semantic representations of urban land use functional zones from vector data or text data such as POIs and road networks, thereby assisting visual representation networks in better classifying land use functions from remote sensing images. However, these two approaches have the following problems: 1) The semantic information used in this system only includes data describing the geometric spatial relationships of geographic objects such as POI and AOI, and lacks descriptions of the physical and chemical properties of geographic objects, making the model unsuitable for identifying natural land surfaces; 2) Semantic representation networks achieve the fusion of semantic and visual information by encoding the semantic information of geographic objects such as POI and AOI into feature vectors, but the implicit fusion method is difficult to explain what specific geometric structure or spatial relationship the identification of land use functions depends on, and it still fails to systematically solve the problem of poor interpretability of deep networks.

[0005] Knowledge graphs, as a type of graph-structured data, can systematically express the semantic information of geometric structures and physicochemical properties of ground objects reflected in remote sensing images. They can explicitly guide deep networks in understanding remote sensing images and are therefore frequently used in land use mapping. For example, Chinese invention patent application number 2022102083906 discloses a system for spatial relation semantic analysis based on knowledge graphs. Its core technology involves extracting geographic entities and their relational expressions from a large-scale corpus, and then using a semantic computing system to semantically quantify the geographic entities and their relational descriptions, analyzing the correspondence between geographic entities and relational expressions, as well as the similarity and dissimilarity of geographic entity / relationship descriptions. However, this system is built on a large-scale corpus of textual data, and the entities, relations, and attributes it learns are not specific to remote sensing image data. The triplet knowledge it constructs is difficult to directly use to guide the identification process of geographic objects in remote sensing images. The key problem lies in the heterogeneity between the structured semantic expression of knowledge graphs and the Euclidean space of image visual features, hindering information interaction between them.

[0006] Therefore, existing technologies urgently need further improvement. Summary of the Invention

[0007] This invention provides a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities to improve the understanding and reasoning ability of the complex Earth surface.

[0008] To achieve the above objectives, the present invention provides a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities, comprising:

[0009] The knowledge graph construction module is used to extract triple knowledge of geographic entities from remote sensing images to construct a knowledge graph, and to convert the triple knowledge of geographic entities in the knowledge graph into natural language.

[0010] The geographic entity segmentation module is used to segment geographic entities with obvious boundary contours in the remote sensing image by taking the natural language and remote sensing image as input and using the first codec model to obtain the segmentation result;

[0011] The geographic entity classification module is used to classify geographic entities without obvious boundary contours in the remote sensing image by taking the natural language and remote sensing image as input and using the second codec model to obtain the classification result;

[0012] The result merging and output module is used to merge the segmentation results and classification results to obtain and output the land cover map with geographic entity scale corresponding to the remote sensing image.

[0013] Furthermore, the knowledge graph construction module includes:

[0014] The geoscientific remote sensing knowledge graph construction unit is used to extract geoscientific remote sensing triplet knowledge of geographic entities from the remote sensing image, and to describe the geographic entities and their characteristics through the geoscientific remote sensing triplet knowledge to obtain a geoscientific remote sensing knowledge graph.

[0015] The topological relationship knowledge graph construction unit is used to describe the spatial topological relationships between the geographic entities to obtain a topological relationship knowledge graph;

[0016] The text conversion unit is used to convert the geoscientific remote sensing triplet knowledge in the geoscientific remote sensing knowledge graph and the spatial topological relationships in the topological relationship knowledge graph into the natural language.

[0017] Furthermore, the geoscientific remote sensing knowledge graph construction unit includes:

[0018] The information extraction subunit is used to extract the triplet knowledge of geographic entities from the remote sensing image using a general semantic segmentation model, so as to obtain geoscientific remote sensing triplet knowledge containing entity-relationship-attribute.

[0019] The comprehensive description subunit is used to describe the semantics, phenotype, function, attributes, spectral features, texture features, temporal features, and polarization features of the geographic entity represented on the remote sensing image using the geoscientific remote sensing triplet knowledge, thereby obtaining the geoscientific remote sensing knowledge graph.

[0020] Furthermore, the topological relationship knowledge graph construction unit includes:

[0021] The spatial connection graph structure sub-unit is used to model each geographic entity and its interaction with other geographic entities to obtain the data structure corresponding to each geographic entity.

[0022] The relationship judgment subunit is used to judge and describe the spatial topological relationship between the geographic entities through the data structure corresponding to each geographic entity, and obtain a topological relationship knowledge graph; the spatial topological relationship structure is entity-relationship-entity; the spatial topological relationship includes adjacency, adjacent, disjoint and containment.

[0023] Furthermore, the text conversion unit is specifically used to convert the geoscientific remote sensing triplet knowledge in the geoscientific remote sensing knowledge graph and the spatial topological relationships in the topological relationship knowledge graph into a complete sentence or short text containing a subject, verb, and object.

[0024] Furthermore, the first encoder-decoder model includes an image encoder, a first knowledge encoder, and a segmentation decoder; wherein, the geographic entity segmentation module includes:

[0025] The first image encoding unit is used to extract features of geographic entities with obvious boundary contours in the remote sensing image through the image encoder to obtain the first image encoding data.

[0026] The first text encoding unit is used to perform knowledge text encoding on the natural language through the first knowledge encoder to obtain first knowledge encoded data;

[0027] The entity segmentation unit is used to process the first image encoded data and the first knowledge encoded data through the segmentation decoder to obtain the segmentation result.

[0028] Further, the first image encoding unit includes:

[0029] Image reconstruction unit is used to reconstruct the visual information of the occluded area in the remote sensing image based on the spatial context to obtain the reconstructed remote sensing image;

[0030] The block processing subunit is used to divide the restored remote sensing image into blocks by a mask autoencoder, randomly mask the image blocks according to a preset ratio, and perform linear mapping and encoding on the unmasked image blocks to obtain the intrinsic feature representation of the restored remote sensing image.

[0031] The self-supervised training subunit is used to reconstruct the image from the intrinsic feature representation through the reconstruction decoder to obtain the reconstructed image, and to supervise the image encoder through the remote sensing image and the reconstructed image to achieve self-supervised training and obtain the pre-trained image encoder.

[0032] The supervised training subunit is used to supervise the training of the pre-trained image encoder using the labeled remote sensing images to obtain a trained image encoder.

[0033] The feature extraction subunit is used to enable the trained image encoder to learn the visual dimensions in the global context through attention convergence, and to extract image features of geographic entities with obvious boundary contours in the remote sensing image through the visual dimensions to generate the first image encoding data.

[0034] Further, the first text encoding unit includes:

[0035] The text lexical subunit is used to perform feature projection on the natural language to obtain the index, key, and value of each word;

[0036] The multi-head self-attention computation subunit is used to perform vector multiplication of the index of the target word with the keys of other words in the sentence to obtain the interactive attention value; and to perform vector multiplication of the interactive attention value with the value of the target word to obtain the knowledge feature representation.

[0037] The knowledge text location encoding subunit is used to generate the location encoding corresponding to each visual word and text word in the natural language, and to merge the location encoding and knowledge feature representation to obtain the first knowledge encoding data.

[0038] Furthermore, the entity segmentation unit includes:

[0039] The feature projection subunit is used to perform feature projection on the first image encoded data and the first knowledge encoded data respectively to obtain image encoded data features and knowledge encoded data features;

[0040] The feature calculation subunit is used to perform cross-attention calculation on the image encoded data features and knowledge encoded data features to obtain fused features;

[0041] The feature processing subunit is used to perform feature transformation on the fused features to obtain two-dimensional matrix features and then upsample them to obtain the segmentation result.

[0042] Furthermore, the second encoder-decoder model includes a recurrent encoder, a second knowledge encoder, and a classification decoder; wherein, the geographic entity classification module includes:

[0043] The second image encoding unit is used to enable the cyclic encoder to extract features of geographic entities without obvious boundary contours in the remote sensing image in the time series based on the long short-term memory network model, so as to obtain the second image encoding data.

[0044] The second text encoding unit is used to perform knowledge text encoding on the natural language through the second knowledge encoder to obtain second knowledge encoded data;

[0045] An entity classification unit is used to process the second image encoded data and the second knowledge encoded data through the classification decoder to obtain the classification result.

[0046] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:

[0047] 1. This invention optimizes traditional remote sensing big data intelligent semantic analysis and land cover mapping. It addresses the problem that traditional land cover mapping technology narrowly abstracts geographic space into isolated objects in space and time at the scale of "pixels, objects, and scenes," which contradicts human cognition and understanding of geographic objects in the real world. By segmenting and classifying geographic entities, geographic knowledge (such as attribute characteristics and spatial relationships) is embedded into the type prediction process of geographic entities, achieving spatially clear, semantically explicit, and geographically accurate land cover mapping.

[0048] 2. This invention employs a top-down triplet knowledge extraction approach, starting from the knowledge system of the geosciences domain, to describe the semantics, phenotype, and attributes of geographic entities. Simultaneously, it extracts spectral, textural, temporal, and polarization features of various land cover features from optical remote sensing imagery for knowledge description, achieving a correlation between geosciences knowledge and remote sensing parameters. This is used to guide the classification of land cover entity types such as forests, grasslands, and shrubs that lack clear boundary outlines. Furthermore, through a spatial relationship graph structure, it extracts topological relationships such as adjacency, proximity, disjointness, and inclusion from multiple publicly available land cover datasets, thus constructing a topological relationship knowledge graph between geographic entities.

[0049] 3. This invention constructs a multimodal knowledge embedding engineering. In the knowledge text modality, knowledge lexical embeddings are obtained by mining the global context representation of the knowledge text; in the remote sensing image modality, visual lexical units that significantly contribute to the semantic reasoning of target entities in the global context are learned through attention convergence, and the mechanism of action of text lexical units on visual lexical units is explored through interactive attention mechanism, thereby realizing intelligent semantic analysis of land cover under the interaction of knowledge and image modalities. Attached Figure Description

[0050] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a device diagram of a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities provided in a certain embodiment of the present invention;

[0052] Figure 2 This is a structural diagram of a knowledge graph construction module 1 provided in a certain embodiment of the present invention;

[0053] Figure 3 This is a structural diagram of a geoscience remote sensing knowledge graph construction unit 11 provided in a certain embodiment of the present invention;

[0054] Figure 4 This is a technical flowchart of the geoscience remote sensing knowledge graph construction unit 11 provided in a certain embodiment of the present invention;

[0055] Figure 5 This is a structural diagram of a topological relationship knowledge graph construction unit 12 provided in a certain embodiment of the present invention;

[0056] Figure 6 This is a technical flowchart of the topological relationship knowledge graph construction unit 12 provided in a certain embodiment of the present invention;

[0057] Figure 7 This is a structural diagram of the geographic entity segmentation module 2 provided in a certain embodiment of the present invention;

[0058] Figure 8 This is a structural diagram of the first image encoding unit 21 provided in a certain embodiment of the present invention;

[0059] Figure 9 This is a structural diagram of the first text encoding unit 22 provided in a certain embodiment of the present invention;

[0060] Figure 10 This is a structural diagram of the entity segmentation unit 23 provided in a certain embodiment of the present invention;

[0061] Figure 11 This is a technical flowchart of the geographic entity segmentation module 2 provided in a certain embodiment of the present invention;

[0062] Figure 12 This is a structural diagram of the geographic entity classification module 3 provided in a certain embodiment of the present invention;

[0063] Figure 13This is a technical flowchart of the geographic entity classification module 3 provided in a certain embodiment of the present invention;

[0064] Figure 14 This is a flowchart of a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities, provided in one embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0067] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0068] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0069] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0070] In one embodiment, such as Figure 1 As shown, this invention provides a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities, comprising:

[0071] Knowledge graph construction module 1 is used to extract triple knowledge of geographic entities from remote sensing images to construct a knowledge graph, and to convert the triple knowledge of geographic entities in the knowledge graph into natural language.

[0072] In one embodiment, the structure of the knowledge graph construction module 1 is as follows: Figure 2 As shown, it includes:

[0073] The geoscientific remote sensing knowledge graph construction unit 11 is used to extract geoscientific remote sensing triplet knowledge of geographic entities from the remote sensing image, and to describe the geographic entities and their characteristics through the geoscientific remote sensing triplet knowledge to obtain a geoscientific remote sensing knowledge graph.

[0074] In one embodiment, the structure of the geoscience remote sensing knowledge graph construction unit 11 is as follows: Figure 3 As shown, it includes:

[0075] The information extraction subunit 111 is used to extract the triplet knowledge of geographic entities from the remote sensing image through a general semantic segmentation model, so as to obtain geoscientific remote sensing triplet knowledge containing entity-relationship-attribute.

[0076] The comprehensive description subunit 112 is used to describe the semantics, phenotype, function, attributes, spectral features, texture features, temporal features, and polarization features of the geographic entity represented on the remote sensing image through the geoscientific remote sensing triplet knowledge, so as to obtain the geoscientific remote sensing knowledge graph.

[0077] Specifically, the multimodal remote sensing big data intelligent semantic analysis system proposed in this application can not only extract triplet knowledge of geographic entities from high-resolution remote sensing images, but also from time-series medium-resolution remote sensing images, synthetic aperture radar images, and publicly available land cover mapping products. The general semantic segmentation models used for extracting triplet knowledge include, but are not limited to, SegGPT and Segment Anything Model (SAM). The technical process of the geoscientific remote sensing knowledge graph construction unit 11 is as follows: Figure 4As shown, this is used to output a geoscientific remote sensing knowledge graph. Through "entity-relationship-attribute" triplet knowledge extraction, it provides a comprehensive description of geographic entities (for 10 land cover types) from semantic, phenotypic, physicochemical, process, function, and attribute perspectives. It also utilizes the spectral, geometric, textural, temporal, and polarization characteristics of geographic entities represented on remote sensing images for a comprehensive description. For example, from a phenotypic perspective, triples are constructed such as "Forest—Height—Above 5 meters" and "Forest—Crown Diameter—0.5 to 10 meters." Through overlay analysis of medium-resolution remote sensing images (including Landsat-8 and Sentinel-2) with publicly available land cover mapping products (including GlobeLand30, FROMGLC10, and WorldCover), the spectral characteristics of the 10 land cover types are statistically analyzed (including vegetation indices, water indices, soil indices, and urban indices). For example, "Forest—NDVI Index—between 0.5 and 1," and texture... Features (gray-level co-occurrence matrix, entropy, inverse difference moment, second-order angular moment, etc.); by performing gradient analysis on synthetic aperture radar remote sensing images (including Sentinel-1) and publicly available land cover mapping products, the polarization characteristics of 10 land cover types (including vertical-vertical polarization and vertical-horizontal polarization) were statistically analyzed; by performing overlay analysis on multi-year time-series medium-resolution remote sensing images and publicly available land cover mapping products, the temporal characteristics of farmland and forests were statistically analyzed (mainly referring to the growth cycle of various crops reflected by the time-series vegetation index curve, as well as the time-series vegetation index curves of evergreen forests and deciduous forests in different climate zones throughout the year), etc.

[0078] This application innovatively proposes a method for constructing a triplet knowledge graph based on large-scale high-resolution remote sensing images. Geographic entities are extracted from remote sensing images through a general semantic segmentation model or a geographic entity-scale semantic segmentation network. Using geographic entities as basic units, the method fully utilizes the spectral, temporal phenological, and geometric topological attributes of remote sensing images to construct triplet knowledge for each geographic entity, enabling it to better guide the deep network's understanding and perception of geographic objects.

[0079] The topological relationship knowledge graph construction unit 12 is used to describe the spatial topological relationships between the geographic entities to obtain a topological relationship knowledge graph;

[0080] In one embodiment, the structure of the topological relationship knowledge graph construction unit 12 is as follows: Figure 5 As shown, it includes:

[0081] The spatial connection graph structure subunit 121 is used to model each geographic entity and its interaction with other geographic entities to obtain the data structure corresponding to each geographic entity.

[0082] The relation judgment subunit 122 is used to judge and describe the spatial topological relationship between the geographic entities through the data structure corresponding to each geographic entity, and obtain a topological relationship knowledge graph; the spatial topological relationship structure is entity-relationship-entity; the spatial topological relationship includes adjacency, adjacent, disjoint and containment.

[0083] Specifically, this application uses publicly available land cover mapping products as input. First, it vectorizes the land cover vector plots to obtain initial geographic entities for data modeling. Then, it models a set of geographic entities (nodes) and their interactions (edges) to form a data structure, represented as G(V,E), where V represents a node in the graph and E represents an edge. Each geographic entity is represented by a node, and the spatial interaction between a pair of geographic entities is represented by an edge.

[0084] The technical process of the topological relationship knowledge graph construction unit 12 is as follows: Figure 6 As shown, when judging spatial topological relationships, "adjacency" is first used as the spatial relationship between geographic entities. That is, if two geographic entities are in contact with each other and share part of their boundary, they are spatially adjacent. At the same time, entity nodes with the same land cover type generally have similar neighborhood structures in the graph topology, and similar neighborhood structures also mean that geographic entities have similar functions in the graph structure.

[0085] Based on different neighborhood structures, the adjacency relationships between geographic entities are finely divided. If an entity type is adjacent to multiple other entities, it plays a dominant role in space; if an entity type F is only adjacent to another specific entity type E, and not adjacent to other types, it can be determined that entity types F and E have an "inclusion" relationship.

[0086] For non-adjacent entity types F and E, buffer analysis is performed. A buffer zone is constructed on entity type F with a certain radius. It is then counted whether entity type E falls into the buffer zone. If it does, entity types F and E are considered "adjacent"; otherwise, they are considered "disjoint". Spatial topological relationships can characterize the explicit spatial connections between geographic objects and reflect the neighborhood influence of geographic entities. Their structural characteristics are closely related to the connectivity, function, and stability of the geographic landscape.

[0087] This invention extracts the edge contours of entities such as buildings, trees, fields, and water bodies from high-resolution remote sensing images using a general semantic segmentation model or a geographic entity-scale semantic segmentation network. It then encodes the physicochemical properties reflected in electromagnetic wave reflection characteristics and the topological relationships reflected in geometric space for all entities using triplet encoding, forming a "geological remote sensing knowledge graph" and a "spatial topological relationship knowledge graph." This invention not only constructs a knowledge graph describing topological relationships such as adjacency, connection, separation, and inclusion through spatial connection graph structure subunits 121 and relationship judgment subunits 122, but also constructs a geological remote sensing knowledge graph describing vegetation indices, water body indices, polarization characteristics, and time-series phenological characteristics, thereby achieving spatially clear, semantically explicit, and geographically accurate land cover mapping.

[0088] The text conversion unit 13 is used to convert the geoscientific remote sensing triple knowledge in the geoscientific remote sensing knowledge graph and the spatial topological relations in the topological relation knowledge graph into natural language, so as to serve as the input of the subsequent knowledge encoder for extracting knowledge feature vectors.

[0089] In one embodiment, the text conversion unit 13 is specifically used to convert the geoscientific remote sensing triplet knowledge in the geoscientific remote sensing knowledge graph and the spatial topological relationship in the topological relationship knowledge graph into a complete sentence or short text containing a subject, verb, and object; for example, "forest - NDVI index - between 0.5 and 1" is converted into "the NDVI index of the forest is between 0.5 and 1", and "forest (urban area) - adjacent - building" is converted into "buildings are adjacent to urban forests".

[0090] This invention employs a top-down triplet knowledge extraction approach, starting from the geoscience knowledge system, to describe geographic entities from semantic, phenotypic, and attribute perspectives. Simultaneously, it extracts spectral, textural, temporal, and polarization features of various land cover types from medium-resolution optical remote sensing images and synthetic aperture radar remote sensing images for knowledge description, achieving a correlation between geoscience knowledge and remote sensing parameters. This is used to guide the classification of land cover entity types such as forests, grasslands, and shrubs that lack clear boundary outlines. Furthermore, through a spatial relationship graph structure, it extracts topological relationships such as adjacency, proximity, disjointness, and inclusion from multiple public land cover datasets, thus constructing a topological relationship knowledge graph between geographic entities.

[0091] The geographic entity segmentation module 2 is used to segment geographic entities with clear boundary contours in the remote sensing image using the natural language and remote sensing image as input, and obtain segmentation results through the first codec model. Specifically, the geographic entity segmentation module 2 of this application is based on knowledge embedding engineering and uses natural language text containing triple knowledge as input to guide the model to simulate the human brain's cognitive understanding of geographic objects in the real world, so as to achieve accurate segmentation and semantic inference of five types of geographic entities with clear boundary contours in the remote sensing image, such as farmland, impermeable surface, buildings, roads, and forests (cities).

[0092] In one embodiment, the first encoder-decoder model includes an image encoder, a first knowledge encoder, and a segmentation decoder; wherein, the structure of the geographic entity segmentation module 2 is as follows: Figure 7 As shown, it includes:

[0093] The first image encoding unit 21 is used to extract features of geographic entities with obvious boundary contours in the remote sensing image through the image encoder to obtain the first image encoding data.

[0094] In one embodiment, the structure of the first image encoding unit 21 is as follows: Figure 8 As shown, it includes:

[0095] Image reconstruction unit 211 is used to restore the visual information of the occluded area in the remote sensing image based on the spatial context to obtain the restored remote sensing image.

[0096] The block processing subunit 212 is used to divide the restored remote sensing image into blocks by a mask autoencoder, randomly mask the image blocks according to a preset ratio, and perform linear mapping and encoding on the unmasked image blocks to obtain the intrinsic feature representation of the restored remote sensing image.

[0097] Self-supervised training subunit 213 is used to reconstruct the intrinsic feature representation through the reconstruction decoder to obtain the reconstructed image, and to supervise the image encoder through the remote sensing image and the reconstructed image to achieve self-supervised training and obtain the pre-trained image encoder.

[0098] The supervised training subunit 214 is used to supervise the training of the pre-trained image encoder using the labeled remote sensing images to obtain a trained image encoder.

[0099] The feature extraction subunit 215 is used to enable the trained image encoder to learn the visual dimensions in the global context through attention convergence, and to extract image features of geographic entities with obvious boundary contours in the remote sensing image through the visual dimensions to generate the first image encoding data.

[0100] Specifically, this image encoder is used to perform preliminary image feature extraction on high-resolution remote sensing images and generate image codes. The image encoder is a Swin Transformer. The image encoder is completed in two steps: self-supervised training and supervised training. Visual information has a large information density and redundancy at the pixel level. Some occluded areas can be restored by understanding the spatial information of the neighborhood based on certain high-level features (spatial context). By artificially occluding some areas, the network learns how to restore these areas, which can enhance the high-level feature representation ability of the autoencoder. Based on this, a masked autoencoder with an asymmetric encoding and decoding structure is constructed. First, the input image is divided into blocks, and the image blocks are randomly masked according to a certain ratio. The visible image blocks that are not masked are linearly mapped and encoded to obtain the intrinsic feature representation of the remote sensing image. The dimension of the encoded features is expanded to represent the potential representation of the masked image blocks, and the masked image blocks are reconstructed using the decoder. Finally, the image encoder is supervised using the original input image and the reconstructed image to achieve self-supervised training.

[0101] Then, on the pre-trained image encoder obtained through self-supervised training, supervised training is performed using a large amount of labeled semantic segmentation data. Through attention convergence, visual lexical units that significantly contribute to semantic reasoning of target entities in the global context are learned, and preliminary image feature extraction is performed to generate image codes. In the field of remote sensing image modalities, this application utilizes visual lexicalization and attention convergence to learn visual lexical units that significantly contribute to semantic reasoning of target entities in the global context, thereby improving the accuracy of image prediction and recognition.

[0102] The first text encoding unit 22 is used to perform knowledge text encoding on the natural language through the first knowledge encoder to obtain first knowledge encoded data;

[0103] In one embodiment, the structure of the first text encoding unit 22 is as follows: Figure 9 As shown, it includes:

[0104] The text lexical subunit 221 is used to perform feature projection on the natural language to obtain the index, key, and value of each word;

[0105] The multi-head self-attention computation subunit 222 is used to perform vector multiplication of the index of the target word with the keys of other words in the sentence to obtain the interactive attention value; and to perform vector multiplication of the interactive attention value with the value of the target word to obtain the knowledge feature representation;

[0106] The knowledge text location encoding subunit 223 is used to generate the location encoding corresponding to each visual word and text word in the natural language, and to merge the location encoding and knowledge feature representation to obtain the first knowledge encoding data.

[0107] Specifically, this application uses text lexicalization to focus attention on non-volitional cues such as other words, using the target word as the volitional cue. This integrates and mines the global contextual representation of the knowledge text to obtain a knowledge code. Specifically: First, the index, key, and value information of each word are obtained by projecting natural language features. Then, the interaction attention value is obtained by multiplying the target word's query with the keys of all other words in the sentence. This interaction attention value is then multiplied with the target unit's value to obtain a knowledge feature representation containing contextual semantic information. On the other hand, considering that knowledge text lacks location information while remote sensing images have explicit location information, a location code is generated for each visual and text lexical unit during fusion to ensure that the knowledge text can accurately influence the semantic inference of ground features at specific locations on the remote sensing image. This code is used to learn the coordinate domain of the knowledge text. Finally, the location codes and knowledge feature representations are merged to obtain the first knowledge code data. In terms of knowledge text modality, this application mines the global contextual representation of the knowledge text through text lexicalization to obtain knowledge lexical embeddings, thereby improving the accuracy of image prediction and recognition.

[0108] The entity segmentation unit 23 is used to process the first image encoded data and the first knowledge encoded data through the segmentation decoder to obtain the segmentation result;

[0109] In one embodiment, the structure of the entity segmentation unit 23 is as follows: Figure 10 As shown, it includes:

[0110] The feature projection subunit 231 is used to perform feature projection on the first image encoded data and the first knowledge encoded data respectively to obtain image encoded data features and knowledge encoded data features;

[0111] The feature calculation subunit 232 is used to perform cross-attention calculation on the image encoded data features and the knowledge encoded data features to obtain fused features;

[0112] The feature processing subunit 233 is used to perform feature transformation on the fused features to obtain two-dimensional matrix features and perform upsampling to obtain the segmentation result.

[0113] Specifically, the technical process of geographic entity segmentation module 2 is as follows: Figure 11As shown, in the segmentation decoder, the first knowledge-encoded data is first projected to obtain Query, Key, and Value; similarly, the first image-encoded data is projected to obtain Query, Key, and Value. Then, cross-attention calculation is performed on the image-encoded data features and the knowledge-encoded data features. Specifically, the Query encoded by the target feature in the image is multiplied by all the Keys encoded by the knowledge, and then multiplied by the Value encoded by the target feature, achieving deep coupling between geographic knowledge and image segmentation. Finally, the output fused features are transformed from word vector features to two-dimensional matrix features, and the final geographic entity segmentation result is obtained through upsampling.

[0114] This application explores the mechanism by which textual terms affect visual terms through interactive attention, enabling intelligent semantic analysis of land cover under the interaction of knowledge and image modalities. Through multimodal knowledge embedding engineering, geographical knowledge (such as attribute features and spatial relationships) is embedded into the type prediction process of geographical entities, achieving spatially clear, semantically explicit, and geographically accurate land cover mapping.

[0115] The geographic entity classification module 3 is used to classify geographic entities without obvious boundary contours in the remote sensing image by taking the natural language and remote sensing image as input and using the second codec model to obtain the classification result. Specifically, the geographic entity classification module 3 is based on long short-term memory and uses natural language with triple knowledge as input to realize intelligent classification and semantic inference of six types of geographic entities without obvious boundary contours in the remote sensing image, namely forest (suburb), grassland, shrub, wetland, water body and bare land.

[0116] In one embodiment, the second encoder-decoder model includes a recurrent encoder, a second knowledge encoder, and a classification decoder; wherein the structure of the geographic entity classification module 3 is as follows: Figure 12 As shown, it includes:

[0117] The second image encoding unit 31 is used to enable the cyclic encoder to extract features of geographic entities without obvious boundary contours in the remote sensing image in the time series based on the long short-term memory network model, so as to obtain the second image encoding data.

[0118] The second text encoding unit 32 is used to perform knowledge text encoding on the natural language through the second knowledge encoder to obtain second knowledge encoded data;

[0119] The entity classification unit 33 is used to process the second image encoded data and the second knowledge encoded data through the classification decoder to obtain the classification result.

[0120] Specifically, the technical process of the geographic entity classification module 3 is as follows: Figure 13 As shown, the recurrent encoder, based on a Long Short-Term Memory (LSTM) network model, performs preliminary image feature extraction on time-series medium-resolution remote sensing images to generate image codes. Since the core of LSTM is the memory unit, it can remember historical information and utilize this information when processing current data. Specifically, the memory unit is a basic component of a recurrent neural network (RNN) that can save and update its state, combining historical and current information. When processing time-series images, LSTM takes each pixel or feature as input, treating each row or column of the image as an element of the sequence. Information from each pixel is passed through LSTM units and used to update the state of the memory units. Furthermore, LSTM has three gates (Input Gate, Forget Gate, and Output Gate), each performing a different task. The Input Gate controls the degree of new information input, the Forget Gate determines the degree of retention of historical information, and the Output Gate determines the final output.

[0121] Compared to traditional convolutional neural networks (CNNs), LSTMs are better able to capture temporal-dependent information and improve classification accuracy for image classification. In land cover mapping tasks using remote sensing imagery, land cover types such as forests, grasslands, shrubs, wetlands, and bare land, which lack clear boundary outlines and geometric shapes, are significantly affected by temporal features in classification results because they typically exhibit specific periodic fluctuations over time. LSTMs can effectively utilize this information for classification by leveraging their memory units and gating mechanisms.

[0122] The second knowledge encoder is used to encode the text converted from the geoscientific remote sensing knowledge graph input to the model. The input knowledge serves as guiding information to assist in the classification of geographic entities. The specific steps are the same as those of the first knowledge encoder in the geographic entity segmentation module. This classification decoder is based on the image feature input of the recurrent encoder and the knowledge encoding input of the knowledge encoder, and provides the final geographic entity classification result output. The specific process is also the same as that of the segmentation decoder in the geographic entity segmentation module. The knowledge encoder and classification / segmentation decoder mentioned in this paper are commonly used in the field and are not specifically limited thereto.

[0123] This invention optimizes traditional remote sensing big data intelligent semantic analysis and land cover mapping. It addresses the problem that traditional land cover mapping technology narrowly abstracts geographic space into isolated objects in space and time at the scale of "pixels, objects, and scenes," which contradicts human cognition and understanding of geographic objects in the real world. By using a geographic entity segmentation module, geographic knowledge (such as attribute characteristics and spatial relationships) is embedded into the geographic entity type prediction process, achieving spatially clear, semantically explicit, and geographically accurate land cover mapping.

[0124] The result merging and output module 4 is used to merge the segmentation results and classification results to obtain and output the land cover map with geographic entity scale corresponding to the remote sensing image.

[0125] Specifically, the workflow of a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities is as follows: Figure 14 As shown, the merging methods include, but are not limited to, overlay and intersection, etc., without further specific limitations. In the training phase, this application inputs a geoscientific remote sensing knowledge graph and a topological relationship knowledge graph, as well as a high-resolution semantic segmentation remote sensing dataset with image segmentation annotations. Supervised training is performed using the intersection-union ratio (IUU) between predicted geographic entities and reference geographic entities as the loss function. In the testing phase, the application inputs the geoscientific remote sensing knowledge graph and the topological relationship knowledge graph, as well as high-resolution remote sensing images of unknown regions, to obtain land cover mapping results at the geographic entity scale for the corresponding region. After achieving geographic entity-level image segmentation, the application uses the geographic objects expressed in the knowledge graph, along with their physicochemical properties and spatial topological relationships, to explicitly guide the deep neural network learning process, constraining the identification of land cover types for each geographic entity, achieving more robust land cover mapping, and improving the understanding and reasoning ability regarding the complex Earth's surface.

[0126] This application proposes a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities to address the challenge of overcoming the multimodal heterogeneity between knowledge graphs and image visual features. The system includes: a knowledge graph construction module for extracting triplet knowledge of geographic entities from remote sensing images to construct a knowledge graph and converting this triplet knowledge into natural language; a geographic entity segmentation module for segmenting geographic entities with clear boundary contours in the remote sensing images using the natural language and remote sensing images as input, through a first codec model; a geographic entity classification module for classifying geographic entities without clear boundary contours in the remote sensing images using the natural language and remote sensing images as input, through a second codec model; and a result merging and output module for merging the segmentation and classification results to obtain and output a land cover map with geographic entity scale corresponding to the remote sensing images. This achieves spatially clear, semantically explicit, and geographically accurate land cover mapping.

[0127] In summary, this invention relates to the fields of computer vision and geographic information systems, and provides a multimodal remote sensing big data intelligent semantic analysis system based on geographic entities. The system includes a knowledge graph construction module for extracting triple knowledge of geographic entities from remote sensing images to construct a knowledge graph, and converting the triple knowledge of geographic entities in the knowledge graph into natural language; a geographic entity segmentation module for segmenting geographic entities with clear boundary contours in the remote sensing images using the natural language and remote sensing images as input, through a first codec model, to obtain segmentation results; a geographic entity classification module for classifying geographic entities without clear boundary contours in the remote sensing images using the natural language and remote sensing images as input, through a second codec model, to obtain classification results; and a result merging and output module for merging the segmentation results and classification results to obtain and output a land cover map with geographic entity scale corresponding to the remote sensing images. This research extracts spectral, textural, temporal, and polarization features of various land cover features from optical remote sensing images for knowledge description, enabling the association between geoscientific knowledge and remote sensing parameters. This information is used to guide the classification of land cover entities such as forests, grasslands, and shrubs that do not have clear boundary outlines. Furthermore, it extracts topological relationships such as adjacency, proximity, disjointness, and inclusion from land cover datasets, enabling the construction of a knowledge graph of topological relationships between geographic entities. Finally, it embeds geographic knowledge (such as attribute features and spatial relationships) into the type prediction process of geographic entities, achieving spatially clear, semantically explicit, and geographically accurate land cover mapping.

[0128] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts among the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the system embodiments, so the description is relatively simple; refer to the description of the system embodiments for relevant parts. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0129] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A multimodal remote sensing big data intelligent semantic analysis system based on geographic entities, characterized in that, include: The knowledge graph construction module is used to extract triple knowledge of geographic entities from remote sensing images to construct a knowledge graph, and to convert the triple knowledge of geographic entities in the knowledge graph into natural language. A geographic entity segmentation module is used to segment geographic entities with clear boundary contours in the remote sensing image using the natural language and remote sensing image as input, through a first encoder-decoder model, to obtain segmentation results. The geographic entity segmentation module includes: a first image encoding unit, used to extract features from geographic entities with clear boundary contours in the remote sensing image using an image encoder, to obtain first image encoded data; a first text encoding unit, used to perform knowledge text encoding on the natural language using a first knowledge encoder, to obtain first knowledge encoded data; an entity segmentation unit, used to process the first image encoded data and the first knowledge encoded data using a segmentation decoder, to obtain the segmentation results; the first image encoding unit includes: an image reconstruction unit, used to reconstruct the visual information of occluded areas in the remote sensing image based on spatial context, to obtain a reconstructed remote sensing image; and a block processing subunit, used for... The restored remote sensing image is divided into blocks by a masked autoencoder, and the image blocks are randomly masked according to a preset ratio. The unmasked image blocks are linearly mapped and encoded to obtain the intrinsic feature representation of the restored remote sensing image. A self-supervised training subunit is used to reconstruct the image using the intrinsic feature representation through a reconstruction decoder to obtain a reconstructed image. The image encoder is then supervised by the remote sensing image and the reconstructed image to achieve self-supervised training, resulting in a pre-trained image encoder. A supervised training subunit is used to supervise the training of the pre-trained image encoder using the labeled remote sensing image to obtain a trained image encoder. A feature extraction subunit is used to enable the trained image encoder to learn the visual dimensions in the global context through attention convergence, and to extract image features of geographic entities with obvious boundary contours in the remote sensing image through the visual dimensions to generate the first image encoding data. A geographic entity classification module is used to classify geographic entities without obvious boundary contours in the remote sensing imagery using the natural language and remote sensing imagery as input, through a second encoder-decoder model, to obtain classification results. The geographic entity classification module includes: a second image encoding unit, used to enable a recurrent encoder to extract features from geographic entities without obvious boundary contours in the time-series remote sensing imagery based on a long short-term memory network model, to obtain second image encoded data; a second text encoding unit, used to perform knowledge text encoding on the natural language using a second knowledge encoder, to obtain second knowledge encoded data; and an entity classification unit, used to process the second image encoded data and the second knowledge encoded data using a classification decoder, to obtain the classification results. The result merging and output module is used to merge the segmentation results and classification results to obtain and output the land cover map with geographic entity scale corresponding to the remote sensing image.

2. The intelligent semantic analysis system for multimodal remote sensing big data based on geographic entities according to claim 1, characterized in that, The knowledge graph construction module includes: The geoscientific remote sensing knowledge graph construction unit is used to extract geoscientific remote sensing triplet knowledge of geographic entities from the remote sensing image, and to describe the geographic entities and their characteristics through the geoscientific remote sensing triplet knowledge to obtain a geoscientific remote sensing knowledge graph. The topological relationship knowledge graph construction unit is used to describe the spatial topological relationships between the geographic entities to obtain a topological relationship knowledge graph; The text conversion unit is used to convert the geoscientific remote sensing triplet knowledge in the geoscientific remote sensing knowledge graph and the spatial topological relationships in the topological relationship knowledge graph into the natural language.

3. The intelligent semantic analysis system for multimodal remote sensing big data based on geographic entities according to claim 2, characterized in that, The geoscientific remote sensing knowledge graph construction unit includes: The information extraction subunit is used to extract the triplet knowledge of geographic entities from the remote sensing image using a general semantic segmentation model, so as to obtain geoscientific remote sensing triplet knowledge containing entity-relationship-attribute. The comprehensive description subunit is used to describe the semantics, phenotype, function, attributes, spectral features, texture features, temporal features, and polarization features of the geographic entity represented on the remote sensing image using the geoscientific remote sensing triplet knowledge, thereby obtaining the geoscientific remote sensing knowledge graph.

4. The intelligent semantic analysis system for multimodal remote sensing big data based on geographic entities according to claim 2, characterized in that, The topological relationship knowledge graph construction unit includes: The spatial connection graph structure sub-unit is used to model each geographic entity and its interaction with other geographic entities to obtain the data structure corresponding to each geographic entity. The relationship judgment subunit is used to judge and describe the spatial topological relationship between the geographic entities through the data structure corresponding to each geographic entity, and obtain a topological relationship knowledge graph; the spatial topological relationship structure is entity-relationship-entity; the spatial topological relationship includes adjacency, adjacent, disjoint and containment.

5. The intelligent semantic analysis system for multimodal remote sensing big data based on geographic entities according to claim 2, characterized in that, The text conversion unit is specifically used to convert the geoscientific remote sensing triplet knowledge in the geoscientific remote sensing knowledge graph and the spatial topological relationships in the topological relationship knowledge graph into a complete sentence or short text containing a subject, verb, and object.

6. The intelligent semantic analysis system for multimodal remote sensing big data based on geographic entities according to claim 1, characterized in that, The first text encoding unit includes: The text lexical subunit is used to perform feature projection on the natural language to obtain the index, key, and value of each word; The multi-head self-attention computation subunit is used to perform vector multiplication of the index of the target word with the keys of other words in the sentence to obtain the interactive attention value; and to perform vector multiplication of the interactive attention value with the value of the target word to obtain the knowledge feature representation. The knowledge text location encoding subunit is used to generate the location encoding corresponding to each visual word and text word in the natural language, and to merge the location encoding and knowledge feature representation to obtain the first knowledge encoding data.

7. The intelligent semantic analysis system for multimodal remote sensing big data based on geographic entities according to claim 1, characterized in that, The entity segmentation unit includes: The feature projection subunit is used to perform feature projection on the first image encoded data and the first knowledge encoded data respectively to obtain image encoded data features and knowledge encoded data features; The feature calculation subunit is used to perform cross-attention calculation on the image encoded data features and knowledge encoded data features to obtain fused features; The feature processing subunit is used to perform feature transformation on the fused features to obtain two-dimensional matrix features and then upsample them to obtain the segmentation result.

Images