A visualization analysis method and system for deep learning representation of map elements

Abstract

The present invention discloses a method and system for visual analysis of deep learning representations of map elements. First, a generative model is trained through an encoder-decoder structure, and the trained decoder is capable of mapping the deep learning representations of map elements from the feature space to the visually perceptible map space. Then, through the transformation, decoding and visual analysis of the deep learning representations, the recognizable concepts of the corresponding map elements are extracted, and the dimensions of the feature space are aligned with the recognizable concepts; finally, the concept alignment results are evaluated through direct verification of artificially designed cases and indirect verification methods of observing classifier weights. The present invention can effectively avoid data labeling and reduce the training burden of the generative model; it can be applied to map elements such as points, lines, and surfaces, and has a certain adaptability; it can use visual human-computer interaction to provide an intuitive explanation for the deep learning representations of map elements, which is helpful for the evaluation, optimization and application of related deep learning models.

Description

Technical Field

[0001] The present invention relates to the fields of cartography and geographic information engineering, and more specifically to a method and system for visual analysis of deep learning representations of map elements, serving the interpretation, evaluation, and optimization of deep learning models related to map elements. Background Art

[0002] In recent years, maps and geographic information systems (GIS) have been highly integrated with artificial intelligence (AI) technologies, gradually moving towards intelligent systems. Representation learning is a key area of ​​AI development. Its goal is to automatically discover and identify structures or patterns in raw data and encode them into feature vectors, commonly known as representations, that can be used for downstream tasks such as classification, clustering, or generation. Currently, researchers have developed numerous deep models to generate representations of various map features, such as points of interest, roads, rivers, and buildings. These representations have been applied to solve a variety of classic spatial problems, including map generalization, urban regionalization, and geographic knowledge question answering. However, the "black box" nature of deep learning models makes it difficult to precisely answer the specific information encoded in these deep learning representations, hindering their understanding, optimization, and application. The key to interpreting deep learning representations of map features is to answer the question, "What information is encoded in the map data representation?" To this end, it is necessary to develop methods that provide human-understandable explanations of the learning process and results of deep learning models. This will help improve the transparency, generalizability, and reliability of intelligent GIS.

[0003] Existing explainable AI methods can be categorized in various ways. First, they can be classified based on the time point of explanation: pre-model training, model embedding, or post-model training. Second, they can be categorized based on whether the explanation method is limited to a specific model or applicable to different models, namely model-specific methods and model-independent methods. Furthermore, based on the scale of explanation, they can be categorized as global or local explanations. The former reveals the overall behavior of the model, including inputs, outputs, and parameters, while the latter focuses on explaining the model's performance in a specific region of interest or case. Furthermore, they can be categorized into visualization and non-visualization methods. Visualization methods often provide intuitive displays of feature importance or the relationship between features and decisions, providing intuitive results, particularly for non-expert users. However, existing explainable AI methods lack analysis of deep learning representations of map features, and the integration of general methods with map scenarios needs to be improved. Therefore, it is necessary to explore intuitive and efficient explanation methods for deep learning representations of map features to facilitate the understanding, evaluation, and optimization of deep learning models for mapping tasks. Taking into account the need for visual interaction of maps as a visual language, as well as the complexity and diversity of map elements, the present invention aims to provide a model-independent, highly versatile, visual analysis method to explore the meaning of deep learning representations of map elements in various spatial dimensions, and provide explanations that can be observed and understood by people.

[0004] As an effective unsupervised learning framework, autoencoders have been widely used in representation learning and interpretability research of various spatial data such as remote sensing images, map vectors, point clouds, etc. Its self-supervised nature can effectively avoid the burden of data annotation. In addition, the structure of the encoder-decoder is very similar to the process of map information transmission: cartographers encode spatial representations in their brains by reading materials and observing geographic space, and then express and disseminate spatial information in a graphical map language through decoding. Inspired by this, the present invention proposes an encoder-decoder structure to map the deep learning representation of map elements back to a visually perceptible map space, allowing users to intuitively observe the visualization results corresponding to different representations, so as to analyze the information contained in the deep representation and provide explanations that are consistent with people's cognition, thereby improving the interpretability, credibility and performance of the relevant deep neural network. Summary of the Invention

[0005] This paper provides a visualization analysis method for deep learning representations of map elements and constructs a clear, intuitive, easy-to-use, and highly scalable analysis system, thereby improving the interpretability of deep learning representations of map element data and providing support for the design and optimization of deep neural networks related to map elements. Details are as follows:

[0006] First, the present invention provides a visualization analysis method for deep learning representation of map elements, which mainly includes the following three steps:

[0007] S1. Generative model training based on the encoder-decoder structure. The specific process is as follows:

[0008] S11. Loading the model to be interpreted and the original dataset: The model to be interpreted usually refers to a deep neural network that has been trained and can output different types of map feature representations; the original dataset refers to the data involved in the training and testing of the original model; here, the present invention uses the model to be interpreted as the encoder, and the original dataset is used for subsequent decoder training.

[0009] S12. Construction of a universal vector data decoder based on deformation concept: Map elements include point, line, and surface elements, among which line and surface elements are also composed of ordered points.

[0010] Therefore, the present invention abstracts all types of map element data into a collection of spatial points, and the design of the decoder also uses points as the basic operation unit to meet the universal requirements of vector data operations. Deep neural networks usually extract high-level features of the input data and cannot fully retain low-level position and topological information. Therefore, the decoder draws on the idea of ​​deformation, and the input consists of two parts: 1) points, lines or surfaces with basic spatial and semantic information and topological relationships, which are called basic elements; 2) deep learning representations output by the encoder. The training goal of the decoder is to find a deformation operation for each point that constitutes the basic element based on the corresponding deep learning representation information, and then make the map elements generated by the decoder as consistent as possible with the input elements of the encoder through transformation.

[0011] S13. Loss function design and data preprocessing: This invention follows the principle of vector data priority and adopts the Chamfer distance for calculating the similarity of point sets as the basic loss function to calculate the similarity between the encoder input and the decoder output. For line features or surface features, length and area can also be introduced as reconstruction constraints. In addition, preprocessing operations such as point resampling and raster data vectorization are performed on long line segments of line and surface features to meet the input requirements of the decoder. If the input information contains semantic information, the semantics need to be quantified to ensure that the semantic information is measurable so that the transformation operations in the decoding process have physical meaning.

[0012] S14. Encoder-Decoder Training: Connect the model to be interpreted to the decoder, load the original data, and optimize the decoder parameters through backpropagation and iterative training to minimize loss. During training, the encoder only outputs deep representations; its parameters are not updated. The trained decoder can map the deep learning representations to a map space, providing support for subsequent visualization analysis.

[0013] S2. Align deep learning representations with cognizable concepts based on visual analysis. The specific process is as follows:

[0014] S21. Define the transformation operation of deep learning representation and apply decoding:

[0015] A deep learning representation can be viewed as a point in a feature space. A series of feature space transformations can be defined, including translation, rotation, subtraction, and generating unit vectors. By applying different transformations to alter the values ​​of various dimensions of the deep learning representation, the transformed representation is then fed into a trained decoder to generate visually perceptible map features.

[0016] S22. Extracting cognizable concepts through visual analysis:

[0017] Observe how the visualization generated by the decoder changes. Through testing various cases, it is possible to summarize and extract recognizable concepts related to map elements at the geometric and semantic levels, such as specific shapes, frequencies, directions, sizes, spatial distribution patterns, levels, categories, etc.

[0018] S23. Cognitive Concept Alignment:

[0019] By analyzing the correspondence between transformation operations and changes in visualization results, each dimension of deep learning representation is matched with different cognizable concepts, and the paired deep representation dimensions and cognizable concepts are output.

[0020] S3. Verification of direct and indirect analytical results:

[0021] Direct verification: Based on the recognizable concepts extracted in the previous step, design map elements that contain the above concepts, input them into the decoder, and directly observe whether the changes in each dimension of the deep learning representation output by the decoder are consistent with the results of concept alignment.

[0022] Indirect verification: Apply deep learning representation to downstream classification tasks, use multi-layer perceptron to train the classifier, and then observe whether the importance of each dimension of deep learning representation when making specific classification predictions is consistent with the results of concept alignment.

[0023] During the specific implementation process, a certain number of direct and indirect verifications are performed for each paired deep representation dimension and cognizable concept, and each verification result is scored to provide a quantitative evaluation result to provide people with an intuitive explanation.

[0024] Secondly, the present invention provides a visualization analysis system for deep learning representation of map elements. The system consists of the following modules:

[0025] a) Encoder management module: This module is responsible for the management of the model to be interpreted (encoder), including deep learning model loading, storage, parameter setting, calling and other functions;

[0026] b) Data management module: This module is responsible for the management of training data and explanation cases, including data loading, preprocessing, storage and retrieval functions;

[0027] c) Decoder management module: This module is used for decoder management, including decoder parameter setting, basic element selection, decoder calling and other functions;

[0028] d) Self-supervised training module: controls the decoder training process, including setting training parameters, loading data sets, loss function selection, basic element setting, etc.

[0029] e) Concept alignment module: Serves the process of aligning deep learning representations with cognizable concepts based on visual analysis, including: definition and execution of representation transformation operations, concept extraction and storage, visual interaction of map elements, and alignment result management.

[0030] f) Result verification module: serves the analysis result verification process, including: deep representation visualization, classifier training, visualization of classifier weights, scoring result statistics and other functions.

[0031] Compared with the existing technology, the present invention has the following beneficial technical effects: the present invention can visualize deep learning representations in map language, provide users with intuitive analysis materials, and is suitable for a large number of map practitioners; the present invention adopts a self-supervised learning architecture, which can effectively avoid data labeling and reduce the training burden of the generation model; the present invention can be applied to various map elements such as points, lines, and surfaces, and has a certain adaptability; the present invention is not limited to interpreting a specific type of deep neural network and has a certain versatility; the present invention can provide an intuitive explanation for the deep learning representation of map elements, which is conducive to the evaluation, optimization and application of deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the embodiments or the description of the prior art. The drawings described below are used together with the following detailed description to explain the present invention, but do not constitute a limitation of the present invention.

[0033] Figure 1 Schematic diagram of the map deep learning representation visualization analysis method and system provided by the present invention.

[0034] Figure 2 This is a flow chart of the generative model training based on the encoder-decoder structure provided by the present invention.

[0035] Figure 3 This is a schematic diagram of the decoding process of the building footprint provided by the present invention.

[0036] Figure 4Schematic diagram of the encoder-decoder framework provided by the present invention

[0037] Figure 5 A schematic diagram of the cognitive concept alignment process provided by the present invention.

[0038] Figure 6 Case study of building element representation concept alignment provided by this invention

[0039] Figure 7 Schematic diagram of the analysis result verification method provided by the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention are described in full below. The described embodiments are only some embodiments of the present invention. The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the present invention.

[0041] The map deep learning representation visualization analysis method proposed in this invention is as follows: Figure 1 As shown, there are three main steps:

[0042] S1. Generative model training based on encoder-decoder structure;

[0043] S2. Aligning deep learning representations with cognizable concepts based on visual analysis;

[0044] S3. Validation of direct and indirect analytical results.

[0045] also, Figure 1 The corresponding relationship between the modules and steps of the proposed visualization analysis system for deep learning representation of map elements is also shown. The encoder management module, data management module, and decoder management module are the foundational functional modules, which are called upon in each of the above steps. The self-supervised training module, concept alignment module, and analysis and verification module serve the aforementioned steps S1, S2, and S3, respectively.

[0046] First, step S1 generates a model through self-supervised strategy training, and then maps the deep learning representation of map features to the map space to meet visualization requirements. It can be divided into 4 sub-steps:

[0047] S11. Loading the model to be interpreted and the original dataset: Load a trained deep neural network capable of outputting different types of map feature representations through the encoder management module; load the dataset involved in training and testing the model to be interpreted through the data management module; here, the model to be interpreted will serve as the encoder, and the original dataset will serve as the training set for subsequent decoder training.

[0048] S12. Construction of a universal vector data decoder based on deformation ideas: Map elements are usually divided into point, line, and surface elements, among which line and surface elements can also be regarded as composed of ordered points. Therefore, the present invention abstracts all types of map element data into a set of points, and the decoder also takes a multi-dimensional point set as input to meet the universal requirements of map element data operations. The deep neural network to be interpreted usually extracts high-level features of the input data and cannot fully retain low-level position and topological information. To this end, the decoder introduces points, lines or surfaces with basic spatial and semantic information and topological relationships as input on the basis of deep learning representation. The present invention calls them basic elements. Figure 3 As shown, if the feature to be decoded is a building footprint, you can choose an isotropic circle as the basic feature. The points maintain an orderly connection relationship and are connected end to end, which is topologically consistent with the building footprint; if the input is a line feature, you can choose a straight line as the basic feature; if the input data is a point of interest, you can choose a set of points evenly distributed in space as the basic feature; there is no absolute restriction on the choice of basic features, but it is recommended to choose uniform and simple features as much as possible. Figure 3 The decoding process of the deep-learned representation of building footprints is demonstrated: First, the decoder replicates the k-dimensional deep-learned representation of building footprints based on the number m of points in the base element, constructing an m×k-dimensional feature matrix. The decoder then concatenates the 2D spatial features of the base element to construct an m×(k+2)-dimensional feature matrix, which is then fed into a deep neural network constructed for point set transformation. Through training, the network can map each row of k+2 vectors to a 2D map space, thereby generating a new building footprint. This process can be viewed as the decoder spatially transforming each point of the base element in the map space based on the deep-learned representation of the building footprint, deforming the base element so that the new map element generated by the decoder is as similar as possible to the encoder input element. If each base element point also contains j-dimensional quantized semantic information, the m×(k+2)-dimensional feature matrix can be set to m×(k+2+j) dimensions, and the output of the deep neural network for point set transformation should also be set to 2+j dimensions.

[0049] S13. Loss function design and data preprocessing: This paper follows the principle of vector data priority and regards map elements as a set of points. Therefore, when measuring the similarity between the encoder input and the decoder output, the Chamfer distance based on the point set is used as the loss function L chamfer :

[0050]

[0051] Where I represents the input point set and O represents the output point set. This loss function is often used to measure the similarity between point sets and is not constrained by the number of input and output points. If the decoded features are line features and surface features, length and area can also be introduced as reconstruction constraints respectively:

[0052] L length =‖Length I -Length O ‖

[0053] L area =‖Area I -Area O ‖

[0054] The total loss function is:

[0055] L t2tal =L chamfer +λ l L length +λ a L area

[0056] It can be calculated based on the input element type and task scenario. l and λ a Set different values.

[0057] In addition, if the input data is a line or surface feature, it is necessary to resample the points of the longer line segments to ensure that the points representing the line segments can provide sufficient spatial information. If the input map feature is raster data, it is necessary to convert it into vector data when calculating the loss function. If the input information contains semantic information, it is necessary to quantify the semantics to ensure that the semantic information is measurable so that the transformation operation in the decoding process has physical meaning. For example, the point of interest p i The type can use the pre-trained natural language model to generate low-dimensional embeddings i , so that the distance between embeddings ||e i -e j ||2 can represent p i and p j The similarity of types.

[0058] S14. Encoder-decoder training: Figure 4 As shown, the model to be explained is connected as an encoder and a decoder, the original data is loaded, the encoder generates a deep learning representation, and the decoder maps the representation back to the map space and then brings it into L t2tal Calculate the similarity between input and output, optimize the decoder parameters through back propagation and iterative training, so that L t2tal As small as possible. During training, the encoder only outputs the deep representation and does not participate in parameter updates. The trained decoder can map the deep learning representation to the map space, providing support for subsequent visual analysis.

[0059] After completing the above generation model training, you can enter step S2. Figure 5 As shown, this step can be divided into 3 sub-steps:

[0060] S21. Define the transformation operation of deep learning representation and apply decoding:

[0061] A deep learning representation can be viewed as a point in a feature space D. A series of feature space transformations can be defined, including translation, rotation, subtraction, and generating unit vectors. First, a representative set of map features is selected and a deep learning representation is generated using an encoder. Next, the defined feature space transformations are applied to change the values ​​of different dimensions of the deep learning representation. Finally, the trained decoder is used to visualize the representation.

[0062] S22. Extracting cognizable concepts through visual analysis:

[0063] Observe how the decoder output visualization changes under different transformations. Based on the task scenarios with the explanation model, continuously test and iterate various cases. Then, summarize and extract recognizable concepts related to map elements from the geometric and semantic levels, such as specific shapes, frequencies, directions, sizes, spatial distribution patterns, levels, categories, etc., and ultimately form a set of recognizable concepts C for the corresponding task.

[0064] S23. Cognitive Concept Alignment:

[0065] Analyze the correspondence between transformation operations and changes in visualization results, and correspond each dimension of deep learning representation to different cognizable concepts or features, thereby providing a human-understandable explanation of the deep learning representation space.

[0066] Various cases can be selected, and the above steps can be iterated multiple times to ensure the stability of the concept alignment results. It should also be noted that if the deep learning representation is not a decoupled representation, it may not be possible to achieve a single dimension corresponding to a single concept, but rather several dimensions jointly corresponding to the change of a single concept. The analysis at this stage needs to make full use of people's visual perception ability and the decoder's visualization ability for deep learning representations, and continuously iterate and optimize through human-computer visual interaction. Finally, the correspondence relationship R{r1, r2, r 3, …, r i}:

[0067]

[0068] where r i is a tuple <d p , C k > or <d p , …, d q , C k > composed of the dimensions of the deep learning representation and the recognizable concept. Figure 6 is a case of the visualization analysis of the deep learning representation of Y-shaped building elements. By applying translation operations to dimension 1 and dimension 3 and decoding with the trained decoder, it can be observed that dimension 1 and dimension 3 of the representation respectively correspond to the two concepts of size and direction, that is, the tuples <d1, "size"> and <d3, "direction"> are generated.

[0069] After completing the construction of the above correspondence relationship, enter step S3, and further verify the correspondence relationship output by S2 in two ways:

[0070] Figure 7 (a) of shows the direct verification method: First, according to the recognizable concepts or features extracted in the previous link, design typical map elements containing the above concepts or features; then, input the decoder, and then visualize the deep learning representation and observe whether the values of each dimension conform to the results of concept alignment. For example, if one dimension represents the concept of "building size", then building footprints with different sizes can be designed and input into the decoder to observe whether the value of this dimension changes.

[0071] Figure 7(b) shows an indirect verification method: deep learning representations are applied to downstream classification tasks, such as POI pattern recognition, road pattern recognition, and building footprint shape classification, using a multi-layer perceptron to train a classifier. The classifier weights are then visualized to observe whether the importance of each dimension of the deep learning representation when making a specific classification prediction is consistent with the results of concept alignment. For example, a deep learning representation can be used to construct a classifier for building footprint shape. After inputting test cases, the classifier weights can be observed to determine whether specific dimensions play a role in the classifier's correct prediction.

[0072] In this step, m direct verifications and n indirect verifications are set for each corresponding relationship, and each verification result is scored g. The score of concept alignment verification can be calculated by the following formula:

[0073]

[0074] Finally, the set of corresponding relationships and scores can be output:

[0075] {<r0,G0> ,<r1,G1> ,<r2,G2> ,…, <r i ,G i >}

[0076] Verification result G i The score also represents the corresponding relationship r i This set can serve as an explanation scheme for the deep learning representation of map features and provide a reference for subsequent deep learning model evaluation, optimization, and application.

[0077] In general, this paper fully considers the characteristics of map feature data, the user's knowledge background and interaction habits, and uses the self-supervision strategy of the encoder-decoder structure and the visual analysis method to provide an intuitive and highly versatile solution for interpreting the deep learning representation of map features.

[0078] The foregoing description is merely a specific embodiment of the present invention, and the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein are intended to be encompassed by the scope of protection of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visualization analysis method for deep learning representation of map elements, characterized by: The following steps are involved: S1. Generative model training based on the encoder-decoder structure includes the following sub-steps: S11. Load the model to be interpreted and the original dataset. The model to be interpreted is a deep neural network that has been trained and can output representations of different types of map elements, serving as the encoder for this step. The original dataset will serve as the training set for subsequent decoder training. S12. Construct a universal vector data decoder based on deformation. The decoder input consists of two parts: (1) points, lines, or surfaces containing basic spatial information, semantic information, and topological relationships, which are called basic elements; (2) deep learning representations output by the deep learning model of the map elements to be interpreted; the training goal of the decoder is to generate a corresponding deformation operation for each point that constitutes the basic element based on the deep learning representation information, and then through the deformation operation, make the map elements generated by the decoder as consistent as possible with the input elements of the encoder; S13. Design a loss function and perform data preprocessing, wherein the loss function uses the Chamfer distance for calculating the similarity of a point set as the basic loss function, calculates the similarity between the encoder input and the decoder output, and can be directly applied to point features. For line features, the length needs to be introduced as a reconstruction constraint, and for surface features, the perimeter and area need to be introduced as reconstruction constraints. Data preprocessing includes three aspects: (1) resampling the points of the long line segments that constitute the line and surface features to ensure that the points that constitute the line segments can provide sufficient spatial information; (2) if the input data of the encoder is raster data, the raster data needs to be vectorized when calculating the loss; (3) if the input data contains semantic information, the semantics needs to be quantized to ensure that the semantic information can be measured; S14. Train a universal vector data decoder using an encoder-decoder structure. During the training process, the model to be interpreted is connected to the decoder as an encoder and loaded with the original dataset. The encoder only outputs the deep learning representation. The decoder parameters are optimized by minimizing the loss function designed in step S13. The trained decoder is able to convert the deep learning representation of map elements output by the model to be interpreted into visually perceptible map elements for subsequent visualization analysis. S2. Aligning deep learning representations with cognizable concepts based on visual analysis, including the following sub-steps: S21. Define transformation operations for the deep learning representations of map features, and use the trained decoder to convert the transformed deep learning representations into visual map features. Representation transformation operations include translation, rotation, subtraction, and generating unit vectors. Different transformation operations can change the values ​​of various dimensions of the deep learning representations. S22. Extract recognizable concepts through visual analysis. Based on the task characteristics of the model to be interpreted, observe how the decoder's visualization output changes with the transformed deep learning representation. Extract recognizable concepts related to map elements from the geometric and semantic levels, including specific shapes, frequencies, directions, sizes, spatial distribution patterns, levels, and categories, ultimately forming a set of recognizable concepts for the corresponding task. S23. Cognitive concept alignment, where alignment involves associating each dimension of the deep learning representation of map elements output by the model to be interpreted with the cognizable concepts by analyzing the correspondence between transformation operations and changes in visualization results. This ultimately yields a correspondence between the feature space of the deep learning representation of the model to be interpreted and the set of cognizable concepts, serving as an explanation of the model to be interpreted. S3. Direct and indirect verification of analytical results, including the following two verification methods: S31. Direct Verification: Based on the set of recognizable concepts extracted in step S22, design map feature data that includes these concepts, input them into the decoder, and directly observe whether the numerical changes in each dimension of the deep learning representation output by the decoder conform to the results of the concept alignment in step S23; S32. Indirect verification: Apply the deep learning representation to a downstream classification task, train a classifier using a multilayer perceptron, and then observe whether the weights assigned to different dimensions of the deep learning representation in the classifier when making a specific classification prediction are consistent with the concept alignment results from step S23. This step performs a certain number of direct and indirect verifications for each paired deep representation dimension and cognizable concept, scores each verification result, and then calculates the mean of all scoring results as the score for the correspondence. It then outputs a set of each correspondence and score as a quantitative evaluation of the interpretation result obtained in step S23.

2. A visualization analysis system for deep learning representation of map elements, characterized by: include: At least one processor, at least one memory, at least one display, a communication interface, and a bus; wherein the processor, memory, display, and communication interface communicate with each other via the bus; The memory stores one or more program instructions, the processor executes and calls the program instructions, and the display visualizes the execution results of the program instructions to implement the method of claim 1 and perform visual analysis of deep learning representations of map elements according to the method.

Images