A method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction

Abstract

The present invention discloses a method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph simplification. The method performs manifold approximation on high-dimensional data to generate a weighted undirected graph, and then simplifies the graph to obtain a thumbnail of the weighted graph. The process can keep the high-dimensional data, the weighted graph and the thumbnail having the same topological structure, and the interpretability is strong. The grid where the image space is located is meshed based on the resolution parameter. Since the intersection between the grid elements is empty, multiple mapping features located in the grid elements do not intersect, so that the topological structure of the thumbnail obtained by aggregating the mapping feature subsets is the same as the topological structure of the high-dimensional data, and thumbnails of different resolutions can be obtained by adjusting the resolution parameter, so that any resolution parameter can be adjusted.

Description

Technical Field

[0001] The present invention belongs to the field of high-dimensional data visualization, and in particular relates to a high-dimensional data topological structure visualization method based on manifold approximation and graph reduction. Background Art

[0002] With the development of science and technology and the continuous expansion of application scenarios, a large amount of data has been generated in many different fields. For example, in the field of medical health, there are patient medical records, gene sequences, epidemiological data, medical images, drug development information and other data; in the field of scientific research, there are scientific research data, observation results, simulation data and other data; in the field of resource management, there are geographic information data, environmental monitoring data and other data; in the field of e-commerce, e-commerce platforms have accumulated a large amount of user behavior data, transaction data, commodity information, etc.; in the public field, there are information on public services, urban management, environmental protection, public safety and other aspects; in the field of media, there are users' reading habits, content consumption preferences, social media interactions and other data; in the field of security, there are video surveillance, face recognition, behavior analysis and other data; in the field of transportation, there are traffic flow, public transportation data, accident data and other data; in the field of finance, there are transaction records, credit information, market data, investment portfolio data, economic indicators and other data.

[0003] These data are not only huge in scale, but also diverse and growing rapidly, bringing unprecedented opportunities and challenges to data analysis in various industries. Data has become the core asset in various fields. Through data analysis, more accurate decision-making support can be provided for enterprises and institutions, promoting innovation and development of the industry.

[0004] Data visualization is the most commonly used and important component in the data analysis process. Through graphical means, data visualization can not only reveal the inherent patterns, trends, outliers and other information of the data, but also help users intuitively display and understand the key information in the data, greatly improving the efficiency and accuracy of information interpretation. In a broad sense, data visualization refers to the graphical representation of data. For example, line graphs, scatter plots, histograms, pie charts, box plots, etc. In a narrow sense, data visualization refers to the visualization of data topological structure, that is, embedding data (usually high-dimensional) into a low-dimensional space, such as a two-dimensional plane or a three-dimensional space, to discover or present the topological structure of the data in the original space, such as clustering, connectivity and other features. The method invented by this patent application belongs to the latter, that is, a visualization technology for the topological structure of high-dimensional data.

[0005] According to the different visualization results, the visualization methods of the topological structure of high-dimensional data are mainly divided into two categories: projection and graph abstraction. The projection method embeds each data point in the high-dimensional space into the low-dimensional space, while trying to keep the global structure or local topological structure of the data in the original space consistent with the structure in the low-dimensional space. Therefore, in the field of machine learning, the projection method is also called dimensionality reduction or embedding. Some commonly used projection methods include principal component analysis (PCA), multi-dimensional scaling (MDS), t-distributed stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP). The graph abstraction method will generate a graph, also known as a network, in which the nodes in the graph represent a set of points in the high-dimensional data, and the edges between the nodes represent the adjacent relationship between the point sets or the existence of common data points. The graph is used to display the topological structure of the high-dimensional data, and then the graph layout algorithm is used to embed the graph into the low-dimensional space, so as to achieve the purpose of visualizing the topological structure of the data. Typical representatives of graph abstraction methods are Mapper and its various improvements in the field of topological data analysis, such as F-Mapper, G-Mapper, etc., and its trajectory inference method PAGA (Partition-Based Graph Abstraction) for single-cell sequencing data.

[0006] Although projections in visualization methods can preserve some structure in the data, they are not very good at preserving both global and local structures simultaneously.

[0007] People tend to observe images at different resolutions to perceive features at different scales. Therefore, observers usually reduce the image to observe global information and enlarge the image to observe local details. In the visualization algorithm of high-dimensional data topology, projecting all high-dimensional data points into different points in low-dimensional space can be regarded as the highest resolution. Gluing and aggregating close or similar nodes in high-dimensional space and projecting them into fewer points in low-dimensional space can be regarded as medium resolution. Gluing and aggregating all high-dimensional data points into a whole and projecting them to a point in low dimension can be regarded as the lowest resolution. The projection method only provides the highest resolution result, while the graph abstraction method is sensitive to the resolution parameter and can only adjust the resolution parameter within a small range. Slightly larger or smaller resolution parameters will affect the clustering method on which the algorithm relies, resulting in unstable calculation results.

[0008] Both projection and graph abstraction methods have the phenomenon that the results are not easy to explain. For example, points with close distances in the PCA results cannot guarantee that the points are also close in the original high-dimensional space. It is possible that the algorithm compresses points with far distances during the dimensionality reduction process, resulting in close distances in the low-dimensional space. Non-adjacent clusters in the MDS, t-SNE, and UMAP results cannot be explained as the two clusters being far apart in the high-dimensional space. It is possible that the two clusters are very close but sparsely connected, resulting in the phenomenon of being far away during the dimensionality reduction process. Each node in the results of the graph abstraction method Mapper and its improved method represents a set of high-dimensional data points. However, there may be intersections between the point sets corresponding to the two nodes, that is, a high-dimensional data point may correspond to multiple low-dimensional graph nodes at the same time, so that the network in the method is not a division of high-dimensional data points, nor a cluster, making the results difficult to understand and apply. Summary of the invention

[0009] The present invention provides a method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction. The method has good interpretability, has the same global and local topological structures, and can arbitrarily adjust resolution parameters.

[0010] The present invention provides a method for visualizing a high-dimensional data topological structure based on manifold approximation and graph reduction, comprising:

[0011] Convert high-dimensional datasets into weighted undirected graphs through manifold approximation methods;

[0012] The high-dimensional data set is mapped to the image space to obtain a corresponding mapping feature set, and the image space is divided into a grid composed of multiple grid elements based on the obtained resolution parameter, and the intersection between the grid elements is empty, and multiple mapping features located in the grid elements are used as mapping feature subsets, and multiple mapping feature subsets construct a mapping feature set;

[0013] Aggregate each mapping feature subset to obtain the corresponding node. If there is an undirected edge at the node corresponding to the mapping feature subset in the weighted undirected graph, there is an undirected edge between the aggregated nodes. The maximum value or average value of the weights of the undirected edges of the nodes corresponding to the mapping feature subset in the weighted undirected graph is used as the weight of the undirected edge between the aggregated nodes. A thumbnail is constructed from the aggregated nodes, the undirected edges between the nodes, and the corresponding weights to achieve graph simplification.

[0014] A force-directed graph layout method is used to project the thumbnails to obtain a visual graph layout.

[0015] Preferably, the high-dimensional data set is converted into a weighted undirected graph by a manifold approximation method, comprising:

[0016] The high-dimensional data is converted into an unweighted directed KNN graph, and the kernel function is used to assign weights to each directed edge based on the distance between nodes. The directed edge between two nodes is converted into an undirected edge, and the weights of the directed edges in different directions between the two nodes are fused to obtain the weight of the undirected edge between the two nodes, thereby obtaining a weighted undirected graph.

[0017] Preferably, the weights of directed edges of different directions between two nodes are merged to obtain the weight of the undirected edge between the two nodes, where i Nodes and j Undirected edges between nodes for:

[0018] ,in, For the i Node to j The weight of the directed edge of the node, For the j Node to i The weight of the directed edge of each node.

[0019] Preferably, the weights of directed edges of different directions between two nodes are merged to obtain the weight of the undirected edge between the two nodes, where i Nodes and j Undirected edges between nodes for:

[0020] ,in, For the i Node to j The weight of the directed edge of the node, For the j Node to i The weight of the directed edge of each node.

[0021] Preferably, a kernel function is used to assign weights to each directed edge based on the distance between nodes, where i Node to j The weight of the directed edge of the node for:

[0022] ,in is the kernel function, Representation Node i and nodes j Coordinate distance in high-dimensional space, Representation Node i The normal factor of .

[0023] Preferably, converting the high-dimensional data set into an unweighted directed KNN graph comprises:

[0024] The data point of the high-dimensional data set is taken as the current node, the set of K nearest neighbor nodes of the current node is obtained, the edge from the current node to the nearest neighbor node is taken as the directed edge, and an unweighted directed KNN graph is constructed based on the current node, the nearest neighbor node and the directed edge.

[0025] Preferably, the high-dimensional data set manifold is approximated as a weighted undirected graph, wherein G for: ,in, V is a node set, representing a high-dimensional data set X The data points in E is an undirected edge set, is the weight function of the undirected edge, , the larger the weight of the undirected edge, the closer the distance between the corresponding two nodes is.

[0026] Preferably, if the image space is one-dimensional, each grid element is a half-open and half-closed interval;

[0027] If the image space is two-dimensional, each grid element is a half-open, half-closed square;

[0028] If the image space is three-dimensional or higher, each grid element is the analog of a half-open, half-closed interval or a square in three-dimensional or higher dimensions.

[0029] Preferably, a feature mapping method is used to map the high-dimensional data set to the image space to obtain a corresponding mapping feature set, and the feature mapping method is principal component analysis of one or more components, projection on a coordinate axis, an axial plane or an axial space, kernel density estimation or identity mapping.

[0030] Preferably, the number of grid elements corresponds to a resolution parameter.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The method provided by the present invention performs manifold approximation on high-dimensional data to generate a weighted undirected graph, and then simplifies the graph to obtain a thumbnail of the weighted undirected graph. This process can keep the high-dimensional data, the weighted undirected graph and the thumbnail with the same topological structure, and has strong interpretability.

[0033] The present invention divides the grid where the image space is located into grids based on the resolution parameter. Since the intersection between grid elements is empty, multiple mapping features located in the grid elements do not intersect, so that the topological structure of the thumbnail obtained by aggregating the mapping feature subsets is the same as the topological structure of the high-dimensional data, and thumbnails of different resolutions can be obtained by adjusting the resolution parameter, thereby achieving adjustable resolution parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for visualizing a high-dimensional data topology structure based on manifold approximation and graph reduction provided by a specific embodiment of the present invention;

[0035] Figure 2 A schematic diagram of manifold approximation provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0037] In the face of the technical problems that the existing high-dimensional data topological structure visualization technology has poor interpretability, the resolution parameters cannot be adjusted arbitrarily, and the global and local topological structures corresponding to the low-dimensional results and high-dimensional data points are different, the present invention approximates the high-dimensional data set manifold as a weighted undirected graph, and represents the distance between the high-dimensional data by the weighted value of the undirected edge obtained. At the same time, the high-dimensional data is mapped to the image space through feature mapping, and grid division is performed based on the resolution parameter, so that the number of grid elements can be changed accordingly based on the regulation of the resolution parameter, that is, the more the number of grid elements, the fewer the aggregated mapping features, the more nodes obtained by aggregation, and the higher the resolution, thereby realizing arbitrary resolution parameter adjustment. In addition, the present invention determines the nodes of the thumbnail and its edges and corresponding weights based on the weighted undirected graph, so that the thumbnail is the same as the high-dimensional data topological structure, and the thumbnail of the low-dimensional space can also be regarded as a classification of the high-dimensional space data point. Each node of the graph represents a data point set in the high-dimensional space, and different nodes correspond to different and non-intersecting data point sets. The edges between the nodes in the graph represent that different data point sets have an adjacent relationship, and the interpretability is strong.

[0038] The specific embodiment of the present invention provides a method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction, such as Figure 1 As shown, including:

[0039] S1. Converting a high-dimensional data set into a weighted undirected graph by a manifold approximation method: The specific embodiment of the present invention is based on the manifold assumption, that is, assuming that the data points in the high-dimensional space are located on a low-dimensional manifold, such as Figure 2 As shown in Figure 2, by generating a weighted undirected graph from a high-dimensional dataset G , to approximate the manifold on which the data lies.

[0040] The weighted undirected graph provided by the specific embodiment of the present invention G for: ,in, V is a node set, representing a high-dimensional data set X The data points in Eis an undirected edge set, indicating that the data points have a close relationship. is the weight function of the undirected edge, , the larger the weight of the undirected edge, the closer the distance between the corresponding two nodes is.

[0041] The specific embodiment of the present invention provides a method for converting a high-dimensional data set into a weighted undirected graph by a manifold approximation method, including:

[0042] S11. Construct an unweighted directed KNN graph (K-Nearest Neighbors Graph): The nodes in the graph represent the data set X Data points in the graph, nodes in the graph i To Node j Directed edge Represents data points j Belongs to data points i The set of K nearest neighbors.

[0043] S12, Convert to a weighted directed KNN graph: In the obtained weighted directed graph, the nodes and directed edges are consistent with those in step S11. The difference is that the kernel function is used to assign weights to the unweighted directed edges. Here, the nodes i To Node j Directed edge The weight is: , where the function is the kernel function, which converts distance into probability. Representation Node i and nodes j Coordinate distance in high-dimensional space, Represents data points i The normal factor at .

[0044] S13, convert to weighted undirected graph again G :Based on the weighted directed graph obtained in step S12, if and only if every pair of nodes i and j There are directed edges between or , convert it into an undirected edge , and its weight can be: , indicating a node i and nodes j The probability that there is at least one directed edge between them. Or , indicating a directed edge and The mean of .

[0045] S2. Simplify the weighted undirected graph into a thumbnail: The specific embodiment of the present invention creates a weighted undirected graph G A thumbnail ofH , the thumbnail is also a weighted undirected graph, similar to the original image G In contrast, thumbnails H Has fewer nodes and edges, but usually has the same G Same connectivity. Thumbnail H Each node in represents a subset of data points, and the thumbnail H There is no intersection between the data point subsets corresponding to different nodes in . Therefore, the thumbnail H It can be regarded as a division of the original data, and at the same time has the same topological structure as the original data, thereby effectively extracting the topological structure of the original data. The specific simplification steps of this stage are as follows:

[0046] S21. Extract features of high-dimensional data using feature mapping to obtain a mapping feature set: Use the specified feature mapping to map the original high-dimensional data set to the image space, thereby extracting the corresponding mapping feature set. The specified feature mapping provided in the specific embodiment of the present invention is a PCA (principal component analysis) with one or more components, a projection on a coordinate axis, an axial plane or an axial space, a kernel density estimation, an identity mapping, etc. Even some common dimensionality reduction methods can be used as feature mapping.

[0047] S22, meshing the image space: using a given resolution parameter, the image space is divided into a grid, and the grid provided by the specific embodiment of the present invention makes the image of the high-dimensional data point under the feature map all located in the grid. The grid is divided into multiple units based on the resolution parameter, and each unit in the grid is called a grid element. The intersection of any two grid elements is empty, and the union of all grid elements is the entire grid, such as Figure 1 As shown, the resolution parameter is 2*2, the number of grid elements is 4, and multiple mapping features located in the grid elements are taken as mapping feature subsets, and multiple mapping feature subsets construct a mapping feature set.

[0048] In a specific embodiment, if the image space is one-dimensional, the grid elements are half-open and half-closed ( or ,in a and b Represent the two ends of the interval respectively); if the image space is two-dimensional, the network element is half-open and half-closed ( or ,in and represents the two ends of the grid in the first dimension, and denoting the two ends of the grid in the second dimension); if the image space is three or higher dimensional, the grid elements are half-open, half-closed intervals and the analogues of the grid in higher dimensions.

[0049] S23. Using the partitioned grid and weighted undirected graph G Get thumbnail H : Aggregate the subset of mapping features within the grid elements to obtain thumbnails H If two mapping feature subsets are in a weighted undirected graph G If there is an undirected edge between the corresponding nodes in the graph, then there is an undirected edge between the nodes obtained by aggregation. If there are multiple undirected edges, the maximum or average value of the weights of the existing undirected edges is used as the weight of the undirected edges between the thumbnail nodes, so that the thumbnail H The topological structure of is the same as that of the high-dimensional dataset.

[0050] S3. Use force-directed graph layout method to project thumbnails to obtain visual graph layout: In this stage, a force-directed graph layout algorithm is used to project thumbnails of the dataset. H Projecting it into a low-dimensional space such as two-dimensional or three-dimensional, and using a method of visualizing thumbnails, the purpose of visualizing the topological structure of high-dimensional data is achieved. H The data set, that is, the aggregated nodes, undirected edges and corresponding weights are input into a force-directed graph layout algorithm (Force-directed graph layout) to obtain a visualized graph layout. The force-directed graph layout algorithms include: Fruchterman-Reingold method, Kamada-Kawai method, ForceAtlas2 method, Spring-Electrical method, t-SNE, etc.

[0051] The high-dimensional data topological structure visualization method based on manifold approximation and graph simplification provided by the present invention can make data points with similar distances in high-dimensional space still close to the corresponding same nodes in low-dimensional space, while data points with relatively large distances in high-dimensional space are not adjacent to the corresponding different nodes in low-dimensional space, so as to keep the low-dimensional result and the high-dimensional data point having the same global and local topological structure; it can also be that when the resolution parameter is large, there are the same number of nodes in the result as the original data points, and the high-dimensional data points correspond to the low-dimensional nodes one by one, when the resolution parameter is moderate, the nodes in the result will also be appropriately reduced, when the resolution parameter continues to decrease, the number of nodes in the result will continue to decrease, and even drop to one node, so that the resolution parameter can be adjusted arbitrarily; the result of the present invention is a graph in a low-dimensional space, which can be regarded as a classification of high-dimensional space data points, and can also be regarded as a thumbnail of a data set. Each node in the graph represents a data point set in a high-dimensional space, different nodes correspond to different and non-intersecting data point sets, and the edges between the nodes in the thumbnail represent that different data point sets in the high-dimensional space have adjacent relationships, so as to have better interpretability.

Claims

1. A method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction, characterized in that: include: Convert high-dimensional datasets into weighted undirected graphs through manifold approximation methods; The high-dimensional data set is mapped to the image space to obtain a corresponding mapping feature set, and the image space is divided into a grid composed of multiple grid elements based on the obtained resolution parameter, and the intersection between the grid elements is empty, and multiple mapping features located in the grid elements are used as mapping feature subsets, and multiple mapping feature subsets construct a mapping feature set; Aggregate each mapping feature subset to obtain the corresponding node. If there is an undirected edge at the node corresponding to the mapping feature subset in the weighted undirected graph, there is an undirected edge between the aggregated nodes. The maximum value or average value of the weights of the undirected edges of the nodes corresponding to the mapping feature subset in the weighted undirected graph is used as the weight of the undirected edge between the aggregated nodes. A thumbnail is constructed from the aggregated nodes, the undirected edges between the nodes, and the corresponding weights to achieve graph simplification. A force-directed graph layout method is used to project thumbnails to obtain a visualized graph layout; If the image space is one-dimensional, the grid elements are half-open and half-closed intervals; If the image space is two-dimensional, the network elements are half-open and half-closed squares; If the image space is three-dimensional or higher, then the grid elements are half-open, half-closed intervals and the higher-dimensional analogues of squares.

2. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 1, characterized in that: Convert high-dimensional datasets into weighted undirected graphs through manifold approximation methods, including: The high-dimensional data is converted into an unweighted directed KNN graph, and the kernel function is used to assign weights to each directed edge based on the distance between nodes. The directed edge between two nodes is converted into an undirected edge, and the weights of the directed edges in different directions between the two nodes are fused to obtain the weight of the undirected edge between the two nodes, thereby obtaining a weighted undirected graph.

3. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 2, characterized in that: The weights of directed edges of different directions between two nodes are merged to obtain the weight of undirected edges between two nodes, where i Nodes and j Undirected edges between nodes for: ,in, For the i Node to j The weight of the directed edge of the node, For the j Node to i The weight of the directed edge of each node.

4. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 2, characterized in that: The weights of directed edges of different directions between two nodes are merged to obtain the weight of undirected edges between two nodes, where i Nodes and j Undirected edges between nodes for: ,in, For the i Node to j The weight of the directed edge of the node, For the j Node to i The weight of the directed edge of each node.

5. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 2, characterized in that: The kernel function is used to assign weights to each directed edge based on the distance between nodes, where i Node to j The weight of the directed edge of the node for: , where the function is the kernel function, Representation Node i and nodes j Coordinate distance in high-dimensional space, Representation Node i The normal factor of .

6. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 2, characterized in that: Convert a high-dimensional data set into an unweighted directed KNN graph, including: The data point of the high-dimensional data set is taken as the current node, the set of K nearest neighbor nodes of the current node is obtained, the edge from the current node to the nearest neighbor node is taken as the directed edge, and an unweighted directed KNN graph is constructed based on the current node, the nearest neighbor node and the directed edge.

7. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 1 or 2, characterized in that: The high-dimensional data set manifold is approximated as a weighted undirected graph. G for: ,in, V is a node set, representing a high-dimensional data set X The data points in E is an undirected edge set, is the weight function of the undirected edge, , the larger the weight of the undirected edge, the closer the distance between the corresponding two nodes is.

8. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 1, characterized in that: A feature mapping method is used to map the high-dimensional data set to the image space to obtain a corresponding mapping feature set. The feature mapping method is principal component analysis of one or more components, projection on a coordinate axis, an axis plane or an axis space, kernel density estimation or identity mapping.

9. The method for visualizing the topological structure of high-dimensional data based on manifold approximation and graph reduction according to claim 1, characterized in that: The number of grid elements corresponds to the resolution parameter.

Images