Data sampling quality evaluation method and system for scatter diagram visualization task

Abstract

The invention relates to the field of data visualization, in particular to a scatter diagram visualization task-oriented data sampling quality evaluation method and system. The method comprises the following steps: giving a high-dimensional data set, and generating a high-dimensional data subset by using a data sampling method; obtaining a low-dimensional data subset according to the high-dimensional data subset; randomly extracting clusters of the high-dimensional data set to obtain high-dimensional space sub-clusters; obtaining a low-dimensional space sub-cluster according to the high-dimensional space sub-cluster; measuring the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster to obtain the difference between different dimensional space sub-cluster pairs; obtaining a global error according to the difference of the sub-cluster pairs in the different dimension spaces; and evaluating the sampling quality of the high-dimensional data through the global error. According to the method, by comparing the sub-clustering differences of different dimension spaces and measuring the retention conditions of different sampling methods after data projection, the authenticity of the high-dimensional data in the projection scatter diagram is effectively evaluated.

Description

Technical Field

[0001] The present invention relates to the field of data visualization, and in particular to a data sampling quality evaluation method and system for scatter plot visualization tasks. Background Art

[0002] In high-dimensional data visualization analysis, the choice of projection technology and scatter plot sampling technology is crucial to understanding high-dimensional data. Different projection technologies combined with different high-dimensional data sampling methods will produce different visualization effects for scatter plots.

[0003] In the prior art, sampling methods based on projection space and sampling methods based on high-dimensional space are usually used to perform visualization analysis of high-dimensional data. The sampling method based on projection space only considers the data distribution characteristics of the projection space, but does not consider the distribution characteristics of the original high-dimensional data, resulting in the problem of selection bias of sampling points; while the sampling method based on high-dimensional space solves the problem of selection bias of sampling points in the projection space, it is difficult to present a good visualization effect; for different projection technologies, although different high-dimensional data sampling methods will bring different effects, how to choose a high-dimensional data sampling method that matches the projection technology so that the scatter plot presents the best visualization effect is an urgent problem to be solved.

[0004] At present, there is not much research on the evaluation of high-dimensional data sampling quality, and there is no specific high-dimensional data sampling quality evaluation method for scatter plot visualization tasks. Summary of the invention

[0005] In view of the defects in the prior art, the present invention provides a data sampling quality evaluation method and system for scatter plot visualization tasks.

[0006] In the first aspect, the present invention provides a data sampling quality evaluation method for scatter plot visualization tasks, comprising the following steps: given a high-dimensional data set, using a data sampling method to generate a high-dimensional data subset; according to the high-dimensional data subset, a low-dimensional data subset is obtained; randomly extracting clusters of the high-dimensional data set to obtain high-dimensional space sub-clusters; according to the high-dimensional space sub-clusters, a low-dimensional space sub-clusters is obtained; measuring the difference between the high-dimensional space sub-clusters and the low-dimensional space sub-clusters to obtain the difference between the high-dimensional space and low-dimensional space sub-clusters; according to the difference between the high-dimensional space and low-dimensional space sub-clusters, a global error is obtained; through the global error, the evaluation of the sampling quality of high-dimensional data is realized. The present invention has the following effects: first, by sampling the high-dimensional data set and projecting it to the low-dimensional space, a better visualization effect is obtained; second, by comparing the difference between the high-dimensional space and low-dimensional space sub-clusters, the retention of the projected data is judged; third, by measuring the global error, a clear and quantifiable evaluation standard is provided for the sampling quality of high-dimensional data, thereby ensuring that the sampling quality of high-dimensional data in the scatter plot visualization task is accurately evaluated.

[0007] Optionally, given a high-dimensional data set, generating a data subset using a data sampling method includes: given a high-dimensional data set, analyzing the data distribution characteristics of the high-dimensional data set to obtain the distribution information of the data as a whole; based on the distribution information, generating a high-dimensional data subset using a data sampling method, wherein the data in the high-dimensional data subset has diversity and retains the main features of the original data set. The present invention has the following effects: first, by conducting an in-depth analysis of the data distribution characteristics of the high-dimensional data set, the overall distribution information of the data is fully captured, ensuring that the sampling process is targeted and more accurately reflects the characteristics of the original data set; second, based on the distribution information of the data as a whole, using a data sampling method, a high-dimensional data subset is efficiently generated, ensuring the diversity and representativeness of the high-dimensional data; third, the generated high-dimensional data subset better retains the main features of the original data set, providing strong support for subsequent high-dimensional data visualization analysis.

[0008] Optionally, the data sampling method includes a random sampling method, a blue noise sampling method, a singular value decomposition sampling method, a Z-order sampling method and a farthest point sampling method. The present invention obtains data subsets with different effects through a variety of data sampling methods, so that the data therein retains the main features of the original data set and has a certain diversity. These methods together constitute a powerful data sampling toolbox, which can flexibly select applicable sampling strategies according to different needs, and provide strong support for data processing and analysis.

[0009] Optionally, obtaining a low-dimensional data subset according to the high-dimensional data subset includes: using a projection method to project the high-dimensional data subset into a low-dimensional data subset, and the projection method includes a principal component analysis method, a unified manifold approximation and projection method, a local linear embedding method, a Laplace eigenmapping method, a non-negative matrix decomposition method, a factor analysis method, and Distributed random neighborhood embedding method. The present invention has the following effects: first, the structural characteristics of the data are retained from different angles, ensuring that the projected data subset can still reflect the key information of the original data; second, by adopting a variety of projection methods, flexible selection and accurate satisfaction of different specific needs are achieved; third, by projecting a high-dimensional data subset into a low-dimensional data subset, the inherent structure and characteristics of the original data are retained to the greatest extent, providing convenience for subsequent data analysis and processing.

[0010] Optionally, the randomly extracting clusters of the high-dimensional data set to obtain high-dimensional space sub-clusters includes: randomly selecting a seed point in the original space containing the high-dimensional data set, and using a clustering extraction function to obtain a high-dimensional space sub-cluster formed by points near the seed point. The present invention has the following effects: first, a seed point is randomly selected as the starting point to ensure the randomness and diversity of the clustering; second, the clustering extraction function is used to effectively identify and aggregate data points adjacent to the seed point, thereby forming a preliminary high-dimensional space sub-cluster; third, the obtained high-dimensional space sub-cluster retains the main features of the original data set, providing a basis for subsequent data processing and analysis.

[0011] Optionally, obtaining low-dimensional space sub-clusters according to the high-dimensional space sub-clusters includes: sampling data on the high-dimensional space sub-clusters to obtain sampling results; based on the sampling results, projecting the high-dimensional space sub-clusters to low-dimensional space to obtain low-dimensional space sub-clusters. The present invention has the following effects: first, by sampling data on high-dimensional space sub-clusters, representative sampling results are obtained, thereby better retaining the key information of the original data; second, by using projection technology to map high-dimensional space sub-clusters to low-dimensional space, the structure and characteristics of the original data are better retained; third, the obtained low-dimensional space sub-clusters reflect the key characteristics and distribution patterns of the original data, providing a basis for visual analysis of the data.

[0012] Optionally, the measuring the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster to obtain the difference between the high-dimensional space and low-dimensional space sub-cluster pairs includes: calculating the Gromov-Wasserstein distance between the density estimates of the high-dimensional space sub-cluster and the low-dimensional space sub-cluster; measuring the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster according to the Gromov-Wasserstein distance to obtain the difference between the high-dimensional space and low-dimensional space sub-cluster pairs. The present invention has the following effects: first, it quantifies the degree of difference in structure and characteristics between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster, not only considering the distance between data points, but also taking into account the overall shape and local characteristics of the data distribution; second, based on the Gromov-Wasserstein distance, it effectively measures the similarities and differences between sub-clusters in different dimensional spaces; third, the obtained difference between the high-dimensional space and low-dimensional space sub-cluster pairs not only reveals the impact of dimensional changes on data structure, but also provides a tool for comparison and analysis in different dimensional spaces, which is helpful to deeply understand the inherent laws and potential information of the data.

[0013] Optionally, obtaining the global error according to the difference between the high-dimensional space and the low-dimensional space sub-cluster pairs includes: performing a weighted sum operation on the differences between all high-dimensional space and low-dimensional space sub-cluster pairs; obtaining the global error according to the weighted sum operation. The present invention has the following effects: first, by performing a weighted sum operation on the differences between all high-dimensional space and low-dimensional space sub-cluster pairs, the error information between each sub-cluster pair is fully integrated, and through reasonable weight distribution, the sub-cluster pairs with a greater impact on the overall error are highlighted, thereby ensuring the accuracy and sensitivity of the global error calculation; second, through the global error index, the overall deviation degree of the structure and feature retention in the process of data conversion from high-dimensional space to low-dimensional space is intuitively reflected, providing an objective basis for evaluating the sampling quality of high-dimensional data.

[0014] Optionally, the evaluation of the sampling quality of high-dimensional data through the global error includes: obtaining a higher evaluation of the sampling quality of high-dimensional data through a smaller global error; obtaining a lower evaluation of the sampling quality of high-dimensional data through a larger global error. The present invention has the following effects: First, it intuitively reflects the ability of data sampling in maintaining the structure and characteristics of the original data. The smaller the global error, the higher the sampling quality and the less information loss of the data during the projection process; second, it provides an objective and unified evaluation standard, avoiding the uncertainty caused by subjective judgment, so that the sampling quality between different data sets can be fairly compared; third, it guides the optimization of data sampling strategies. When the global error is large, it prompts the need to improve the sampling method or increase the number of samples to improve the quality and representativeness of the data.

[0015] In the second aspect, the data sampling quality evaluation system for scatter plot visualization tasks provided by the present invention includes an input device, a processor, an output device and a memory, wherein the input device, the processor, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the system uses the data sampling quality evaluation method for scatter plot visualization tasks. The system has the following effects: first, by comparing the difference between the high-dimensional space sub-clustering and the low-dimensional space sub-clustering, the information loss and deformation degree of the data in the projection process are quantified, thereby accurately reflecting the representativeness and accuracy of the sampled data; second, through standardized evaluation processes and methods, it is ensured that the sampling quality between different data sets can be fairly compared, providing a reliable basis for data sampling quality evaluation; third, according to the evaluation results of the global error, the problems and deficiencies in the sampling process are pointed out, prompting the user to improve the sampling method or increase the number of samples, which is helpful to improve the quality and representativeness of the data, and can also lay a solid foundation for subsequent scatter plot visualization and data analysis.

[0016] Compared with the prior art, the beneficial effects of the present invention include: by calculating the differences in spatial sub-clusters of different dimensions, the retention of data after projection by different sampling methods is measured, and an effective evaluation of whether the projected scatter plot truly reflects high-dimensional data information is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a data sampling quality evaluation method for a scatter plot visualization task according to an embodiment of the present invention; Figure 2 A visualized scatter plot obtained by projection using different sampling methods according to an embodiment of the present invention; Figure 3 A schematic diagram of a cluster structure extracted by a cluster extraction method based on shared nearest neighbor similarity according to an embodiment of the present invention; Figure 4 A schematic diagram of performing different spatial comparisons based on the Gromov-Wasserstein distance according to an embodiment of the present invention; Figure 5 It is a schematic diagram of the structure of a data sampling quality evaluation system for scatter plot visualization tasks according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details do not need to be adopted to implement the present invention. In other examples, in order to avoid confusing the present invention, known circuits, software or methods are not specifically described.

[0019] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or subcombination. In addition, it should be understood by those of ordinary skill in the art that the figures provided herein are for illustrative purposes and that the figures are not necessarily drawn to scale.

[0020] See also Figure 1 The embodiment of the present invention provides a data sampling quality evaluation method for a scatter plot visualization task, the method comprising the following steps: S1. Given a high-dimensional data set, use data sampling methods to generate high-dimensional data subsets.

[0021] Among them, S1 includes the following steps: S11. Given a high-dimensional data set, analyze the data distribution characteristics of the high-dimensional data set to obtain the overall distribution information of the data.

[0022] In one embodiment, a high-dimensional data set is given , expressed as The matrix has indivual Dimensional data.

[0023] Furthermore, the statistics of each dimension, such as mean, standard deviation, maximum value, and minimum value, are calculated to analyze the correlation between dimensions.

[0024] Furthermore, feature selection methods are applied to screen key features and evaluate the contribution of feature subsets to data analysis.

[0025] Furthermore, based on the analysis results, the overall distribution characteristics of the data are summarized and potential problems and anomalies in the data are identified.

[0026] S12. Based on the distribution information, a high-dimensional data subset is generated using a data sampling method.

[0027] In one embodiment, based on the distribution information, a plurality of sampling methods are used to sample the high-dimensional data to generate a representative high-dimensional data subset. , the data in the high-dimensional data subset retains the main features of the original data set and has a certain diversity; the data sampling methods include random sampling method, blue noise sampling method, singular value decomposition sampling method, Z-order sampling method and farthest point sampling method. Using the above data sampling methods, high-dimensional data subsets with different effects are obtained, which are expressed as: .

[0028] Specifically, for the random sampling method, from the dataset Random selection data, expressed as: in Indicates the data set data. Represents the total number of data in the dataset. Represents a dataset generating function.

[0029] The blue noise sampling method aims to generate a uniformly distributed high-dimensional data subset. The basic idea is to ensure that the sampled high-dimensional data subset maintains the "blue noise" characteristics, that is, the distance between samples and the uniformity of distribution, through the optimization algorithm. The specific algorithm idea is: use the Thiessen polygon (Voronoi) diagram to divide the data space into several regions, each corresponding to a data point. For each data point, all points in its Voronoi unit are relatively close to the data point. By selecting points with smaller Voronoi units, it is ensured that the selected data is evenly distributed in the entire data set, thereby avoiding concentration in a small area. The data subset finally generated It can effectively represent the distribution characteristics of the original data. The specific formula is: in, is the Voronoi unit, is the distance between data points.

[0030] For sampling methods based on singular value decomposition (SVD), for high-dimensional data sets The expression for singular value decomposition is as follows: in, , , is the result of singular value decomposition, is the left singular vector, is a diagonal matrix of singular values, are right singular vectors, , Satisfies the following expression; in, The columns correspond to The principal components of A subset of Very good match, then the obtained data subset It can represent the original data set. The specific calculation process is as follows: To obtain The columns in The correlation of the columns in gives the following expression: Further, select The subset with higher correlation As a representative example, the expression is as follows: in, yes No. Column No. Coordinates, representing The columns in The correlation of the columns in for The number of columns, .

[0031] Further, Sum and get Correlations . The summation is only performed up to the parameter , which corresponds to The front It should be noted that only Before The reason for the column is that Most of the information in The basis vectors associated with the largest singular value of express.

[0032] Furthermore, based on the sampling method of random singular value decomposition, the Truncated singular values, by correlation Sort the data points and select the top ones with the greatest correlation. data points, forming a data subset .

[0033] For the Z-order based sampling method, high-dimensional data is mapped to a one-dimensional space by sorting according to the Z-order, and uniform sampling is performed in the one-dimensional space. The uniformity of sampling is ensured by balancing the binary tree.

[0034] Furthermore, the order between the data is determined by comparing the binary numbers of the high-dimensional data, thereby achieving Z-order sorting of the high-dimensional data.

[0035] Further, these data bits are checked one by one until a position different from the first data bit is found, which is the most significant dimension of the binary value. The data with "1" at this position will be sorted before other data.

[0036] Further, the most significant dimension will be found without performing a bitwise crossover. Specifically, the binary values ​​of two different high-dimensional data points are compared to determine which bit value is the most important for the two compared binary numbers, that is, the values ​​of the highest bits of the binary numbers that are different.

[0037] Furthermore, high-dimensional data points sorted according to Z-order are obtained.

[0038] Furthermore, the binary numbers of high-dimensional data are compared to determine the order between the data. The algorithm idea is as follows: Given points sorted in Z-order, label each point with a binary digit starting from 000...000, 000...001,... and fill the dataset with dummy points so that the total number of data points in the dataset is a power of 2; Reverse the order of the binary bits, that is, 1010 becomes 0101; Take a random mask M, XOR the mask with all flipped numbers, and sort the points into the new binary order; Remove the virtual points and form a new order. points, i.e., the sample subset obtained by Z-order sampling .

[0039] For the farthest point sampling method, a data subset is selected based on a distance metric .set up The selected point set in , currently selected point The following conditions must be met: in, Represents data points To selected point The minimum distance Indicates selecting the farthest point in the currently selected point set.

[0040] See also Figure 2 , Figure 2 It is a visual scatter plot using different sampling methods on the OpenStreetMap dataset and projected using the UMAP method. Figure 2 (a) shows the visual scatter plot obtained without sampling. Figure 2 (b)~ Figure 2 (f) represents the visualized scatter plots obtained by the farthest point sampling method, blue noise sampling method, singular value decomposition sampling method, Z-order sampling method and random sampling method respectively.

[0041] It should be noted that through the above sampling method, the generated subset It has good representativeness and balance, which not only effectively simplifies the data scale, but also provides a clearer original data set structure for subsequent analysis and visualization.

[0042] S2. Obtain a low-dimensional data subset based on the high-dimensional data subset.

[0043] In one embodiment, for a representative data subset , use the projection method to get the projection data subset Different projection methods focus on different characteristics of data and retain different data structures during the projection process to ensure the diversity and accuracy of the analysis results. The projection methods include principal component analysis, unified manifold approximation and projection method, local linear embedding method, non-negative matrix decomposition method, factor analysis method and Distributed random neighborhood embedding method.

[0044] Specifically, the principal component analysis method projects the data to the direction of maximum variance by finding the main direction of the data. It mainly retains the global linear structure information and is suitable for overall distribution analysis and principal component extraction of data.

[0045] The unified manifold approximation and projection method, as a nonlinear projection method, focuses on maintaining the local neighborhood structure, is suitable for preserving the local neighbor relationship of nonlinear data, and is usually used for visualization and clustering of high-dimensional data.

[0046] The local linear embedding method maps high-dimensional data to low-dimensional space by maintaining the local linear relationship of the data. It emphasizes the local linear structure of the data, is suitable for the scenario of data manifold embedding, and can better preserve the low-dimensional topological structure of the manifold.

[0047] The Laplace eigenmapping method utilizes the eigenvalue decomposition of the graph Laplace matrix to preserve the local geometric structure of data, and is suitable for preserving local connection relations and low-dimensional manifold structures.

[0048] The non-negative matrix decomposition method decomposes data into the product of non-negative matrices, is suitable for sparse feature extraction and pattern recognition of data, and is helpful for extracting principal components under non-negative constraints.

[0049] The factor analysis method assumes that the data is generated by latent factors, emphasizes the extraction of latent factors behind the data, and is suitable for latent factor modeling and feature simplification of data.

[0050] Said The distributed random neighborhood embedding method emphasizes maintaining the proximity relationship of data points in the high-dimensional space in the projection space. It is mainly used for the visualization of nonlinear data, especially the display of high-dimensional clustering structures.

[0051] It should be noted that through the above projection method, the high-dimensional data subset Projection to a low-dimensional subset of data , which can preserve the structural characteristics of the data from different perspectives and effectively provide support for subsequent data analysis and visualization.

[0052] S3. Randomly extract clusters of the high-dimensional data set to obtain high-dimensional space sub-clusters.

[0053] In one embodiment, first, a high-dimensional dataset is A seed point is randomly selected in the original space.

[0054] Furthermore, we use the shared nearest neighbor similarity as the cluster extraction function to obtain subclusters formed by points near the seed point. Among them, the specific calculation process of shared nearest neighbor similarity is as follows: Determine the corresponding nearest neighbors; For any two data points and , calculate the number of their shared nearest neighbors, that is, and The number of neighbors that exist in common among the neighbors of ; calculate and It should be noted that the shared nearest neighbor similarity is defined as the ratio of the number of shared neighbors to the total number of neighbors, and the formula is: in, To share the nearest neighbor similarity, and They are points and Point The set of neighbors.

[0055] Furthermore, setting the threshold ,if , then it is considered that point and Point Similarity, that is, point and Point belong to the same subcluster.

[0056] Furthermore, by iteratively randomly selecting seed points, the generated sub-clusters can cover almost all clustering structures in the space, thus ensuring that most clustering situations can be observed.

[0057] See also Figure 3 , Figure 3 It is a schematic diagram of the cluster structure extracted by the cluster extraction method based on shared nearest neighbor similarity. The light-colored part represents the extracted cluster structure. Figure 3 (a) shows that the clustering method based on shared nearest neighbor similarity calculates clusters in a low-dimensional space and obtains a good clustering structure. Figure 3 (b) shows that the clustering method based on shared nearest neighbor similarity calculates clusters in high-dimensional space and projects them into low-dimensional space, obtaining a deformed clustering structure.

[0058] S4. Obtain low-dimensional space sub-clusters based on the high-dimensional space sub-clusters.

[0059] Among them, S4 includes the following steps: S41. Perform data sampling on the high-dimensional space sub-clustering to obtain sampling results.

[0060] In one embodiment, the high-dimensional space sub-clustering Data sampling was performed to obtain sub-clusters of the high-dimensional space containing specific data.

[0061] S42. Based on the sampling results, the high-dimensional space sub-clusters are projected into the low-dimensional space to obtain the low-dimensional space sub-clusters.

[0062] In one embodiment, the sub-cluster of the high-dimensional space containing the specific data is projected into the low-dimensional space to obtain the sub-cluster of the low-dimensional space. , which satisfies the following expression: in, The data points in The projection subset of In an ideal situation, form a good cluster.

[0063] It should be noted that due to the errors caused by sampling and projection, This does not fully reflect good clustering.

[0064] S5. Measure the difference between the high-dimensional space sub-clusters and the low-dimensional space sub-clusters to obtain the difference between the high-dimensional space and low-dimensional space sub-clusters.

[0065] Among them, S5 includes the following steps: S51. Calculate the Gromov-Wasserstein distance between the density estimates of the high-dimensional space sub-cluster and the low-dimensional space sub-cluster.

[0066] In one embodiment, the differences between subclusters in different spaces are compared by calculating the distance-based density estimates of high-dimensional space subclusters and low-dimensional space subclusters, and calculating the Gromov Wasserstein distance. Among them, the distance-based density estimation method quantifies the distribution of subclusters in different spaces, thereby obtaining the local structure and distribution characteristics of the subclusters. In order to compare data of different spaces and different sizes, it is difficult to directly calculate the distance between them because their points may not have a direct correspondence. The Gromov-Wasserstein distance solves this problem by focusing on the point pair relationship inside the space, and has strong flexibility. It is a tool for comparing probability distributions between different metric spaces, and is particularly suitable for structural data that are not aligned or in different spaces.

[0067] Specifically, given two metric spaces and , by minimizing the difference in distance in the metric space, the Gromov-Wasserstein distance is calculated to find the optimal transmission method in high-dimensional space and low-dimensional space The Gromov-Wasserstein distance satisfies the following expression: in, is the Gromov-Wasserstein distance between high-dimensional space and low-dimensional space, and are the metric matrices of high-dimensional space and low-dimensional space respectively, , are the sizes of high-dimensional space matrices and low-dimensional space matrices, respectively, and the matrix elements and Indicates point pair and The distance in their respective spaces, , are all transfer matrices from high-dimensional space to low-dimensional space, and is the probability distribution in high-dimensional space and low-dimensional space, is the set of all joint distributions that satisfy the marginal constraints, i.e. Satisfies the following relationship: in, is the identity matrix of the high-dimensional space, is the identity matrix of the low-dimensional space. Is a loss function used to measure the difference in distance between two spaces. The common square loss function satisfies the following relationship: It is important to note that the degree to which the clustering structure changes throughout the process is quantitatively assessed by calculating the Gromov-Wasserstein distance between the density estimates of the high-dimensional sub-clusters and the low-dimensional sub-clusters.

[0068] S52. According to the Gromov-Wasserstein distance, the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster is measured to obtain the difference between the high-dimensional space and low-dimensional space sub-cluster pairs.

[0069] In one embodiment, in order to measure the difference between the sub-clusters in the high-dimensional space and the sub-clusters in the low-dimensional space, the distance-based density estimates of the two sub-clusters are calculated, and the differences in different spaces are compared.

[0070] Specifically, the clustering set As input, for each sub-cluster pair , respectively, calculate their distance-based density estimates in the projected space and the original space. For each data point , the calculation expression satisfied by the distance-based density estimation is as follows: in, is the distance-based density estimation, is the number of nearest neighbors, represents the Euclidean distance from a data point 𝑥 to its 𝑖th neighbor. The distance-based density estimate is Lipschitz continuous and insensitive to small perturbations.

[0071] Furthermore, the Euclidean distance metric is used to calculate and Distance matrix of internal data points and . and The corresponding probability density and Depend on The normalized representation is as follows: .

[0072] According to the probability density distribution and Initialize the transmission matrix . Using the squared loss function To measure the difference between the elements of the distance matrices in the two metric spaces, that is, for each pair of points, the square of the similarity difference between the two spaces is calculated to capture the difference in the geometric structures of the two spaces.

[0073] Furthermore, the low-dimensional space metric matrix and high-dimensional space metric matrix Substituting into the expression of the Gromov-Wasserstein distance in step S51, we obtain: in, is the Gromov-Wasserstein distance between each pair of sub-clusters in the low-dimensional space and the high-dimensional space, , is the probability density distribution, is the order of the elements of the matrix, is the square loss function, , are all transfer matrices. The conjugate gradient method is used to adjust Until the error converges, the final It measures the similarity of two metric spaces. The smaller the value, the closer the structures of the two spaces are. Figure 4 This is a schematic diagram of the comparison of different spaces based on the Gromov-Wasserstein distance, comparing the structural differences in the distribution of scattered points in different spaces.

[0074] S6. Obtain a global error based on the differences between the sub-clustering pairs in the high-dimensional space and the low-dimensional space.

[0075] Wherein, S6 further comprises the following steps: S61. Perform a weighted sum operation on the differences between all sub-clustering pairs in the high-dimensional space and the low-dimensional space.

[0076] In one embodiment, the difference of each sub-cluster pair calculated in step S5 is first obtained. .

[0077] Furthermore, the difference Perform weighted sum operation.

[0078] S62. Obtain a global error according to the weighted summation operation.

[0079] In one embodiment, according to the weighted summation operation described in step S61, a global error is obtained, and the global error satisfies the following expression: in, is the global error, is the difference for each pair of sub-clusters, Represents the weight, satisfying the following expression: in, for The number of data points in is the number of high-dimensional data points.

[0080] It should be noted that the error caused by sub-clusters with larger data volumes should be more reflected in the global error.

[0081] S7. Through the global error, the sampling quality of high-dimensional data is evaluated.

[0082] Wherein, S7 further comprises the following steps: S71. Obtain a higher evaluation of the sampling quality of high-dimensional data through a smaller global error.

[0083] In this embodiment, the smaller the global error is, the closer the sampled projection data is to the original data.

[0084] Furthermore, it shows that the sampling quality of high-dimensional data is higher.

[0085] S72. Obtain a lower evaluation of the sampling quality of high-dimensional data through a larger global error.

[0086] In this embodiment, the larger the global error is, the more the sampled projection data deviates from the original data.

[0087] Furthermore, it indicates that the sampling quality of high-dimensional data is low.

[0088] See also Figure 5 , Figure 5 The present invention is a schematic diagram of the structure of a data sampling quality evaluation system for scatter plot visualization tasks in an embodiment of the present invention. The system includes an input device, a processor, an output device, and a memory, wherein the input device, the processor, the output device, and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the system uses the data sampling quality evaluation method for scatter plot visualization tasks.

[0089] In this embodiment, the input device has a data acquisition function, including a high-precision sensor and a scanning device, which is used to collect raw data and convert it into a format recognizable by the system for subsequent processing.

[0090] The processor has powerful computing capabilities and data analysis algorithms, and is used to process input data and perform data cleaning, sampling, and quality evaluation.

[0091] The output device is used to display the evaluation results of data sampling quality, including a high-resolution display screen and a printer.

[0092] The memory uses a high-speed solid-state hard disk to store original data and data processed by the processor. It has the characteristics of fast reading and writing speed, large capacity, and high reliability, and can meet the needs of large data storage.

[0093] In summary, the data sampling quality evaluation method for scatter plot visualization tasks provided by the present invention obtains a good visualization effect by sampling a high-dimensional data set and projecting it into a low-dimensional space; by comparing the differences between the sub-clusters of the high-dimensional space and the low-dimensional space, the retention of the projected data is judged; by measuring the global error, a clear and quantifiable evaluation standard is provided for the high-dimensional data sampling quality, thereby ensuring that the high-dimensional data sampling quality in the scatter plot visualization task is accurately evaluated. The data sampling quality evaluation system for scatter plot visualization tasks provided by the present invention quantifies the information loss and deformation degree of the data in the projection process by comparing the differences between the sub-clusters of the high-dimensional space and the sub-clusters of the low-dimensional space, thereby accurately reflecting the representativeness and accuracy of the sampled data; through standardized evaluation processes and methods, it ensures that the sampling quality between different data sets can be fairly compared, providing a reliable basis for data sampling quality evaluation; according to the evaluation results of the global error, the problems and deficiencies in the sampling process are pointed out, prompting the user to improve the sampling method or increase the number of samples, which is helpful to improve the quality and representativeness of the data, and can also lay a solid foundation for subsequent scatter plot visualization and data analysis.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A data sampling quality evaluation method for scatter plot visualization tasks, characterized in that: The method comprises the following steps: Given a high-dimensional data set, use data sampling methods to generate high-dimensional data subsets; According to the high-dimensional data subset, a low-dimensional data subset is obtained; Randomly extracting clusters of the high-dimensional data set to obtain high-dimensional space sub-clusters; According to the high-dimensional space sub-clustering, obtaining low-dimensional space sub-clustering; Measuring the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster to obtain the difference between the high-dimensional space and low-dimensional space sub-cluster pairs; Obtaining a global error according to the difference between the sub-clustering pairs in the high-dimensional space and the low-dimensional space; The global error is used to evaluate the sampling quality of high-dimensional data.

2. A data sampling quality evaluation method for scatter plot visualization tasks according to claim 1, characterized in that: Given a high-dimensional data set, generating a high-dimensional data subset using a data sampling method includes: Given a high-dimensional data set, analyzing the data distribution characteristics of the high-dimensional data set to obtain the overall distribution information of the data; Based on the distribution information, a high-dimensional data subset is generated using a data sampling method. The data in the high-dimensional data subset has diversity and retains the main features of the original data set.

3. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 2 is characterized in that: The data sampling methods include random sampling method, blue noise sampling method, singular value decomposition sampling method, Z-order sampling method and farthest point sampling method.

4. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 1, characterized in that: The step of obtaining a low-dimensional data subset according to the high-dimensional data subset comprises: The high-dimensional data subset is projected into a low-dimensional data subset by using a projection method, wherein the projection method includes a principal component analysis method, a unified manifold approximation and projection method, a local linear embedding method, a Laplace eigenmapping method, a non-negative matrix decomposition method, a factor analysis method and Distributed random neighborhood embedding method.

5. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 1, characterized in that: The randomly extracting clusters of the high-dimensional data set to obtain high-dimensional space sub-clusters comprises: In the original space containing the high-dimensional data set, a seed point is randomly selected, and the cluster extraction function is used to obtain the high-dimensional space sub-cluster formed by the points near the seed point.

6. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 1, characterized in that: The step of obtaining low-dimensional space sub-clusters according to the high-dimensional space sub-clusters comprises: Performing data sampling on the high-dimensional space sub-clustering to obtain sampling results; Based on the sampling results, the high-dimensional space sub-clusters are projected into the low-dimensional space to obtain the low-dimensional space sub-clusters.

7. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 1 is characterized in that: The measuring the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster to obtain the difference between the high-dimensional space and the low-dimensional space sub-cluster pair includes: Calculating the Gromov-Wasserstein distance between the density estimates of the high-dimensional space sub-cluster and the low-dimensional space sub-cluster; According to the Gromov-Wasserstein distance, the difference between the high-dimensional space sub-cluster and the low-dimensional space sub-cluster is measured to obtain the difference between the high-dimensional space and low-dimensional space sub-cluster pairs.

8. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 1, characterized in that: The obtaining of the global error according to the difference between the high-dimensional space and the low-dimensional space sub-clustering pair comprises: Perform a weighted sum operation on the differences between all high-dimensional space and low-dimensional space sub-clustering pairs; According to the weighted summation operation, a global error is obtained.

9. The data sampling quality evaluation method for scatter plot visualization tasks according to claim 1, characterized in that: The evaluation of the sampling quality of high-dimensional data by using the global error includes: Through smaller global error, a higher evaluation of high-dimensional data sampling quality is obtained; A lower evaluation of the sampling quality of high-dimensional data is obtained through a larger global error.

10. A data sampling quality evaluation system for scatter plot visualization tasks, the system using a data sampling quality evaluation method for scatter plot visualization tasks according to any one of claims 1 to 9, characterized in that: The system includes an input device, a processor, an output device and a memory, wherein the input device, the processor, the output device and the memory are connected to each other, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions.

Images