Incremental interactive clustering visualization method and system for trusted sharing of cloud data

Abstract

The present invention discloses an incremental interactive clustering visualization method and system for trusted sharing of cloud data, the method comprising: writing a data sharing smart contract and running the smart contract; extracting target data from a data set according to the written data sharing smart contract; clustering the extracted target data using a clustering algorithm and outputting clustering results; performing multi-dimensional scaling dimensionality reduction on the clustering results to achieve projection in two-dimensional space and perform visual display; performing corresponding projection interaction, clustering analysis and visualization operations according to the user's demand for changes in target data. Based on the trusted sharing of blockchain data, incremental interactive analysis and mining of data are performed, which facilitates users to more intuitively analyze and mine data while effectively ensuring data security and improving data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of system engineering technology, and in particular to an incremental interactive clustering visualization method and system for trusted sharing of cloud data. Background Art

[0002] Data visualization is one of the important ways of data application. After data is transformed, it is presented intuitively in the form of charts or graphics, thereby helping people to better understand and analyze the information hidden in the data. Clustering is a common mining method for data analysis, which is often used for multivariate relational data mining, sequence and anomaly detection, etc. At present, many data analysis and mining platforms provide clustering algorithms for data analysis and mining, and visualize the results. In most cases, the general process is to first gather all the data to be analyzed from various systems, then process all the data at the same time in batch processing, and finally present the clustering results. The above processing flow and method have many problems in the process of data analysis and mining. First, in terms of data processing, all data must be gathered before the subsequent steps can be carried out, and the data collection time is long. Especially in a shared system with dispersed data, each data processing process needs to wait for the aggregation of all data before subsequent operations. At the same time, in the process of collecting and processing all the original data, there is a risk of data leakage, and data security cannot be well protected. Secondly, in the above process, the entire process of using clustering for data analysis is a static process. After each parameter is set, the algorithm runs to obtain the result and displays it to the user. The user cannot adjust the clustering parameters more intuitively.

[0003] Therefore, based on the existing data analysis and mining technology, how to provide an incremental interactive clustering visualization method and system for reliable sharing of cloud data to greatly improve data processing efficiency, ensure data security and reliability, and make the data processing process intuitive has become an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0004] In view of the above problems, the present invention proposes an incremental interactive clustering visualization method and system for trusted sharing of cloud data that at least solves some of the above technical problems. While effectively ensuring data security and improving data processing efficiency, it facilitates users to conduct data analysis and mining more intuitively.

[0005] The embodiment of the present invention provides an incremental interactive clustering visualization method for trusted sharing of cloud data, which is characterized by comprising the following steps:

[0006] S1. Write data sharing smart contracts and run smart contracts;

[0007] S2. Extract target data from the data set according to the written data sharing smart contract;

[0008] S3, clustering the extracted target data using a clustering algorithm, and outputting the clustering results;

[0009] S4, performing multi-dimensional scaling on the clustering results, realizing projection of the clustering results in a two-dimensional space, and performing visual display;

[0010] S5. Perform corresponding projection interaction, cluster analysis and visualization operations according to the user's demand for changing the target data.

[0011] Furthermore, in step S4, the clustering result is reduced in dimension using multidimensional scaling, including:

[0012] S41, calculating a distance matrix according to the clustering result and the set low-dimensional space dimension;

[0013] S42, calculating an inner product matrix according to the distance matrix;

[0014] S43, performing eigenvalue decomposition on the inner product matrix, calculating the first n largest eigenvalues ​​and their eigenvectors, and generating a diagonal matrix and an eigenvector matrix composed of the first n largest eigenvalues;

[0015] S44. Calculate a reduced-dimensional matrix based on the diagonal matrix and eigenvector matrix composed of the first n largest eigenvalues, and output a low-dimensional representation of the clustering result.

[0016] Furthermore, the step S5 comprises:

[0017] S51. If the user changes the attribute of the target data, a forward projection is constructed, and the result after the data attribute transformation is displayed on the visualization interface to realize the forward projection interaction;

[0018] S52, if the user drags and drops the visualized data point on the target data, a back projection is constructed, and the adjusted attributes of the data point are calculated to realize the back projection interaction;

[0019] S53: If the user adds incremental data based on the target data, cluster analysis and visualization operations are performed on the incremental data.

[0020] Further, the S51 includes:

[0021] S511, obtaining the data points after the attribute change, and the centroids of each cluster in the current clustering model;

[0022] S512, respectively calculating the Euclidean distance from the data point to the centroid of each cluster;

[0023] S513. Obtain the shortest distance among the Euclidean distances, and classify the data point into the cluster corresponding to the shortest distance;

[0024] S514. Project the data point in the scatter plot and perform color coding with the same color of the corresponding cluster to achieve forward projection interaction.

[0025] Further, in S511, the centroid of each cluster in the current clustering model is obtained by using the average value method.

[0026] Further, S52 includes:

[0027] S521. Obtain the low-dimensional data of the projection point after dragging and dropping the visual data point;

[0028] S522. Calculate the change vector from the original projection point to the new projection point according to the low-dimensional data;

[0029] S523. Adopt the PCA dimensionality reduction method to calculate Δx:

[0030] Δx[e0 e1] = Δy

[0031] where Δx is the feature change vector of the original data point; [e0 e1] is the feature vector matrix of the original data point; Δy is the position change vector;

[0032] S524. Solve the optimal solution of Δx according to the regularized least squares method to obtain the adjusted attributes of the data point and achieve backward projection interaction.

[0033] Further, S53 includes:

[0034] S531. Obtain the data points of the newly added incremental data, read the data points into the processing system, and set the Kullback-Leibler divergence threshold;

[0035] S532. Calculate the Kullback-Leibler divergence between the data set after the addition of data points and the original data set;

[0036] S533. Compare the Kullback-Leibler divergence with the Kullback-Leibler divergence threshold, and adopt the out-of-sample extension method and the re-clustering method respectively to achieve the clustering analysis and visualization operation of the incremental data.

[0037] Further, in S533, adopting the out-of-sample extension method to achieve the clustering analysis and visualization operation of incremental data includes: when the Kullback-Leibler divergence < Kullback-Leibler divergence threshold, divide the newly added data points into clusters according to the distance from the points to the cluster centers, and update the cluster centers; and project the newly added data points into the scatter plot by using the out-of-sample extension method.

[0038] Furthermore, in S533, a re-clustering method is used to implement clustering analysis and visualization operations of incremental data, including: when the KL divergence>KL divergence threshold, the data set is re-clustered to find the best overlapping position with the target data dimensionality reduction result and project it; and the projection points after dimensionality reduction are color-coded according to the clustering results.

[0039] The embodiment of the present invention also provides an incremental interactive clustering visualization system for trusted sharing of cloud data, including:

[0040] Write and run modules to write data sharing smart contracts and run smart contracts;

[0041] An extraction module, used to extract target data from the data set according to the written data sharing smart contract;

[0042] The clustering module is used to cluster the extracted target data using a clustering algorithm and output the clustering results;

[0043] A visualization module is used to perform multi-dimensional scaling on the clustering results, realize the projection of the clustering results in two-dimensional space, and perform visual display;

[0044] According to the user's demand for changes to the target data, corresponding projection interaction, cluster analysis and visualization operations are performed.

[0045] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0046] An incremental interactive clustering visualization method for trusted sharing of cloud data provided by an embodiment of the present invention includes: writing a data sharing smart contract and running the smart contract; extracting target data from a data set according to the written data sharing smart contract; clustering the extracted target data using a clustering algorithm and outputting clustering results; performing multi-dimensional scaling dimensionality reduction on the clustering results to achieve projection in two-dimensional space and perform visualization; performing corresponding projection interaction, clustering analysis and visualization operations according to the user's demand for changes in target data. Based on the trusted sharing of blockchain data, incremental interactive analysis and mining of data is performed, which facilitates users to analyze and mine data more intuitively while effectively ensuring data security and improving data processing efficiency.

[0047] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 A flow chart of an incremental interactive clustering visualization method for trusted sharing of cloud data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] The embodiment of the present invention provides an incremental interactive clustering visualization method for reliable sharing of cloud data, referring to Figure 1 As shown, the following steps are included:

[0053] S1. Write data sharing smart contracts and run smart contracts;

[0054] S2. Extract target data from the data set according to the written data sharing smart contract;

[0055] S3, clustering the extracted target data using a clustering algorithm, and outputting the clustering results;

[0056] S4, perform multi-dimensional scaling dimensionality reduction on the clustering results, realize the projection of the clustering results in two-dimensional space, and perform visual display;

[0057] S5. Perform corresponding projection interaction, cluster analysis and visualization operations according to the user's demand for changes in target data.

[0058] This embodiment, based on the trusted sharing of blockchain data, performs incremental interactive analysis and mining of data, which effectively ensures data security and improves data processing efficiency while facilitating users to analyze and mine data more intuitively.

[0059] The following is a detailed description of each of the above steps:

[0060] Specifically, steps S1 and S2 are to write a data sharing smart contract and run the smart contract; and extract target data from the data set according to the written data sharing smart contract. Blockchain technology is used here to effectively realize the trusted sharing of data, realize distributed data sharing, and ensure the security and trustworthiness of shared data.

[0061] Furthermore, step S3 is to cluster the extracted target data using a clustering algorithm and output the clustering results. The incremental clustering analysis and mining method is used to greatly improve the data processing efficiency and speed up the response time of data analysis and mining. Specifically, it includes:

[0062] S31, selecting a clustering algorithm;

[0063] S32, setting clustering parameters;

[0064] S33: Cluster the extracted target data and output the clustering result X.

[0065] For example:

[0066] 1. Choose a k-means algorithm;

[0067] 2. Set the input of k-means, including data set and k value;

[0068] 3. Use the clustering algorithm to divide the data set into k classes.

[0069] Furthermore, step S4 is to perform multi-dimensional scaling on the clustering results, realize the projection of the clustering results of the target data (high-dimensional data) in two-dimensional space (projected into low-dimensional data), and perform visual display. The intuitive display on the interface can more intuitively adjust the clustering parameters. Specifically including:

[0070] S41, according to the clustering result X and the set low-dimensional space dimension n, calculate the distance matrix D, where the element d in the i-th row and j-th column is ij is the distance of the sample;

[0071] S42, according to the principle that the distances before and after dimensionality reduction are as close as possible, the inner product matrix B is calculated according to the distance matrix D;

[0072] S43, perform eigenvalue decomposition on the inner product matrix B, calculate the first n largest eigenvalues ​​and their corresponding eigenvectors, and generate a diagonal matrix U composed of the first n largest eigenvalues n and the eigenvector matrix V n ;

[0073] S44, according to the diagonal matrix U n and the eigenvector matrix V n, calculate the reduced-dimensional matrix Z and output the low-dimensional representation of the clustering result, that is, the low-dimensional representation of the initial target data.

[0074] Furthermore, step S5 is to perform corresponding projection interaction, cluster analysis and visualization operations according to the user's demand for changes to the target data. Among them, the visualization is to reduce the multi-dimensional to two-dimensional, that is, to generate points of horizontal and vertical coordinates, and display these points directly on the graph. When the user adds data and changes the data set of the target data, the corresponding operation is performed on the visualization interface. Specifically including:

[0075] S51. If the user changes the data attributes, a forward projection is constructed, and the result after the data attribute transformation is displayed on the visualization interface to realize the forward projection interaction;

[0076] S52, if the user drags and drops the visualized data point, a back projection is constructed, and the adjusted attributes of the data point are calculated to realize the back projection interaction;

[0077] S53: If the user adds incremental data, perform cluster analysis and visualization operations on the incremental data.

[0078] Specifically, step S51 includes:

[0079] S511, the user interacts, changes the attribute value of the data point, obtains the data point x after the attribute change, and uses the average method to obtain the centroid m of the k clusters in the current clustering model j , 0 <j<k;

[0080] S512, respectively calculate the Euclidean distance l from the data point x to the centroids of k clusters j ;

[0081] S513. Get the shortest distance min l in the Euclidean distance j , the data point x is classified into the cluster corresponding to the shortest distance;

[0082] S514 , projecting the data point x onto the scatter plot and color-coding it with the same color as the corresponding cluster to achieve forward projection interaction.

[0083] Step S52 includes:

[0084] S521, the user drags a node to a new position in the projection space in the front-end interface to obtain low-dimensional data y of the projection point;

[0085] S522, calculating the change vector Δy from the original projection point to the new projection point according to the low-dimensional data y;

[0086] S523. Use PCA dimensionality reduction method to calculate Δx:

[0087] Δx[e0 e1] = Δy

[0088] Wherein, Δx is the feature change vector of the original data points; [e0 e1] is the feature vector matrix of the original data points; Δy is the position change vector;

[0089] S524. Solve the optimal solution of Δx according to the regularized least squares method to obtain the adjusted attributes of the data points, and implement back-projection interaction.

[0090] Wherein, in step S522, the original projection point refers to the point before drag-and-drop; the new projection point refers to the point after drag-and-drop; that is, a point is moved to another place by drag-and-drop. In step S523, the original data point refers to the point before drag-and-drop.

[0091] Furthermore, in order to ensure the accurate and rapid response of the change of incremental data during the visualization process, two different methods are adopted. When the influence of the newly added incremental data on the current clustering result is small, the method of out-of-sample extension is used for the projection of the newly added data. When the influence of the newly added incremental data on the current clustering result is large, the method of re-clustering is used for the construction of visualization, and the Procrustes transformation is used to find the best overlap with the data projection position before re-clustering. The specific steps S53 include:

[0092] S531. Obtain the data point x of the newly added incremental data, read the data point x (from the database, etc.) into the processing system, and set the K-L divergence threshold k;

[0093] S532. Calculate the K-L divergence between the data set after the addition of data points and the original data set (referring to the data set in step S2);

[0094] S533. Compare the K-L divergence with the K-L divergence threshold k, and respectively adopt the out-of-sample extension method and the re-clustering method to implement the clustering analysis and visualization operation of the incremental data. Specifically, it includes:

[0095] When the K-L divergence < the K-L divergence threshold k, divide the newly added data points into clusters according to the distance from the points to the cluster centers, and update the cluster centers; project the newly added data point x into the scatter plot by using the out-of-sample extension method;

[0096] When the K-L divergence > the K-L divergence threshold k, re-cluster the data set, apply the Procrustes geometric transformation to find the best overlap position with the previous target data dimensionality reduction result and perform projection; color-code the projected points after dimensionality reduction according to the clustering result.

[0097] The above steps are illustrated by a specific application example as follows:

[0098] Requirements: Data in different systems need to be clustered and interactively operated;

[0099] process:

[0100] 1. Write the contract;

[0101] 2. Extract data;

[0102] 3. Perform clustering;

[0103] 4. Reduce dimension to two dimensions;

[0104] 5. Display to the visual interface;

[0105] 6. Establish forward interaction from data to interface, that is, manually change the attributes of data and then display it on the interface;

[0106] 7. Establish backward interaction from the interface to the data, that is, drag and drop points on the interface, and then get the result after the attribute changes;

[0107] 8. Establish a visualization plan for the newly added data, that is, how to combine it with the original plan after adding a piece of data.

[0108] The incremental interactive clustering visualization method for trusted sharing of cloud data provided in this embodiment further optimizes the clustering data analysis and mining process, and optimizes it from three aspects: first, the data calculation is transformed from the calculation of the full amount of data to incremental calculation, and cluster analysis is performed according to the time series reached by the data without changing the data operation results; second, in the data sharing process, it avoids the direct centralized sharing of the original data, adopts a distributed strategy, and uses encryption, dimensionality reduction and other methods to transform the data, and then shares the transformed data without affecting the data processing; third, it does not only display the data of the processing results, but directly analyzes and processes it on the interface, and displays the processing results, so as to more intuitively mine the data value. In this way, the data processing efficiency can be effectively improved; the problem of data leakage caused by centralized data sharing and processing can be avoided; and interactive visualization clustering data analysis and mining can be realized.

[0109] On the other hand, an embodiment of the present invention further provides an incremental interactive clustering visualization system for trusted sharing of cloud data, which is applicable to the incremental interactive clustering visualization method for trusted sharing of cloud data as described above, including:

[0110] Write and run modules to write data sharing smart contracts and run smart contracts;

[0111] The extraction module is used to extract target data from the data set according to the written data sharing smart contract;

[0112] The clustering module is used to cluster the extracted target data using a clustering algorithm and output the clustering results;

[0113] The visualization module is used to perform multi-dimensional scaling on the clustering results, realize the projection of the clustering results in two-dimensional space, and perform visual display;

[0114] According to the user's demand for changes in target data, corresponding projection interaction, cluster analysis and visualization operations are performed.

[0115] Since the incremental interactive clustering visualization system for trusted sharing of cloud data is applicable to the aforementioned incremental interactive clustering visualization method for trusted sharing of cloud data, the implementation of the system can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0116] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it is applicable to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0117] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An incremental interactive clustering visualization method for trusted sharing of cloud data, characterized by: The steps include: S1. Write data sharing smart contracts and run smart contracts; S2. Extract target data from the data set according to the written data sharing smart contract; S3, clustering the extracted target data using a clustering algorithm, and outputting the clustering results; S4, performing multi-dimensional scaling on the clustering results, realizing projection of the clustering results in a two-dimensional space, and performing visual display; S5. Perform corresponding projection interaction, cluster analysis and visualization operations according to the user's demand for changing the target data; The step S5 comprises: S51. If the user changes the attribute of the target data, a forward projection is constructed, and the result after the data attribute transformation is displayed on the visualization interface to realize the forward projection interaction; S52, if the user drags and drops the visualized data point on the target data, a back projection is constructed, and the adjusted attributes of the data point are calculated to realize the back projection interaction; S53: If the user adds incremental data based on the target data, cluster analysis and visualization operations are performed on the incremental data; The S51 includes: S511, obtaining the data points after the attribute change, and the centroids of each cluster in the current clustering model; S512, respectively calculating the Euclidean distance from the data point to the centroid of each cluster; S513, obtaining the shortest distance among the Euclidean distances, and classifying the data points into a cluster corresponding to the shortest distance; S514, projecting the data points into a scatter plot and color-coding them with the same color as the corresponding cluster to achieve forward projection interaction; The S52 includes: S521, obtaining low-dimensional data of the projection point after dragging and dropping the visualization data point; S522, calculating a change vector from the original projection point to the new projection point according to the low-dimensional data; S523. Use PCA dimensionality reduction method to calculate Δx: Δx[e0 e1]=Δy Among them, Δx is the characteristic change vector of the original data point; [e0 e1] is the characteristic vector matrix of the original data point; Δy is the position change vector; S524. Solve the optimal solution of Δx according to the regularized least squares method, obtain the adjusted attributes of the data points, and realize the back-projection interaction.

2. The incremental interactive clustering visualization method for trusted sharing of cloud data according to claim 1, characterized in that: In step S4, the clustering result is subjected to dimensionality reduction using multidimensional scaling, including: S41, calculating a distance matrix according to the clustering result and the set low-dimensional space dimension; S42, calculating an inner product matrix according to the distance matrix; S43, performing eigenvalue decomposition on the inner product matrix, calculating the first n largest eigenvalues ​​and their eigenvectors, and generating a diagonal matrix and an eigenvector matrix composed of the first n largest eigenvalues; S44. Calculate a reduced-dimensional matrix based on the diagonal matrix and eigenvector matrix composed of the first n largest eigenvalues, and output a low-dimensional representation of the clustering result.

3. The incremental interactive clustering visualization method for trusted sharing of cloud data according to claim 1, characterized in that: In S511, the average value method is used to obtain the centroid of each cluster in the current clustering model.

4. The incremental interactive clustering visualization method for trusted sharing of cloud data according to claim 1, characterized in that: The S53 includes: S531, obtaining data points of newly added incremental data, reading the data points into a processing system, and setting a KL divergence threshold; S532, calculating the KL divergence of the data set after the data points are added and the original data set; S533. Compare the Kullback–Leibler divergence with the Kullback–Leibler divergence threshold, and respectively adopt the out-of-sample extension method and the re-clustering method to implement the clustering analysis and visualization operation of the incremental data.

5. The incremental interactive clustering visualization method for trusted sharing of cloud data according to claim 4, characterized in that: In S533, the out-of-sample extension method is adopted to implement the clustering analysis and visualization operation of incremental data, including: when the Kullback–Leibler divergence < Kullback–Leibler divergence threshold, divide the new data points into clusters according to the distance from the points to the cluster centers, and update the cluster centers; and project the new data points into the scatter plot by using the out-of-sample extension method.

6. The incremental interactive clustering visualization method for trusted sharing of cloud data as claimed in claim 4, characterized in that: In S533, the re-clustering method is adopted to implement the clustering analysis and visualization operation of incremental data, including: when the Kullback–Leibler divergence > Kullback–Leibler divergence threshold, re-cluster the data set, find the best overlapping position with the dimensionality reduction result of the target data and project it; and color-code the projected points after dimensionality reduction according to the clustering results.

7. An incremental interactive clustering visualization system for trusted sharing of cloud data, characterized by: including: A writing and running module, which is used to write a data sharing smart contract and run the smart contract; An extraction module, which is used to extract target data from the data set according to the written data sharing smart contract; A clustering module, which is used to cluster the extracted target data by using a clustering algorithm and output the clustering results; A visualization module, which is used to perform multi-dimensional scaling dimensionality reduction on the clustering results, realize the projection of the clustering results in a two-dimensional space, and perform visual display; According to the user's changing requirements for the target data, perform corresponding projection interaction, clustering analysis and visualization operations; specifically including: If the user changes the attributes of the target data, construct a forward projection, display the result of the transformed data attributes on the visualization interface, and realize the forward projection interaction; If the user drags and drops the visualization data points of the target data, construct a backward projection, calculate the adjusted attributes of the data points, and realize the backward projection interaction; If the user adds incremental data on the basis of the target data, perform the clustering analysis and visualization operation of the incremental data; The forward projection interaction includes: Obtain the data points after attribute change and the centroids of each cluster in the current clustering model; Calculate the Euclidean distances from the data points to the centroids of each cluster respectively; Obtain the shortest distance among the Euclidean distances, and assign the data points to the cluster corresponding to the shortest distance; Project the data points in the scatter plot and color-code them with the same color of the corresponding cluster to realize the forward projection interaction; The backward projection interaction includes: Obtain the low-dimensional data of the projected points after dragging and dropping the visualization data points; According to the low-dimensional data, calculate the change vector from the original projected points to the new projected points; Adopt the PCA dimensionality reduction method to calculate Δx: Δx[e0 e1] = Δy where Δx is the feature change vector of the original data points; [e0 e1] is the feature vector matrix of the original data points; Δy is the position change vector; Solve the optimal solution of Δx according to the regularized least squares method to obtain the adjusted attributes of the data points and realize the backward projection interaction.

Images