High-dimensional data processing method based on deep manifold transformation network

Abstract

The invention provides a high-dimensional data processing method based on a deep manifold transformation network. The method comprises the following steps: acquiring an input space; reducing the dimension of the input space to a hidden space through first nonlinear transformation, and reducing the dimension of the hidden space to an embedding space through second nonlinear transformation; applying bidirectional divergence loss between the input space and the hidden space, and / or applying bidirectional divergence loss between the hidden space and the embedding space, and / or applying bidirectional divergence loss between the input space and the embedding space, so that the structures of the first nonlinear transformation and the second nonlinear transformation are kept unchanged; and clustering the data of the input space in the hidden space, and visualizing the data after dimension reduction in the embedding space. The bidirectional divergence loss is applied between any two layers to ensure local smoothness of the network, and the situation that the final clustering effect becomes poor due to information loss caused by geometric or topological structure damage in original data due to dimension reduction is prevented.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to a high-dimensional data processing method based on a deep manifold transformation network. Background technique [0002] High-dimensional data analysis consists of three basic tasks: data dimensionality reduction, clustering, and visualization. [0003] Classical clustering algorithms, such as K-means clustering, analyze the intra-class similarity and inter-class similarity of the clustering results by testing the clustering effect of all K values, and select the optimal clustering effect corresponding to a large number of clustering results. value of k. Similarly, clustering and spectral clustering based on Gaussian mixture models are clustered by some distance or similarity measures defined in the high-dimensional input space. However, due to the inherent non-Euclidean characteristics of high-dimensional data, that is, the arrangement of Feeuclidian data is ...

AI Technical Summary

Problems solved by technology

The defects of T-SNE and UMAP are: (1) they may destroy the geometric or topological structure of the original data in the process of data dimensionality reduction, resulting in loss of information; (2) they may cause one-to-many Mapping, the same sample point is mapped to multiple different values ​​after dimensionality reduction; (3) They are all non-deep methods, which adopt the method of directly optimizing embedding instead of optimizing network parameters, and it is difficult to compare with existing deep learning techniques. combine

[0006] Data dimensionality reduction, clustering and visualization are three basic tasks closely related to high-dimensional data analysis. However, these three tasks are generally completed independently, which will not only affect their performance, but also easily make the differences between each task There are inconsistencies between them, and there is no way to truly reveal the inherent geometric and topological information of high-dimensional data, which makes it easy to make mistakes and draw misleading conclusions during data analysis

[0007] To sum up, for the above-mentioned problems existing in high-dimensional data analysis in the prior art, no effective solution has been obtained yet

Images