Data dimension reduction method and device

Abstract

The invention provides a data dimension reduction method and device. The method comprises the steps of constructing an initial neural network used for data dimension reduction, using a high-dimensional data set and a low-dimensional data set as input and output of the initial neural network respectively, wherein the dimension of sample points in the high-dimensional data set is larger than that ofsample points in the low-dimensional data set; constructing a neural network objective function based on the sample point distance relationship of the high-dimensional data set and the sample point distance relationship of the low-dimensional data set; optimizing and adjusting parameters of the initial neural network according to the neural network objective function; and carrying out dimension reduction processing on to-be-processed data by using the initial neural network after parameter optimization and adjustment. The low-dimensional data set obtained through the scheme can keep the global characteristics of the high-dimensional data set.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a data dimensionality reduction method and device. Background technique [0002] Data dimensionality reduction refers to reducing the dimensionality of high-dimensional data sets to low-dimensional data sets, and ensuring that the generated low-dimensional data sets are similar to the main information contained in the original high-dimensional data sets. In reality, many data sets to be processed and analyzed are usually large in volume and high in dimensionality. For example, in seismic exploration, in order to identify oil and gas development locations using reflected seismic waveforms, the data sets to be processed and analyzed may contain hundreds of Up to tens of millions of sample points, the dimension of each sample point may reach 100 dimensions. Data dimensionality reduction can reduce the time or space complexity of high-dimensional data sets, save the computat...

AI Technical Summary

Problems solved by technology

[0003] Existing data dimensionality reduction methods can be divided into two categories: linear methods and nonlinear methods: the most classic method in linear methods is principal component analysis (PCA), and PCA uses the K-L transformation with the smallest distortion under the mean square error criterion in linear algebra The original space data set is transformed into the feature vector space, but the dimensionality reduction results often have a certain degree of ambiguity, which is not as complete as the original sample, and the contribution rate is small, but the principal components containing important information of sample differences may be directly discarded, resulting in information loss; Representative methods of nonlinear methods include: kernel PCA method, local linear embedding method (LLE), isometric mapping method (ISOMap), etc. Among them, the dimensionality reduction effect of kernel PCA method depends on the selection of kernel function, and both LLE and ISOMap assume The dataset has a manifold structure and cannot accommodate all dataset types

[0004] There are deficiencies in the above methods: ① all involve matrix operations, which cannot adapt to the dimensionality reduction of large data sets; ② cannot memorize the characteristics of the data set, once new samples are added to the data set, it needs to be recalculated; ③ the dimensionality reduction results of some methods are not very good Preserve the distance relationship between global sample points in high-dimensional data sets

Images