Incremental manifold learning-based noisy data dimension reduction method and system

Abstract

The application relates to a kind of incremental manifold learning-based noisy data dimension reduction method and system, belong to data processing field.The method comprises: obtaining original data containing noise and incremental new data;The original data containing noise is de-noised to obtain the original data after de-noising;According to the original data containing noise and the original data after de-noising, the noise of the original data containing noise is determined;The noise is analyzed, and the probability density function of the noise is determined;According to the probability density function, the manifold boundary of the original data containing noise is determined;According to the manifold boundary, the distribution state of the incremental new data is determined;According to the distribution state, the incremental new data is processed, and the dimension-reduced incremental new data is obtained.The application improves the accuracy of data dimension reduction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data dimensionality reduction in machine learning, and particularly relates to a noisy data dimensionality reduction method and system based on incremental manifold learning. BACKGROUND

[0002] As a nonlinear dimensionality reduction method, manifold learning can mine the low-dimensional manifold of high-dimensional data, thereby solving the problem of "dimension disaster", and is widely used in pattern recognition and machine learning. However, there are two key problems that limit its further development, one is the noise interference problem, most of the manifold learning algorithms based on spectral graph theory use K-NN criterion to construct a neighborhood graph to obtain a limited description of the manifold structure, however, when there is noise, the algorithm is disturbed by the noise and cannot accurately describe the manifold structure of the data set, resulting in significant deviation in the subsequent low-dimensional mapping process; the other is the incremental manifold learning problem, most of the manifold learning algorithms are offline batch algorithms and cannot solve the online incremental learning problem, in addition, considering the independent and identically distributed characteristics of the incremental new data and the original data, the new data may be distributed on the manifold form formed by the original data or outside the manifold form formed by the original data, that is, on the extension structure of the original manifold form, at this time, it is impossible to obtain the low-dimensional features of the new data by only constructing the neighborhood relationship with the original data as the existing incremental manifold learning algorithm does. SUMMARY

[0003] The purpose of the present application is to provide a noisy data dimensionality reduction method and system based on incremental manifold learning, to solve the problem that the existing data dimensionality reduction method is disturbed by noise, resulting in inaccurate dimensionality reduction results.

[0004] To achieve the above purpose, the present application provides the following scheme:

[0005] A noisy data dimensionality reduction method based on incremental manifold learning, comprising:

[0006] obtaining noisy original data and incremental new data;

[0007] performing noise reduction processing on the noisy original data to obtain denoised original data;

[0008] determining the noise of the noisy original data according to the noisy original data and the denoised original data;

[0009] performing probability analysis on the noise to determine the probability density function of the noise;

[0010] determining the manifold boundary of the noisy original data according to the probability density function;

[0011] determine a distribution state of the incremental new data according to the manifold boundary; the distribution state comprises that the incremental new data is distributed on an extended structure of the original manifold shape or the incremental new data is distributed on the original manifold shape;

[0012] perform dimension reduction processing on the incremental new data according to the distribution state, to obtain reduced dimension incremental new data.

[0013] Optionally, the noise-containing original data is subjected to noise reduction processing to obtain noise-reduced original data, and the noise reduction processing specifically comprises:

[0014] K-NN algorithm is adopted to traverse the noise-containing original data, to obtain neighbor points of the noise-containing original data;

[0015] least square method is used to perform hyperplane fitting on the neighbor points of the noise-containing original data, to determine a hyperplane;

[0016] a normal direction of the hyperplane and Euclidean distances from the neighbor points of the noise-containing original data to the hyperplane are obtained;

[0017] the neighbor points of the noise-containing original data are projected onto the hyperplane according to the normal direction and the Euclidean distances, to obtain the noise-reduced original data.

[0018] Optionally, the manifold boundary of the noise-containing original data is determined according to the probability density function, and the determination specifically comprises:

[0019] the digital features of the noise are determined according to the probability density function; the digital features comprise mean value and standard deviation;

[0020] the manifold boundary of the noise-containing original data is determined according to the digital features.

[0021] Optionally, after the manifold boundary of the noise-containing original data is determined according to the probability density function, the method further comprises:

[0022] a weighted graph is constructed in a high-dimensional space according to the noise-reduced original data;

[0023] UMAP algorithm is used to map the weighted graph to a low-dimensional space, to obtain reduced dimension original data.

[0024] Optionally, the distribution state of the incremental new data is determined according to the manifold boundary, and the determination specifically comprises:

[0025] the incremental new data is clustered to obtain a plurality of cluster center points;

[0026] local outlier factor detection algorithm is used to detect outliers of the cluster center points, to determine a local outlier factor;

[0027] determining whether the density difference between the cluster center point and the manifold boundary is greater than a first preset value according to the local outlier factor;

[0028] when the density difference is greater than the first preset value, determining that the incremental new data is distributed on an extension structure of the original manifold shape;

[0029] when the density difference is less than the first preset value, determining that the incremental new data is distributed on the original manifold shape.

[0030] Optionally, the dimensionality reduction processing is performed on the incremental new data according to the distribution state, to obtain reduced dimension incremental new data, and specifically includes:

[0031] when the incremental new data is distributed on the extension structure of the original manifold shape:

[0032] determining a neighbor point of the incremental new data according to the denoised original data;

[0033] determining first low-dimensional data of the neighbor point of the incremental new data;

[0034] calculating weights of the incremental new data and the neighbor point of the incremental new data in a high-dimensional space;

[0035] according to the weights, performing weighted reconstruction on the incremental new data in a low-dimensional space by using the first low-dimensional data, to obtain the reduced dimension incremental new data;

[0036] when the incremental new data is distributed on the original manifold shape:

[0037] determining a neighbor point of the incremental new data according to the denoised original data;

[0038] determining second low-dimensional data of the neighbor point of the incremental new data;

[0039] setting a linear mapping function, linearly mapping the neighbor point of the incremental new data from a high-dimensional space to a low-dimensional space by using the linear mapping function, to obtain third low-dimensional data of the neighbor point of the incremental new data;

[0040] when the difference between the third low-dimensional data and the second low-dimensional data is reduced to a second preset value, mapping the incremental new data to the low-dimensional space by using the linear mapping function, to obtain the reduced dimension incremental new data.

[0041] A noisy data dimensionality reduction system based on incremental manifold learning includes:

[0042] a data acquisition module configured to acquire original noisy data and incremental new data;

[0043] a noise reduction module, configured to perform noise reduction processing on the original data with noise to obtain original data after noise reduction;

[0044] a noise determination module, configured to determine noise of the original data with noise according to the original data with noise and the original data after noise reduction;

[0045] a probability analysis module, configured to perform probability analysis on the noise to determine a probability density function of the noise;

[0046] a manifold boundary determination module, configured to determine a manifold boundary of the original data with noise according to the probability density function;

[0047] a distribution state determination module, configured to determine a distribution state of the incremental new data according to the manifold boundary; the distribution state includes that the incremental new data is distributed on an extension structure of an original manifold form and that the incremental new data is distributed on the original manifold form;

[0048] a first dimension reduction module, configured to perform dimension reduction processing on the incremental new data according to the distribution state to obtain reduced incremental new data.

[0049] Optionally, the noise reduction module comprises:

[0050] a near neighbor point acquisition unit, configured to traverse the original data with noise by using a K-NN algorithm to acquire near neighbor points of the original data with noise;

[0051] a hyperplane determination unit, configured to perform hyperplane fitting on the near neighbor points of the original data with noise by using a least square method to determine a hyperplane;

[0052] a data acquisition unit, configured to acquire a normal direction of the hyperplane and calculate Euclidean distances from the near neighbor points of the original data with noise to the hyperplane;

[0053] a noise reduction unit, configured to project the near neighbor points of the original data with noise to the hyperplane according to the normal direction and the Euclidean distances to obtain the original data after noise reduction.

[0054] Optionally, the manifold boundary determination module comprises:

[0055] a digital feature determination unit, configured to determine a digital feature of the noise according to the probability density function; the digital feature includes a mean value and a standard deviation;

[0056] a manifold boundary determination unit, configured to determine the manifold boundary of the original data with noise according to the digital feature.

[0057] Optionally, the distribution state determination module comprises:

[0058] a clustering unit, configured to cluster the incremental new data to obtain a plurality of cluster centers;

[0059] an outlier detection unit, configured to detect outliers of the cluster centers by using a local outlier factor detection algorithm to determine a local outlier factor;

[0060] a judging unit, configured to judge whether a density difference between the cluster centers and the manifold boundary is greater than a first preset value according to the local outlier factor;

[0061] a first distribution state determining unit, configured to determine that the incremental new data is distributed on an extension structure of the original manifold shape when the density difference is greater than the first preset value;

[0062] a second distribution state determining unit, configured to determine that the incremental new data is distributed on the original manifold shape when the density difference is less than the first preset value.

[0063] According to the embodiments of the present application, the following technical effects are provided.

[0064] The present application first performs noise reduction processing on the original data containing noise, determines the manifold shape formed by the original data as a main manifold after noise reduction, analyzes the probability distribution of the noise and represents the digital features of the noise on the main manifold to obtain the manifold boundary of the original data containing noise, and finally judges the distribution state of the incremental new data based on the manifold boundary and reduces the incremental new data in two distribution states to low dimensions on line using different dimension reduction criteria. The present application avoids the problems that the dimension reduction result is inaccurate due to the interference of the original data by noise and the low-dimensional features of the incremental new data cannot be accurately given when the incremental new data is distributed on the extension structure of the original manifold shape, thereby improving the accuracy of dimension reduction. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 a flow chart of the noise-containing data dimension reduction method based on incremental manifold learning provided by the present application;

[0067] Figure 2 a whole framework diagram of the noise-containing data dimension reduction method based on incremental manifold learning provided by the present application;

[0068] Figure 3An illustrative diagram of an image dataset provided by the present application; wherein, Figure 3 (a) is an illustrative diagram of the MNIST image dataset; Figure 3 (b) is an illustrative diagram of the FASHION-MNIST image dataset; Figure 3 (c) is an illustrative diagram of the DSPRITES image dataset;

[0069] Figure 4 An effect diagram of a noisy data dimensionality reduction method based on incremental manifold learning provided by the present application; wherein, Figure 4 (a) is an effect diagram before data dimensionality reduction; Figure 4 (b) is an effect diagram after data dimensionality reduction;

[0070] Figure 5 Low-dimensional feature maps of different data sets of the present application; wherein, Figure 5 (a) is a low-dimensional feature map with normally distributed noise added in the MNIST dataset; Figure 5 (b) is a low-dimensional feature map with normally distributed noise added in the FASHION-MNIST dataset; Figure 5 (c) is a low-dimensional feature map with normally distributed noise added in the DSPRITES dataset; Figure 5 (d) is a low-dimensional feature map with skew distributed noise added in the MNIST dataset; Figure 5 (e) is a low-dimensional feature map with skew distributed noise added in the FASHION-MNIST dataset; Figure 5 (f) is a low-dimensional feature map with skew distributed noise added in the DSPRITES dataset;

[0071] Figure 6 A structural diagram of a noisy data dimensionality reduction system based on incremental manifold learning provided by the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0073] The purpose of the present application is to provide a noisy data dimensionality reduction method and system based on incremental manifold learning to solve the problem that the existing data dimensionality reduction method is disturbed by noise, resulting in inaccurate dimensionality reduction results.

[0074] The application provides a noisy data dimension reduction method based on incremental manifold learning, which can reveal the low-dimensional essential features of incremental new data and original data containing noise and has good noise resistance.

[0075] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0076] Figure 1 A flowchart of the noisy data dimension reduction method based on incremental manifold learning provided by the present application is shown in the figure, Figure 2 A general framework diagram of the noisy data dimension reduction method based on incremental manifold learning provided by the present application is shown in the figure. Figure 1 and Figure 2 As shown in the figure, the noisy data dimension reduction method based on incremental manifold learning comprises:

[0077] Step 101: obtaining original data containing noise and incremental new data. In actual application, MNIST, FASHION-MNIST and DSPRITES three multi-class image data sets containing noise are input in sequence, and the images of the MNIST, FASHION-MNIST and DSPRITES image data sets are shown in the figure. Figure 3 As shown in the figure, the original data containing noise The sample number N of the original data containing noise in the MNIST data set, the FASHION-MNIST data set and the DSPRITES data set is 1500, 5000 and 5000 respectively; the incremental new data The sample number T of the incremental new data in the MNIST data set, the FASHION-MNIST data set and the DSPRITES data set is 600, 2000 and 2000 respectively. Noise or noise of gamma distribution (gamma distribution is a typical skew distribution) is added to the original data containing noise and the incremental new data

[0078] Step 102: performing noise reduction processing on the original data containing noise to obtain original data after noise reduction.

[0079] Further, the step 102 specifically comprises:

[0080] K-NN algorithm is adopted to traverse the original data containing noise and obtain the near neighbor points of the original data containing noise. In actual application, the near neighbor point set N of the original data containing noise is {N i |i=1,2…N}, wherein, x iThe neighbor points of the original data with noise, k=30.

[0081] The neighbor points of the original data with noise are fitted with a hyperplane by using the least square method to determine the hyperplane. In practical applications, it is assumed that the main manifold is a piecewise linear manifold composed of several local manifolds, and then the neighbor points of x i with noise are fitted with a hyperplane in the local manifold space by using the least square method to approximate the hyperplane as a local manifold.

[0082] The normal direction of the hyperplane and the Euclidean distance of the neighbor points of the original data with noise to the hyperplane are obtained. The normal direction u i of the hyperplane and the Euclidean distance d i of x i to the hyperplane are obtained by using the orthogonal projection theory.

[0083] The neighbor points of the original data with noise are projected to the hyperplane according to the normal direction and the Euclidean distance to obtain the original data after noise reduction. x i is projected to the hyperplane along the normal direction u i by a distance d i , thereby achieving the purpose of noise reduction, and the original data after noise reduction is represented as which is expressed in mathematical language as

[0084] Step 103: According to the original data with noise and the original data after noise reduction, the noise of the original data with noise is determined. In practical applications, after the original data with noise of the three image data sets are projected to the main manifold, their noises are calculated in turn.

[0085] Step 104: The noise is analyzed by probability to determine the probability density function of the noise. The noise is analyzed by probability by using the Parzen window model to infer the probability density function of the noise at different manifold space positions, wherein the window model is a multivariate Gaussian function, and the window width

[0086] Step 105: According to the probability density function, the manifold boundary of the original data with noise is determined.

[0087] Further, the step 105 specifically includes:

[0088] According to the probability density function, the digital features of the noise are determined; the digital features include mean and standard deviation. In practical applications, the noise is parameterized, and the digital features of the noise including mean u and standard deviation σ are calculated according to the obtained probability density function.

[0089] According to the digital feature, a manifold boundary of the noisy original data is determined. In actual application, in order to more intuitively describe the damage degree of noise to different manifold space positions, the interval is approximated as the manifold boundary of the noisy original data.

[0090] Specifically, the step 105 further includes:

[0091] According to the denoised original data, a weighted graph is constructed in a high-dimensional space.

[0092] By using a UMAP algorithm, the weighted graph is mapped to a low-dimensional space to obtain the reduced original data.

[0093] In actual application, after the projection denoising of the noisy original data, a weighted graph G=(V, E, W) is constructed in a high-dimensional space, where V represents a node set corresponding to the original data in the weighted graph, E represents an edge set connecting two nodes in the weighted graph, and W represents a weight value matrix assigned to the edge set, and the calculation of the weight value adopts a Gaussian kernel function; the UMAP algorithm is borrowed to map the weighted graph constructed in the high-dimensional space to a low-dimensional space, and the dimensionality reduction is realized by mining a graph most similar to the weighted graph in the low-dimensional space, and the original data of the three image data sets are sequentially reduced to 3-D, 5-D, 3-D, 10-D, 80-D, and 200-D, and the original data after the dimensionality reduction is represented as wherein,

[0094] Step 106: According to the manifold boundary, a distribution state of the incremental new data is determined. The distribution state includes that the incremental new data is distributed on the extended structure of the original manifold form and the incremental new data is distributed on the original manifold form.

[0095] Further, the step 106 specifically includes:

[0096] The incremental new data is clustered to obtain a plurality of cluster center points. In actual application, in order to avoid the calculation amount problem caused by traversing all the incremental new data, the K-Means algorithm is used to cluster the incremental new data X new to obtain t cluster center points as representative points of the incremental new data, wherein t=20.

[0097] The cluster center points are subjected to outlier detection by using a local outlier factor detection algorithm to determine a local outlier factor.

[0098] According to the local outlier factor, it is judged whether the density difference between the cluster center points and the manifold boundary is greater than a first preset value.

[0099] When the density difference is greater than a first preset value, it is determined that the incremental new data is distributed on an extension structure of the original manifold morphology.

[0100] When the density difference is less than the first preset value, it is determined that the incremental new data is distributed on the original manifold morphology.

[0101] In actual application, the local outlier factor detection algorithm (LOF) is used to detect outliers of the cluster center point. When the local outlier factor is much greater than 1, it means that the cluster center point has a large density difference with the manifold boundary.

[0102] When the cluster center point has a large density difference with the manifold boundary, the cluster center point is determined as an outlier. At this time, it is determined that the incremental new data is distributed on the extension structure of the original manifold morphology; otherwise, it is determined that the incremental new data is distributed on the original manifold morphology.

[0103] Step 107: Dimension reduction processing is performed on the incremental new data according to the distribution state, to obtain reduced incremental new data.

[0104] Further, the step 107 specifically includes:

[0105] When the incremental new data is distributed on the extension structure of the original manifold morphology: the neighbor points of the incremental new data are determined according to the denoised original data; the first low-dimensional data of the neighbor points of the incremental new data are determined; the weights of the incremental new data and the neighbor points of the incremental new data in the high-dimensional space are calculated; the incremental new data is weighted and reconstructed in the low-dimensional space by using the first low-dimensional data according to the weights, to obtain the reduced incremental new data.

[0106] When the incremental new data is distributed on the original manifold morphology: the neighbor points of the incremental new data are determined according to the denoised original data; the second low-dimensional data of the neighbor points of the incremental new data are determined; a linear mapping function is set, the neighbor points of the incremental new data are linearly mapped from the high-dimensional space to the low-dimensional space by using the linear mapping function, to obtain the third low-dimensional data of the neighbor points of the incremental new data; when the difference between the third low-dimensional data and the second low-dimensional data is reduced to a second preset value, the incremental new data is mapped to the low-dimensional space by using the linear mapping function, to obtain the reduced incremental new data.

[0107] In actual application, when the incremental new data is distributed on the original manifold morphology: first, the neighbor points of the incremental new data are searched in the denoised original data ​​The low-dimensional representation of Y is i Then, calculate and Weight w in high-dimensional space ij Finally, using Y in low-dimensional space i right Weighted reconstruction is performed to achieve dimensionality reduction; when incremental new data is distributed on the extended structure of the original manifold: firstly, in the denoised original data... Finding new incremental data nearest neighbor The low-dimensional representation of Y is i Then, let the linear mapping function be... This function will A linear mapping from a high-dimensional space to a low-dimensional space results in a low-dimensional representation. With Y i To minimize the difference, a mapping function is used to map the incremental new data to a low-dimensional space, thereby achieving the goal of dimensionality reduction.

[0108] Most manifold learning algorithms are highly sensitive to noise. When noise is present in the dataset, their ability to extract low-dimensional features decreases significantly. Furthermore, existing incremental manifold learning algorithms do not consider the distribution of new data; when new data is distributed on an extension of the original manifold shape, they fail to accurately provide low-dimensional features. The dimensionality reduction method for noisy data based on incremental manifold learning proposed in this invention avoids these problems.

[0109] To address the issue of noisy data, this invention utilizes projection denoising to project the noisy original data onto a smooth manifold, thereby processing noise that follows a normal or skewed distribution. This not only preserves the manifold structure of the dataset but also ensures excellent denoising performance, giving the invention good noise resistance.

[0110] This invention uses a nonparametric statistical model to perform probabilistic analysis on noise, thereby extracting the manifold boundary of the dataset. This enables the invention to determine the distribution state of incremental new data in a probabilistic sense, thus improving accuracy.

[0111] This invention provides an incremental manifold dimensionality reduction strategy for incremental new data. First, based on the distribution relationship between the incremental new data and the manifold boundary, the distribution state of the incremental new data is accurately determined. Then, using different dimensionality reduction criteria, the incremental new data under the two distribution states is reduced to a lower dimension online. This enables the invention to learn online and accurately reveal the low-dimensional essential characteristics of the incremental new data and the noisy original data.

[0112] The dimensionality reduction method for noisy data based on incremental manifold learning proposed in this invention exhibits excellent performance in dimensionality reduction tasks for synthetic datasets, such as...Figure 4 As shown.

[0113] As shown. Figure 4 It can be seen that when the incremental new data is distributed on the original manifold form, the form structure after dimension reduction is consistent with the original data, and the relative distance between the data is also retained. When the incremental new data is distributed on the extended structure of the original manifold form, the form structure after dimension reduction is also an extension of the structure of the original data containing noise, so that the global manifold structure of the incremental new data and the original data containing noise is unchanged. In summary, the method disclosed by the application can more accurately reveal the low-dimensional features contained in the incremental new data and the original data containing noise.

[0114] The three-dimensional visualization effect of the three image data sets after dimension reduction to 3-D based on the incremental manifold learning noisy data dimension reduction method proposed by the application is as shown in Figure 5 The K-Means clustering algorithm is used as a classifier to classify the incremental new data reduced to different dimensions, and the classification accuracy is calculated as an evaluation index. The classification accuracies of the MNIST, FASHION-MNIST and DSPRITES image data sets are shown in Tables 1, 2 and 3, respectively.

[0115] Table 1 shows the evaluation results of the MNIST data set

[0116]

[0117] Table 2 shows the evaluation results of the FASHION-MNIST data set

[0118]

[0119] Table 3 shows the evaluation results of the DSPRITES data set

[0120]

[0121] As shown. Figure 5 It can be seen that the incremental new data of the same class as the original data containing noise presents the characteristics of intra-class convergence after dimension reduction, while the incremental new data of different classes from the original data containing noise presents the characteristics of inter-class separation after dimension reduction, so that it has a large class distinction. As shown in Tables 1-3, the classification accuracy of the method proposed by the application achieves the optimal value on the three multi-class image data sets. In summary, when facing the multi-class image data set containing noise, the method proposed by the application has better classification performance and noise resistance.

[0122] Figure 6 The structure diagram of a noisy data dimension reduction system based on incremental manifold learning provided by the application is as shown in Figure 6 The noisy data dimension reduction system based on incremental manifold learning comprises:

[0123] The data acquisition module 601 is configured to acquire original data with noise and incremental new data.

[0124] The noise reduction module 602 is configured to perform noise reduction processing on the original data with noise to obtain original data after noise reduction.

[0125] The noise determination module 603 is configured to determine noise of the original data with noise according to the original data with noise and the original data after noise reduction.

[0126] The probability analysis module 604 is configured to perform probability analysis on the noise to determine a probability density function of the noise.

[0127] The manifold boundary determination module 605 is configured to determine a manifold boundary of the original data with noise according to the probability density function.

[0128] The distribution state determination module 606 is configured to determine a distribution state of the incremental new data according to the manifold boundary; the distribution state includes that the incremental new data is distributed on an extension structure of an original manifold form and that the incremental new data is distributed on the original manifold form.

[0129] The first dimension reduction module 607 is configured to perform dimension reduction processing on the incremental new data according to the distribution state to obtain incremental new data after dimension reduction.

[0130] Further, the noise reduction module 602 includes:

[0131] The near neighbor point acquisition unit is configured to traverse the original data with noise by using a K-NN algorithm to acquire near neighbor points of the original data with noise.

[0132] The hyperplane determination unit is configured to perform hyperplane fitting on the near neighbor points of the original data with noise by using a least square method to determine a hyperplane.

[0133] The data acquisition unit is configured to acquire a normal direction of the hyperplane and calculate Euclidean distances of the near neighbor points of the original data with noise to the hyperplane.

[0134] The noise reduction unit is configured to project the near neighbor points of the original data with noise to the hyperplane according to the normal direction and the Euclidean distances to obtain original data after noise reduction.

[0135] Further, the manifold boundary determination module 605 includes:

[0136] The digital feature determination unit is configured to determine a digital feature of the noise according to the probability density function; the digital feature includes a mean value and a standard deviation.

[0137] A manifold boundary determination unit is configured to determine a manifold boundary of the noisy original data according to the digital features.

[0138] Further, the distribution state determination module 606 comprises:

[0139] A clustering unit is configured to cluster the incremental new data to obtain a plurality of cluster center points.

[0140] An outlier detection unit is configured to detect outliers of the cluster center points by using a local outlier factor detection algorithm to determine a local outlier factor.

[0141] A judgment unit is configured to judge whether a density difference between the cluster center points and the manifold boundary is greater than a first preset value according to the local outlier factor.

[0142] A first distribution state determination unit is configured to determine that the incremental new data is distributed on an extended structure of an original manifold shape when the density difference is greater than the first preset value.

[0143] A second distribution state determination unit is configured to determine that the incremental new data is distributed on an original manifold shape when the density difference is less than the first preset value.

[0144] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, the description is relatively simple because it corresponds to the method disclosed in the embodiments. The relevant parts can be referred to the description of the method.

[0145] The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. For those skilled in the art, the specific implementation manners and application scope can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for dimensionality reduction of noisy data based on incremental manifold learning, characterized in that, The method comprises the following steps: obtaining original data with noise and incremental new data; performing noise reduction processing on the original data with noise to obtain original data after noise reduction; performing noise reduction processing on the original data with noise to obtain original data after noise reduction, specifically comprising: using K-NN algorithm to traverse the original data with noise to obtain the near neighbor points of the original data with noise; using least square method to fit the near neighbor points of the original data with noise to a hyperplane to determine the hyperplane; obtaining the normal direction of the hyperplane and calculating the Euclidean distance from the near neighbor points of the original data with noise to the hyperplane; projecting the near neighbor points of the original data with noise to the hyperplane according to the normal direction and the Euclidean distance to obtain the original data after noise reduction; determining the noise of the original data with noise according to the original data with noise and the original data after noise reduction; performing probability analysis on the noise to determine the probability density function of the noise; determining the manifold boundary of the original data with noise according to the probability density function; determining the manifold boundary of the original data with noise according to the probability density function, and then comprising the following steps: constructing a weighted graph in a high-dimensional space according to the original data after noise reduction; mapping the weighted graph to a low-dimensional space using UMAP algorithm to obtain the original data after dimension reduction; determining the distribution state of the incremental new data according to the manifold boundary; the distribution state comprises that the incremental new data is distributed on the extension structure of the original manifold form and the incremental new data is distributed on the original manifold form; determining the distribution state of the incremental new data according to the manifold boundary, specifically comprising: performing clustering on the incremental new data to obtain a plurality of cluster center points; detecting outliers of the cluster center points using local outlier factor detection algorithm to determine the local outlier factor; judging whether the density difference between the cluster center points and the manifold boundary is greater than a first preset value according to the local outlier factor; when the density difference is greater than the first preset value, it is determined that the incremental new data is distributed on the extension structure of the original manifold form; when the density difference is less than the first preset value, it is determined that the incremental new data is distributed on the original manifold form; performing dimension reduction processing on the incremental new data according to the distribution state to obtain the incremental new data after dimension reduction.

2. The incremental manifold learning based noisy data dimension reduction method according to claim 1, wherein, The method for determining the manifold boundary of the original data with noise according to the probability density function specifically comprises: determining the numerical characteristics of the noise according to the probability density function; the numerical characteristics include mean and standard deviation; determining the manifold boundary of the original data with noise according to the numerical characteristics.

3. The incremental manifold learning based noisy data dimension reduction method according to claim 1, wherein, The method for performing dimension reduction processing on the incremental new data according to the distribution state to obtain the incremental new data after dimension reduction specifically comprises: when the incremental new data is distributed on the extension structure of the original manifold form: determining the near neighbor points of the incremental new data according to the original data after noise reduction; determining the first low-dimensional data of the near neighbor points of the incremental new data; calculating the weight of the incremental new data and the near neighbor points of the incremental new data in the high-dimensional space; According to the weight, the first low-dimensional data is used to weight and reconstruct the incremental new data in a low-dimensional space, to obtain the dimension-reduced incremental new data; When the incremental new data is distributed on the original manifold form: According to the denoised original data, a neighbor point of the incremental new data is determined; Second low-dimensional data of the neighbor point of the incremental new data is determined; A linear mapping function is set, and the neighbor point of the incremental new data is linearly mapped from a high-dimensional space to a low-dimensional space by using the linear mapping function, to obtain third low-dimensional data of the neighbor point of the incremental new data; When the difference between the third low-dimensional data and the second low-dimensional data is reduced to a second preset value, the incremental new data is mapped to the low-dimensional space by using the linear mapping function, to obtain the dimension-reduced incremental new data.

4. A system for dimensionality reduction of noisy data based on incremental manifold learning, characterized in that, Comprise: A data acquisition module is configured to acquire noisy original data and incremental new data; A denoising module is configured to perform denoising processing on the noisy original data to obtain denoised original data; The denoising module comprises: A neighbor point acquisition unit is configured to traverse the noisy original data by using a K-NN algorithm to acquire a neighbor point of the noisy original data; A hyperplane determination unit is configured to perform hyperplane fitting on the neighbor point of the noisy original data by using a least square method to determine a hyperplane; A data acquisition unit is configured to acquire a normal direction of the hyperplane and calculate the Euclidean distance from the neighbor point of the noisy original data to the hyperplane; A denoising unit is configured to project the neighbor point of the noisy original data to the hyperplane according to the normal direction and the Euclidean distance to obtain the denoised original data; A noise determination module is configured to determine noise of the noisy original data according to the noisy original data and the denoised original data; A probability analysis module is configured to perform probability analysis on the noise to determine a probability density function of the noise; A manifold boundary determination module is configured to determine a manifold boundary of the noisy original data according to the probability density function; According to the probability density function, the manifold boundary of the noisy original data is determined, and then the method further comprises: According to the denoised original data, a weighted graph is constructed in a high-dimensional space; The weighted graph is mapped to a low-dimensional space by using a UMAP algorithm to obtain dimension-reduced original data; A distribution state determination module is configured to determine a distribution state of the incremental new data according to the manifold boundary; the distribution state comprises that the incremental new data is distributed on an extension structure of the original manifold form and that the incremental new data is distributed on the original manifold form; The distribution state determination module comprises: A clustering unit is configured to cluster the incremental new data to obtain a plurality of cluster center points; An outlier detection unit is configured to perform outlier detection on the cluster center points by using a local outlier factor detection algorithm to determine a local outlier factor; A judgment unit is configured to judge whether a density difference between the cluster center points and the manifold boundary is greater than a first preset value according to the local outlier factor; A first distribution state determination unit is configured to determine that the incremental new data is distributed on the extension structure of the original manifold form when the density difference is greater than the first preset value; The second distribution state determining unit is configured to determine that the incremental new data is distributed on the original manifold shape when the density difference is less than a first preset value. The first dimension reduction module is configured to perform dimension reduction processing on the incremental new data according to the distribution state, to obtain reduced incremental new data.

5. The incremental manifold learning based noisy data dimensionality reduction system of claim 4, wherein, The manifold boundary determining module comprises: The digital feature determining unit is configured to determine a digital feature of the noise according to the probability density function; the digital feature comprises a mean value and a standard deviation; The manifold boundary determining unit is configured to determine a manifold boundary of the original data containing the noise according to the digital feature.

Images