High Dimensional Data Processing Method Based on Deep Manifold Transformation Network

Abstract

The present application proposes a high-dimensional data processing method based on a deep manifold transformation network, the method comprising: obtaining an input space; reducing the dimension of the input space to a latent space through a first nonlinear transformation, and reducing the dimension of the latent space to an embedded space through a second nonlinear transformation; applying a bidirectional divergence loss between the input space and the latent space, and / or applying a bidirectional divergence loss between the latent space and the embedded space, and / or applying a bidirectional divergence loss between the input space and the embedded space, so that the first nonlinear transformation and the second nonlinear transformation maintain the same structure; clustering the data of the input space in the latent space, and visualizing the reduced-dimensional data in the embedded space. Through the present invention, applying a bidirectional divergence loss between any two layers ensures the local smoothness of the network, prevents the destruction of the geometric or topological structure in the original data caused by dimensionality reduction, causes information loss, and deteriorates the effect of the final clustering.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a high-dimensional data processing method based on a deep manifold transformation network. Background Art

[0002] High-dimensional data analysis includes three basic tasks: data dimensionality reduction, clustering, and visualization.

[0003] Classic clustering algorithms, such as K-means clustering, test the clustering effects of all K values, analyze the intra-class similarity and inter-class similarity of the clustering results, and select the k value corresponding to the optimal clustering effect from a large number of clustering results. Similarly, clustering based on Gaussian mixture models and spectral clustering also perform clustering through some distance or similarity metrics defined in high-dimensional input space. However, due to the inherent non-Euclidean characteristics of high-dimensional data, that is, non-Euclidean data is not arranged neatly, it is difficult to define the neighboring nodes for a certain point in the data, or the number of neighboring nodes of different nodes is different. Therefore, when these common clustering algorithms are applied to high-dimensional data, the clustering performance is generally very poor when faced with data with complex structural distribution.

[0004] To deal with this problem, some methods first use data dimensionality reduction methods such as principal component analysis to reduce the data from the high-dimensional input space to a lower-dimensional latent space, and then run the classic clustering algorithm in this lower-dimensional latent space. However, at this time, data dimensionality reduction and clustering are two completely separate processes. The process of data dimensionality reduction may destroy the geometric or topological structure of the original data, resulting in information loss, which will in turn make the clustering effect worse.

[0005] T-SNE and UMAP are the two most popular nonlinear data visualization methods. Both methods first transform the Euclidean distance between sample points in the input space into similarity in a low-dimensional embedding space, and then find a suitable low-dimensional embedding by minimizing the cost function in the form of KL divergence or cross entropy to achieve data visualization. The defects of T-SNE and UMAP are: (1) They may destroy the geometric or topological structure of the original data during the process of data dimensionality reduction, resulting in information loss; (2) They may cause one-to-many mapping, and the same sample point is mapped to multiple different values ​​after dimensionality reduction; (3) They are both non-deep methods that directly optimize embedding rather than optimize network parameters, making them difficult to combine with existing deep learning technologies.

[0006] Data dimensionality reduction, clustering and visualization are three basic tasks closely related to high-dimensional data analysis. However, currently these three tasks are generally completed independently, which not only affects their performance, but also easily leads to inconsistencies between tasks. There is no way to truly reveal the inherent geometric and topological information of high-dimensional data, which makes it easy to make mistakes and draw misleading conclusions during data analysis.

[0007] In summary, there is currently no effective solution to the above-mentioned problems existing in high-dimensional data analysis in the prior art. Summary of the invention

[0008] The embodiment of the present application provides a high-dimensional data processing method based on a deep manifold transformation network, which can apply an unsupervised bidirectional divergence loss between any layers of the network, thereby ensuring that the mapping of the neural network is stable and smooth, and avoiding data collapse and over-smoothing problems during the mapping process.

[0009] In a first aspect, an embodiment of the present application provides a deep manifold transformation network, including an autoencoder, wherein the autoencoder is configured to include an encoder network and a decoder network, wherein the encoder network and the decoder each include a plurality of dense block layers,

[0010] In the encoder network, the dimension of the input space is reduced to the latent space through a first nonlinear transformation of multiple dense block layers, the dimension of the latent space is reduced to the embedding space through a second nonlinear transformation of multiple dense block layers, and in the decoder network, the dimension of the latent space is restored to the reconstruction space through a third nonlinear transformation of multiple dense block layers;

[0011] Calculate the reconstruction loss based on the input space and the reconstruction space, and add the reconstruction loss to the dense block layer;

[0012] A bidirectional divergence loss is applied between the input space and the latent space and / or between the latent space and the embedding space and / or between the input space and the embedding space.

[0013] In a second aspect, an embodiment of the present application provides a high-dimensional data processing method based on a deep manifold transformation network, comprising:

[0014] Get input space;

[0015] The dimension of the input space is reduced to the latent space through the first nonlinear transformation, and the dimension of the latent space is reduced to the embedded space through the second nonlinear transformation;

[0016] Applying a bidirectional divergence loss between the input space and the latent space, and / or applying a bidirectional divergence loss between the latent space and the embedded space, and / or applying a bidirectional divergence loss between the input space and the embedded space, so that the first nonlinear transformation and the second nonlinear transformation maintain the structure unchanged;

[0017] Cluster the data in the input space in the latent space and visualize the clustered data in the embedded space.

[0018] In a third aspect, the present application proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the high-dimensional data processing method based on a deep manifold transformation network as described in the first aspect.

[0019] In a fourth aspect, the present application proposes a storage medium storing a computer program, wherein the computer program is configured to execute, when running, the high-dimensional data processing method based on a deep manifold transformation network as described in the first aspect above.

[0020] The main contributions and innovations of the present invention are as follows:

[0021] 1. This scheme proposes a framework of deep manifold transformation, and proposes an autoencoder based on this framework, which can realize manifold reconstruction and manifold generation. By adding unsupervised bidirectional divergence loss between any layers of the network, the autoencoder model mapping is guaranteed to be stable and smooth, avoiding data collapse or over-smoothing problems.

[0022] 2. This scheme proposes a high-dimensional data processing method based on a deep manifold change network, which unifies the three basic tasks closely related to high-dimensional data analysis: data dimensionality reduction, clustering, and visualization. It solves the problem of inconsistency in the geometric structures of these three tasks when they run independently, and can ensure the consistency of clustering and visualization to the greatest extent while maintaining the changes in the structure.

[0023] 3. This scheme can also complete downstream tasks such as classification and regression by optimizing cost functions in the form of cross entropy and mean square error.

[0024] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a model architecture diagram of a deep manifold transformation network according to an embodiment of the present application;

[0027] Figure 2 is a flowchart of a high-dimensional data processing method based on a deep manifold transformation network according to an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a high-dimensional data processing method in Embodiment 2 of the present application;

[0029] Figure 4 This is an analysis effect diagram of the consistency between clustering and visualization of the present solution and the comparative manifold learning algorithm on 4 different data sets in Example 2 of the present application;

[0030] Figure 5 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0032] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0033] Embodiment 1

[0034] Figure 1 is a model architecture diagram of a deep manifold transformation network according to an embodiment of the present application, such as Figure 1 The deep manifold transformation network includes an automatic encoder, which is configured to include an encoder network and a decoder network, wherein the encoder network and the decoder respectively include a plurality of dense block layers,

[0035] In the encoder network, the dimension of the input space is reduced to the latent space through a first nonlinear transformation of multiple dense block layers, the dimension of the latent space is reduced to the embedding space through a second nonlinear transformation of multiple dense block layers, and in the decoder network, the dimension of the latent space is restored to the reconstruction space through a third nonlinear transformation of multiple dense block layers;

[0036] Calculate the reconstruction loss based on the input space and the reconstruction space, and add the reconstruction loss to the dense block layer;

[0037] A bidirectional divergence loss is applied between the input space and the latent space and / or between the latent space and the embedding space and / or between the input space and the embedding space.

[0038] refer to Figure 1 , the superscript X represents the input-output space, α represents a specific input-output instance, ML-Encoder represents the encoder network, and ML-Decoder represents the decoder network. In the autoencoder, a multi-layer perceptron (MLP), that is, a stack of multiple fully connected layers with nonlinear activation functions such as ReLu or LeakReLu, is used to achieve nonlinear changes. In the process of dimensionality reduction, the reconstruction loss is calculated and added to the dense block layer. This loss can be added to the corresponding dense block layer to ensure that the mapping does not lose any information.

[0039] This solution is different from the existing technology in that the reconstruction loss and the bidirectional divergence loss are used together to train the deep manifold transformation network. The bidirectional divergence loss is applied between any two layers to ensure the local smoothness of the network, prevent the destruction of the geometric or topological structure in the original data caused by dimensionality reduction, cause information loss, and make the final clustering effect worse.

[0040] Specifically, the acquisition of bidirectional divergence loss includes:

[0041] Obtain sample point pairs of any two spatial layers of the input space, latent space, and embedded space respectively, and convert the Euclidean distance between the sample point pairs into similarity through t-distribution;

[0042] Based on the similarity of each pair of sample points, a structure preservation loss and a structure change loss are generated, and the bidirectional divergence loss is calculated by the linear weighted sum method. The structure preservation loss is used to maintain the distance between each sample point and its neighboring sample points, and the structure change loss is used to increase the distance between each sample point and its distant neighboring sample points.

[0043] In this embodiment, it should be noted that the bidirectional divergence loss is essentially a loss function that can be added between any spatial layers. The meaning of "bidirectional" is that this loss is composed of two terms. The first term is used to maintain the distance / geometric structure of each sample point and its neighboring sample points, and the second term is used to increase the distance between each sample point and its distant neighboring sample points, that is, to "push away" those distant neighboring sample points. Each sample point and several of its neighboring sample points constitute a sample point pair.

[0044] Exemplarily, the bidirectional divergence loss calculation process mainly includes the following steps:

[0045] 1. The Euclidean distance between the sample points i and j in layer l Converted to similarity through t-distribution formula

[0046] 2. The Euclidean distance between the sample points i and j in the l′ layer Converted to similarity through t-distribution formula

[0047] 3. Obtain all sample point pairs and calculate and Substitute the following formula to get the bidirectional divergence loss

[0048]

[0049] In the formula, As the weight of the structure preservation loss, As the weight of the structure change loss, the bidirectional divergence loss is calculated by the linear weighted sum method.

[0050] Embodiment 2

[0051] Based on the same concept, this application also proposes a high-dimensional data processing method based on a deep manifold transformation network, referring to Figure 2 , the method comprising:

[0052] S201, obtaining input space;

[0053] S202, reducing the dimension of the input space to the latent space through a first nonlinear transformation, and reducing the dimension of the latent space to the embedded space through a second nonlinear transformation;

[0054] S203, applying a bidirectional divergence loss between the input space and the latent space, and / or applying a bidirectional divergence loss between the latent space and the embedded space, and / or applying a bidirectional divergence loss between the input space and the embedded space, so that the first nonlinear transformation and the second nonlinear transformation maintain the same structure;

[0055] S204: Cluster the data in the input space in the latent space, and visualize the reduced-dimensional data in the embedded space.

[0056] In step S201, refer to Figure 3 The input space refers to the set of original input data. The original data in the input space is reduced to a latent space of lower dimension through a first nonlinear change, and then the data is reduced from the latent space to an embedded space through a second nonlinear change. The bidirectional divergence loss in the embodiment is applied between the three cross-layer spaces of input space-hidden space, latent space-embedded space, and input space-embedded space, so that the first nonlinear transformation and the second nonlinear transformation maintain the structure unchanged.

[0057] In step S203, geometric structure preservation means that during the data dimensionality reduction process, the distance relationship between sample point pairs in different spaces remains consistent. For example, there are three samples A, B, and C. The distance between AB in the input space is 10, and the distance between AC is 100; after dimensionality reduction to the latent space, the distance between AB becomes 1, and the distance between AC becomes 10; although the distances between AB and AC are reduced, the distance between AC is still greater than the distance between AB. Therefore, a bidirectional divergence loss is applied between different spatial layers so that the original data can maintain its geometric or topological structure during the dimensionality reduction process, preventing the problem of poor aggregation effect caused by information loss.

[0058] In step S204, since the two nonlinear transformations from the input space to the latent space and from the latent space to the embedded space in this application are structure-preserving, the consistency between clustering and visualization can be guaranteed to the greatest extent. Figure 4 , Figure 4 This is an analysis of the consistency of aggregation and visualization on four different datasets, among which, Figure 4 It consists of four sub-graphs. The first row of each sub-graph shows the effect of coloring the visualization results according to the true label of each sample, and the second row shows the effect of coloring the visualization results according to the clustering predicted label of each sample. The first column of each sub-graph is the clustering structure comparison diagram of our scheme, the second column is the Raw Data in the prior art, the third column is the t-SNE or AE+TSNE in the prior art, and the fourth column is the UMAP or t-SNE in the prior art. From the annotations on the boxes in the figure, it can be seen that the clustering results based on other manifold learning methods are prone to the following two types of inconsistencies: (1) samples of the same class are clustered into two clusters in visualization; (2) samples of two different classes are clustered into the same cluster in visualization. The framework proposed in this scheme, because it maximizes the consistency between clustering and visualization when there are nonlinear changes, therefore, in visualization, samples of the same class are clustered into one cluster, and samples of different classes are clustered into different clusters. In other words, the deep manifold transformation network and the high-dimensional data processing method based on the deep manifold transformation network proposed in this scheme overcome the problem that existing algorithms have no way to truly reveal the intrinsic geometric and topological information of high-dimensional data, which makes it easy to make errors and misleading conclusions during data analysis. They surpass the comparative manifold learning methods in both clustering and visualization effects.

[0059] Embodiment 3

[0060] The new deep manifold transformation framework proposed in this application can also be used for downstream tasks such as classification and regression. Specifically, the data in the input space is classified by optimizing the cross entropy function in the latent space; and / or the data in the input space is regressed by the mean square error function in the latent space and / or in the embedded space, the data points obtained by dimensionality reduction are plotted in the coordinate system to achieve data visualization.

[0061] It is worth noting that the deep popular transformation network proposed in this application is a flexible and efficient framework that solves the problem of information loss caused by the destruction of the geometric / topological structure of the original data in the process of dimensionality reduction, clustering, and visualization by existing algorithms. This framework can be combined with various existing classification, regression, and clustering algorithms to improve model performance.

[0062] This embodiment also provides an electronic device, referring to Figure 5 , comprises a memory 504 and a processor 502, wherein the memory 504 stores a computer program, and the processor 502 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0063] Specifically, the processor 502 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0064] Among them, the memory 504 may include a large capacity memory 504 for data or instructions. By way of example and not limitation, the memory 504 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 504 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 504 may be inside or outside the data processing device. In a specific embodiment, the memory 504 is a non-volatile memory. In a specific embodiment, the memory 504 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory 504 (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.

[0065] The memory 504 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 502 .

[0066] The processor 502 implements any one of the high-dimensional data processing methods based on the deep manifold transformation network in the above embodiments by reading and executing the computer program instructions stored in the memory 504.

[0067] Optionally, the electronic device may further include a transmission device 506 and an input / output device 508 , wherein the transmission device 506 is connected to the processor 502 , and the input / output device 508 is connected to the processor 502 .

[0068] The transmission device 506 can be used to receive or send data via a network. The specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 506 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0069] The input / output device 508 is used to input or output information. For example, the input / output device may be a display screen, a speaker, a microphone, a mouse, a keyboard or other devices. In this embodiment, the input information may be classified raw data such as images, real-time video, or the input information may be a visualized classification result, etc.

[0070] Optionally, in this embodiment, the processor 502 may be configured to perform the following steps through a computer program:

[0071] Step S201, obtaining input space;

[0072] Step S202: reducing the dimension of the input space to the latent space through a first nonlinear transformation, and reducing the dimension of the latent space to the embedded space through a second nonlinear transformation;

[0073] Step S203: applying a bidirectional divergence loss between the input space and the latent space, and / or applying a bidirectional divergence loss between the latent space and the embedded space, and / or applying a bidirectional divergence loss between the input space and the embedded space, so that the first nonlinear transformation and the second nonlinear transformation maintain the same structure;

[0074] Step S204: cluster the data in the input space in the latent space, and visualize the reduced-dimensional data in the embedded space.

[0075] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0076] In addition, in combination with a high-dimensional data processing method based on a deep manifold transformation network in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any high-dimensional data processing method based on a deep manifold transformation network in the above embodiment is implemented.

[0077] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A deep manifold transformation network, comprising an autoencoder, characterized in that: The autoencoder is configured to include an encoder network and a decoder network, wherein the encoder network and the decoder each include a plurality of dense block layers, In the encoder network, the dimension of the input space is reduced to the latent space through a first nonlinear transformation of multiple dense block layers, the dimension of the latent space is reduced to the embedding space through a second nonlinear transformation of multiple dense block layers, and in the decoder network, the dimension of the latent space is restored to the reconstruction space through a third nonlinear transformation of multiple dense block layers; Calculate the reconstruction loss based on the input space and the reconstruction space, and add the reconstruction loss to the dense block layer; Applying a bidirectional divergence loss between the input space and the latent space and / or between the latent space and the embedding space and / or between the input space and the embedding space; in: Obtain sample point pairs of any two spatial layers of the input space, latent space, and embedded space respectively, and convert the Euclidean distance between the sample point pairs into similarity through t-distribution; Based on the similarity of each pair of sample points, a structure preservation loss and a structure change loss are generated, and the bidirectional divergence loss is calculated by the linear weighted sum method, where the structure preservation loss is used to maintain the distance between each sample point and its neighboring sample points, and the structure change loss is used to increase the distance between each sample point and its distant neighboring sample points; in: The Euclidean distance between the sample point pairs i and j in layer l Converted to similarity through t-distribution formula Will l ′ The Euclidean distance between the sample points i and j of the layer Converted to similarity through t-distribution formula Get all sample point pairs and calculate and Substitute the following formula to get the bidirectional divergence loss Among them, L represents the bidirectional divergence loss, N represents the number of sample point pairs, represents the similarity of layer l, represents the loss of structure preservation, Represents structural change loss.

2. A high-dimensional data processing method based on a deep manifold transformation network, characterized in that: include: Get input space; The dimension of the input space is reduced to the latent space through the first nonlinear transformation, and the dimension of the latent space is reduced to the embedded space through the second nonlinear transformation; Applying a bidirectional divergence loss between the input space and the latent space, and / or applying a bidirectional divergence loss between the latent space and the embedded space, and / or applying a bidirectional divergence loss between the input space and the embedded space, so that the first nonlinear transformation and the second nonlinear transformation maintain the structure unchanged; Cluster the data in the input space in the latent space and visualize the reduced-dimensional data in the embedded space; in: Obtain sample point pairs of any two spatial layers of the input space, latent space, and embedded space respectively, and convert the Euclidean distance between the sample point pairs into similarity through t-distribution; Based on the similarity of each pair of sample points, a structure preservation loss and a structure change loss are generated, and the bidirectional divergence loss is calculated by the linear weighted sum method, where the structure preservation loss is used to maintain the distance between each sample point and its neighboring sample points, and the structure change loss is used to increase the distance between each sample point and its distant neighboring sample points; in: The Euclidean distance between the sample point pairs i and j in layer l Converted to similarity through t-distribution formula Will l ′ The Euclidean distance between the sample points i and j of the layer Converted to similarity through t-distribution formula Get all sample point pairs and calculate and Substitute the following formula to get the bidirectional divergence loss Among them, L represents the bidirectional divergence loss, N represents the number of sample point pairs, represents the similarity of layer l, represents the loss of structure preservation, Represents structural change loss.

3. The high-dimensional data processing method based on the deep manifold transformation network according to claim 2 is characterized in that: include: The data in the input space is classified by optimizing the cross entropy function in the latent space.

4. The high-dimensional data processing method based on the deep manifold transformation network according to claim 2 is characterized in that: include: In the latent space, the data in the input space is regressed through the function of the mean square error.

5. The high-dimensional data processing method based on the deep manifold transformation network according to claim 2 is characterized in that: include: In the embedding space, the data points obtained by dimensionality reduction are plotted in a coordinate system to achieve data visualization.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the high-dimensional data processing method based on a deep manifold transformation network according to any one of claims 2 to 5.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the high-dimensional data processing method based on a deep manifold transformation network according to any one of claims 2 to 5 when running.

Images