Three-stage combined envelope embedded stack auto-encoder

Abstract

The invention discloses a three-stage joint envelope embedded stack auto-encoder, which comprises a three-stage joint envelope module, an embedded stack auto-encoder module and a three-layer space integration mechanism module, and is characterized in that the three-stage joint envelope module is internally provided with a manifold sample pair envelope module, a dimensionality reduction keeping type clustering module and a three-stage consistency keeping module; the manifold sample pair enveloping module is used for acquiring inherent structures of adjacent samples with similar manifolds in a sample enveloping space; the maintenance dimension reduction type clustering module is used for projecting samples to a low-dimensional space; the inter-three-stage consistency maintaining module is used for maintaining the consistency of sample distribution; after an input sample enters the three-stage combined envelope module, the input sample is processed and a three-layer sample space is generated; the built-in stack automatic encoder module carries out built-in stack automatic encoder modeling in parallel on the three layers of sample spaces, and finally the three layers of space integration mechanism modules carry out fusion and obtain output samples. The method has the advantages that the clustering state space is expanded, and the sample characterization capacity is improved.

Description

Technical Field

[0001] The invention relates to artificial intelligence technology, and in particular to a three-level joint envelope embedded stack autoencoder. Background Art

[0002] Machine learning is the core technology of artificial intelligence, and feature learning is of great significance in machine learning. Compared with traditional feature learning methods, the powerful feature learning ability of deep learning models enables them to automatically learn and discover complex nonlinear relationships in data. They perform better for high-dimensional and nonlinear data, and have achieved remarkable performance in various complex tasks. They have become a key technology in many fields. Stacked auto encoder (SAE), as a powerful deep learning method, can learn high-order representations of data through multi-layer nonlinear transformations, thereby capturing the complex structure and regularity of data. It has been widely used in computer vision, natural language processing, recommendation systems and other fields.

[0003] like Figure 1 As shown in the figure, the existing SAE model mainly considers minimizing the error between each input sample and the output sample by using the reconstructed deep features, while ignoring the structural information between samples. This will lead to a decrease in the separability between samples, limit the search for the optimal sample, limit the quality of the optimal features, and thus affect the classification performance of the algorithm. Although SAE can train samples in batches and indirectly obtain the associated structural information between samples, the number of samples in each batch should not be too large, otherwise it will seriously increase the complexity of the autoencoder. However, if it is too small, it cannot reflect the structural information between samples. Even if the entire sample can be input into the autoencoder for training, the number of possible combinations will surge, and the risk of overfitting will increase significantly. Therefore, before modeling SAE, it is necessary to comprehensively consider the associated structural information between input samples in order to better mine the optimal feature representation of each sample. However, the existing SAE does not take into account the problem of associated structural information between samples. Summary of the invention

[0004] In view of this, the present invention provides a three-stage joint envelope embedded stacked autoencoder to solve the problem that the existing stacked autoencoder does not consider the correlation structure information between samples.

[0005] In order to achieve the above object, the specific technical solution adopted by the present invention is as follows:

[0006] A three-level joint envelope embedded stack autoencoder, the key of which is that it includes a three-level joint envelope module, an embedded stack autoencoder module and a three-layer space integration mechanism module, wherein:

[0007] The three-level joint envelope module is provided with a manifold sample pair envelope module, a dimension reduction clustering module and a three-level consistency maintenance module; the manifold sample pair envelope module is used to obtain the inherent structure of manifold-similar adjacent samples in the sample envelope space; the dimension reduction clustering module is used to project samples into a low-dimensional space during the clustering process; the three-level consistency maintenance module is used to maintain the consistency of sample distribution;

[0008] After the input sample enters the three-level joint envelope module, it is jointly processed by the manifold sample pair envelope module, the dimension reduction clustering module and the three-level consistency maintenance module to generate a three-layer sample space; the embedded stack autoencoder module performs embedded stack autoencoder modeling in parallel on the three-layer sample space, and finally the three-layer space integration mechanism module fuses them and obtains the output sample.

[0009] Optionally, the three-layer sample space includes an OSS layer sample space, an MSPE layer sample space and an MDC layer sample space, and the three-level consistency maintaining module includes a first-level and second-level layer consistency maintaining module and a second-level and third-level layer consistency maintaining module, wherein:

[0010] The OSS layer sample space is the original sample space; the MSPE layer sample space is the sample space in which the input sample is processed by the manifold sample pair envelope module and the first and second layer consistency maintaining module is used to make it consistent with the distribution of the OSS layer sample space; the MDC layer sample space is the sample space in which the input sample is first processed by the manifold sample pair envelope module, then processed by the dimension reduction clustering module, and finally made consistent with the distribution of the MSPE layer sample space by the second and third layer consistency maintaining module.

[0011] Optionally, the manifold sample pair envelope module calculates the geodesic distance between the original sample and all other samples, and the sample with the smallest geodesic distance is considered as the nearest neighbor sample, and then concatenates the original sample with its respective multiple manifold nearest neighbor samples to generate a manifold nearest sample pair.

[0012] Optionally, the dimension reduction-preserving clustering module generates a clustered sample space by finding an optimal cluster center and an optimal projection matrix, wherein the optimal cluster center and the optimal projection matrix are determined by optimizing with the goal of minimizing clustering error.

[0013] Optionally, when optimizing with the goal of minimizing clustering error, the objective function also includes a projection matrix constraint term and a projection matrix regularization application term. The projection matrix constraint term constructs an affinity matrix through a local linear embedding method to effectively retain the local structural characteristics of data samples in a high-dimensional space, thereby better mining the relationships and similarities between data; the projection matrix regularization application term enhances the interpretability of the selected features in the projection space by using the L2,1 norm.

[0014] Optionally, in the dimension reduction-preserving clustering module, K-means is first used as a cluster center initialization method to initialize the cluster centers.

[0015] Optionally, the first and second layer consistency maintenance module optimizes the objective function composed of local distribution difference loss, global distribution difference loss and low-rank constraint regularization to obtain a first mapping function, and then generates a clustering envelope sample set that maintains local and global consistency of sample data structure information with the original sample based on the OSS layer sample space, the MSPE layer sample space and the first mapping function.

[0016] Optionally, the second and third level layer consistency maintaining module optimizes the objective function composed of local distribution difference loss, global distribution difference loss and low-rank constraint regularization to obtain a second mapping function, and then generates a clustering envelope sample set that maintains local and global consistency with the sample data structure information of the MSPE layer sample space, the MDC layer sample space and the second mapping function.

[0017] Optionally, an embedding unit is provided between two adjacent hidden layers in the embedded stacked autoencoder module, and the embedding unit introduces input layer features of the autoencoder and connects with output features of the previous hidden layer.

[0018] Optionally, the three-layer space integration mechanism module trains base classifiers by using the generated three-layer envelope sample sets as training data for the embedded stacked autoencoder in their respective sample spaces, and the three base classifiers are integrated into a strong classifier for final prediction, and finally fused to obtain output samples.

[0019] Compared with the prior art, the present invention provides a three-level joint envelope embedded stack autoencoder, which first uses manifold splicing to mine manifold local related information between similar samples to obtain manifold splicing envelope samples; then uses K-means clustering to mine the correlation information between samples to obtain reconstructed new samples-envelope samples; then it is input in parallel with the original sample into the embedded stack autoencoder to construct a new stack autoencoder. In addition, in order to avoid the clustering algorithm from easily falling into local extreme values ​​when searching for sparse solutions in high-dimensional space, clustering and dimensionality reduction are simultaneously integrated into the loss term for joint optimization. In addition, in measuring the similarity between samples, in order to better capture the nonlinear similar structure between samples, the manifold distance is used to replace the Euclidean distance. In order to improve the dimensional representation ability of the sample, before clustering, the original sample is converted into a manifold sample pair, thereby improving the representation ability of the sample. In order to avoid the qualitative distortion of the sample after clustering, thereby reducing the representation ability of the original sample, a three-level inter-layer consistency maintenance mechanism is designed to obtain envelope samples with strong representation ability and compact sample number.

[0020] The main contributions of this patent are as follows:

[0021] (1) Different from the existing single-stage stack autoencoder paradigm based on original samples, the present invention connects the original samples and two envelope samples in parallel to the stack autoencoder, thereby constructing a three-stage envelope embedded stack autoencoder, providing higher quality deep features for subsequent classifiers;

[0022] (2) The original samples are converted into manifold sample pairs for subsequent clustering, which increases the dimensional space of the samples and thus expands the state space of clustering, avoiding missing the optimal solution.

[0023] (3) Design dimensionality reduction clustering to mine the correlation information between samples, avoid falling into local extreme values, and improve the quality of envelope samples.

[0024] (4) A three-level consistency maintenance mechanism is designed to avoid the qualitative distortion of the samples after clustering, thereby ensuring the representation ability of the new samples after clustering for the original samples. In addition, the three-level consistency maintenance problem is transformed into a two-layer iterative cross-domain adaptation problem to be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the schematic diagram of the traditional stacked autoencoder model;

[0026] Figure 2 It is a schematic diagram of a three-stage joint envelope embedded stack autoencoder model provided in a specific embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of the structure of the manifold sample pair envelope module in a specific embodiment of the present invention;

[0028] Figure 4 It is a schematic diagram of the structure of a clustering module maintaining dimensionality reduction in a specific embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of a first- and second-layer consistency maintenance module according to a specific embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of a module for maintaining consistency between the second and third layers in a specific embodiment of the present invention;

[0031] Figure 7 It is a schematic diagram of the structure of the embedded stack autoencoder in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below through specific embodiments. It is pointed out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above invention content.

[0033] like Figure 2As shown, this embodiment provides a three-level joint envelope embedded stacked autoencoder, referred to as TLJE-ESAE (Three Level Joint Envelope Embedded Stacked Autoencoder) in this example. The encoder adopts a three-level joint envelope mechanism (Three Level Joint Envelope, TLJE) to obtain three layers of sample space for the subsequent embedded stacked autoencoder. When extracting manifold neighbor related information between samples, the present invention considers both local correlation and global correlation: it proposes to use the manifold sample pair envelope module (Manifold Sample-pair Envelope Module, MSPE) to mine the manifold local related information between similar samples, and then uses K-means clustering to mine the global related information between similar samples and construct the next layer of sample space. In order to solve the sparsity problem that may result from direct clustering of high-dimensional sample sets, the present invention also adopts a Maintaining Descending Clustering Module (MDC) with neighborhood structure preservation to project samples into a low-dimensional space during the clustering process. Since the goal of clustering is to minimize the sum of the distances from the original sample to the cluster center, there may be a problem of inconsistent sample distribution before and after clustering. In order to solve these problems, the present invention introduces a three-level interlayer consistency maintenance module (Three Level Interlayer Consistency Maintenance Module, TLICM) between sample layers, and uses the manifold sample pair and the transformed sample as the source domain and target domain respectively. The TLICM method considers both the global and local aspects of sample distribution to better ensure the consistency of distribution. At the same time, TLJE is introduced into the preprocessing part of ESAE, and deep feature extraction can be performed on the data and information layers. Unlike the existing SAE, TLJE-ESAE considers both the local correlation and the global correlation between similar samples: the MSPE module is used to extract local correlation, and the MDC module is used to extract global correlation. This module seamlessly integrates the advantages of clustering, dimensionality reduction and neighborhood structure retention, removes the redundant features of samples in the clustering process, and introduces the TLICM module before and after sample clustering to reduce the difference in sample distribution. Finally, the ESAE model under MSPE, MDC and the original sample space is fused through the three-layer space integration mechanism (Three-layer Space Integration Mechanism, TLSIM).

[0034] pass Figure 2It can be seen that the three-level joint envelope embedded stack autoencoder in this embodiment includes a three-level joint envelope module (TLJE), an embedded stack autoencoder module (ESAE) and a three-layer spatial integration mechanism module (TLSIM), wherein: the three-level joint envelope module (TLJE) is provided with a manifold sample pair envelope module (MSPE), a dimension reduction clustering module (MDC) and a three-level consistency maintenance module (TLICM);

[0035] After the input samples are processed by the TLJE mechanism including MSPE, MDC and TLICM modules, a three-layer sample space is generated, and ESAE modeling is performed in parallel between the three layers of samples. Finally, they are fused through the TLSIM three-layer space integration mechanism to achieve sample envelope transformation before ESAE modeling.

[0036] Since the simple Euclidean distance cannot reflect the true distance between the nearest neighbor samples, the present invention adopts the manifold sample pair envelope module (MSPE) to obtain the inherent structure of manifold-similar adjacent samples in the sample envelope space. Geodesic distance (GD) is a well-known nonlinear measurement method that can reveal the similarity between samples on a nonlinear manifold. Therefore, in the specific implementation, the geodesic distance is introduced as the shortest distance in the sample envelope space. The geodesic distance between the original sample and all other samples is calculated, and the sample with the smallest geodesic distance is considered to be the nearest neighbor sample. Then, the original sample is spliced ​​with its respective multiple manifold nearest neighbor samples to generate a manifold nearest sample pair.

[0037] Given a multidimensional sample vector X∈R n×d , which contains n samples {x1,…,x i ,…,x n}, each sample can be represented as {x i 1 ,x i 2 ,...,x i d}. The process of MSPE is as follows:

[0038] First, calculate the original sample x according to the following formula i With all other samples x j The shortest manifold distance between:

[0039]

[0040] Where S g (x i ,x j ) represents x i and x jThe manifold distance between them, consider the data points as the vertices of the graph g = (Δ, Ι), where Δ is the set of vertices in the graph g, Ι is the set of edges in the graph g, let Represents a connection point on a graph and The path where the edge 1≤k<l-1. Represents the connection data x i and x j The set of all paths. is the line segment between two points on the manifold in space and The length of is the Euclidean distance between two points, and ρ (ρ>1) is the stretching factor.

[0041] Then we get the manifold distance matrix S = [S g (x i ,x j )] n×n , select a sample with the smallest distance value in each row of the matrix S as the manifold nearest neighbor sample x nearest Finally, the original sample x original The nearest neighbor sample x nearest Splice into manifold sample pairs X MSPE =[x original ,x nearest ], the MSPE process is as follows Figure 3 As shown, the pseudo code of MSPE is shown in Table 1:

[0042] Table 1 Pseudo code of MSPE algorithm

[0043]

[0044] For the dimensionality reduction clustering module (MDC), we consider using K-means clustering to mine relevant information between similar samples. However, considering the high redundancy and high sparsity between high-dimensional samples, we design an MDC module in the specific implementation to seamlessly combine the advantages of clustering and dimensionality reduction. Figure 4 As shown in the figure. The samples before clustering are samples of the target domain, and the samples after clustering are samples of the source domain. Clustering is performed in the low-dimensional space after projection rather than the original high-dimensional space, which can well preserve the neighborhood structure properties when constructing the next layer of sample space. In addition, MDC integrates the two independent stages of previous clustering research on high-dimensional data into a unified model, thereby achieving simultaneous optimization of dimensionality reduction and clustering. Therefore, the ability to process high-dimensional data sets when constructing multi-layer samples with clustering has been significantly improved.

[0045] (1) The construction of the MDC objective function is as follows:

[0046]

[0047] Among them, M k represents the kth cluster center, P∈R 2d×q represents the projection matrix, n represents the number of samples, c represents the number of cluster centers, represents the i-th manifold splicing sample, Map samples from one dimension to another, where q<<2d, q represents the dimension after mapping. The projected space has a lower dimension and can remove redundant features in the original space. || || 2,1 represents the L2,1 norm, which is used to constrain the sparsity of the projection matrix P. The parameter σ is a balancing factor that quantifies the contribution through the representation loss, where σ∈{0.1,1,10,100,1000}, represents the weight coefficient of the projection matrix P, where

[0048] The first part of the objective function (2) represents the clustering error, which is composed of M and X MSPE The projection matrix P transforms the original space into a new low-dimensional space. P is constrained by the third part to maintain the neighborhood structure of the data. The L2,1 norm is used as the basic measure of clustering loss and representation loss to enhance the robustness of the model. The parameter ζ is a balancing factor that measures the contribution of representation loss. When the parameter value is large, the third part of the objective function (2) plays a greater role, while when the parameter value is small, the first part of the objective function (2), i.e., the clustering loss constructed in the low-dimensional space, is more significant. The second part of the objective function (2) constrains the projection matrix P used in the first part, and the third part is a regularization term that controls the structural sparsity of the projection matrix P involved in the second part. The third part is a regularization application, where the L2,1 norm and P have the properties of row sparseness (joint sparseness), which enhances the interpretability of the selected features in the projection space.

[0049] In the specific implementation, the Locally Linear Embedding (LLE) technology is used to construct the affinity matrix. This method can effectively retain the local structural characteristics of data samples in high-dimensional space, so as to better mine the relationship and similarity between data.

[0050]

[0051] Where N z (X) represents the zth nearest neighbor sample of sample X, W ij Representation sample and neighbor relationship.

[0052] (2) Optimization of MDC objective function:

[0053] Obviously, there are two variables M and P in the MDC objective function (2) that need to be solved. The cluster center M needs to be initialized first. The present invention selects K-means as the cluster center initialization method. The alternating direction method of the multiplier can effectively solve the problem, that is, when optimizing one of the variables, the remaining variable is fixed. The solution can be solved alternately as follows:

[0054] Step 1: When optimizing M, fix P and transform the objective function (2) into:

[0055]

[0056] Introducing auxiliary variables To represent the projection space With M k The distance between them, so the above formula can be expressed as:

[0057]

[0058] Calculation formula (5) for M k The partial derivative of , we can get:

[0059]

[0060] Let formula (6) be 0, and we can get M k :

[0061]

[0062] Step 2: When optimizing P, fix M and transform the objective function (2) into:

[0063]

[0064] Formula (8) can be considered to consist of three parts, so we can first split it into three parts and analyze the mathematical meaning of each part one by one to solve it:

[0065] The first part of the solution: Here, by introducing auxiliary variables and For square calculation, we can get:

[0066]

[0067] Where Tr represents the trace function.

[0068] The second part of the solution: Here, by introducing the auxiliary variable Q = [q ij ] n×n ,in

[0069]

[0070] q ij It can be regarded as W ij Added a weight, distance The farther away, the W ij The smaller the value, the further the similarity between them is reduced. and If they overlap, then W ij is invalid, so:

[0071]

[0072] Where μ represents the initialization ratio of cluster centers, represents the Hadamard product of W and Q, L is composed of The Laplace matrix obtained can be obtained:

[0073]

[0074] The third part of the solution:

[0075] ||P|| 2,1 =2Tr(P T DP) (12)

[0076] Where D can be expressed as:

[0077]

[0078] Finally, integrating the above three parts gives:

[0079]

[0080] Obviously, B is a real symmetric matrix. Therefore, P can be solved based on the eigenvalue decomposition result of B, and the obtained P must be orthogonal. P∈R 2d×q It is composed of eigenvectors, and its corresponding eigenvalue is the non-zero minimum value of q. q represents the number of variables in the projection space. After obtaining the final optimized variables M and P, the second-level sample space can be obtained according to the following formula, that is, the sample space X generated by the dimension reduction clustering module MDC MDC :

[0081] X MDC =MP (15)

[0082] The pseudo code of the MDC algorithm is shown in Table 2.

[0083] Table 2 Pseudo code of MDC algorithm

[0084]

[0085]

[0086] For the three-level inter-layer consistency maintenance module (TLICM), three layers of sample space are obtained through the MSPE and MDC methods, namely the original space (Original Sample Space, OSS), MSPE and MDC, and the three-level inter-layer consistency mechanism is used to eliminate the distribution differences between the samples of the OSS layer and the MSPE layer, and the MSPE layer and the MDC layer. The main purpose of this patent is to make the sample distribution in the OSS and MSPE, MSPE and MDC sample spaces consistent through the TLICM algorithm, thereby improving the model performance.

[0087] The three-level consistency maintenance module includes a first- and second-level consistency maintenance module and a second- and third-level consistency maintenance module.

[0088] For the consistency maintenance module between the first and second layers: To solve the distribution difference between OSS and MSPE, TLICM is introduced between the two sample spaces. The principle is as follows Figure 5 First, for the original sample X, it is spliced ​​through the MSPE module, so that its dimension changes from n to 2n, which is equivalent to multiplying X by a splicing operator P A , then XP A This is equivalent to transforming the dimension of X, which can make XP A With X MSPE Dimensions are consistent. Make XP A =X OSS , and then transpose the matrix in each sample space (i.e. X OSS ∈R 2d×n , X MSPE ∈R 2d×n ). MSPE can generate the transfer matrix Q A (Q A ∈R n×n ) generates an intermediate sample space (ISS) that is similar to the OSS distribution.

[0089] First we introduce the implicit but common transformation To represent the training sets of MSPE, OSS and ISS, and are defined as and For convenience, is defined as the i-th sample in the ISS, Defined as the jth sample in OSS, considering Therefore, the objective function expression of TLICM is:

[0090]

[0091] Where Ψ and Φ are corresponding mapping functions. In general, TLICM consists of three parts: The first part is the local distribution difference (LDD) loss J LDD (Φ,Q A ), that is, using the locality of MSPE to measure the local area difference with ISS; the second part is the global distribution difference (Global Distribution Difference, GDD) loss J GDD (Ψ,Q A ), where τ∈{0,1,10,100,1000} is a balancing factor that minimizes the global difference in marginal distribution between ISS and MSPE; the third part is the Low Rank Constraint (LRC) regularization, where γ∈{0,1,10,100,1000} is a regularization parameter to ensure that the generated transfer matrix Q A The generalization of ||Q A || * is the matrix Q A The low-rank constraint can effectively show the global structure of samples in MSPE and OSS. The various parts of TLICM are described as follows:

[0092] ① Local distribution difference LDD: LDD uses the locality of MSPE to measure local differences, effectively capturing local patterns and structures in the data, thereby indirectly enhancing the consistency of distribution between ISS and OSS. The LDD in the objective function (16) can be rewritten as:

[0093]

[0094] where the matrix D∈R n×n is a diagonal matrix, the elements D ii =∑ j W ij ,i=1,…,n, Tr(·) is the trace of the matrix. ij ∈R n×n Represents the affinity matrix as follows:

[0095]

[0096] Where N z (X) represents the zth nearest neighbor of sample X. Substitute it and use the projection matrix Ψ after linear transformation to find and The latent subspace of , so the LDD loss formula (17) can be rewritten as:

[0097]

[0098] ② Global distribution difference GDD: GDD is to minimize the global difference of marginal distribution between ISS and OSS. The projection matrix Ψ is used to find and The potential common subspace of . Here we introduce the generation transfer matrix Q A , and substitute it into the objective function (16), then the GDD loss formula after projection can be rewritten as:

[0099]

[0100] where Ψ can be expressed as some linear combinations, namely Φ represents the linear combination coefficient matrix, Then the projected MSPE can be expressed as The projected MSPE can be expressed as After this, MSPE and OSS can be simply expressed as Φ T K MSPE and Φ T K OSS Therefore, the GDD loss formula (20) can be rewritten as:

[0101]

[0102] ③ TLICM joint optimization based on GDD and LDD: The TLICM proposed in this paper aims to ensure the consistency of sample distribution between multi-layer sample spaces. Finally, combining the above parts and introducing the trade-off parameter and γ, the objective function (16) of TLICM can be rewritten as:

[0103]

[0104] Obviously, there are two variables Φ and Q in TLICM A Need to be solved. By introducing the auxiliary variable Θ and further utilizing the augmented Lagrangian function, the objective function (22) can be re-expressed as:

[0105]

[0106] In formula (23), E and ρ1 are the augmented Lagrange multiplier and penalty parameter, respectively, and 1 represents a matrix full of 1s. AThe optimization process of Φ and Θ can adopt the variable alternation strategy, that is, when optimizing one of the variables, the remaining two variables are fixed. Therefore, when solving the objective function (23), the variables Φ, Q A and Θ can be solved alternately as follows:

[0107] Step 1: When optimizing Φ, fix Q A and Θ, the objective function in formula (23) is transformed into:

[0108]

[0109] From this, we can deduce the solution Φ of the j-th iteration sequence j By setting formula (24) to j(:,i) The partial derivative of is zero, and we get Φ j The i-th column vector in :

[0110]

[0111] Step 2: Optimize Q A When Φ and Θ are fixed, the objective function in formula (23) is converted to:

[0112]

[0113] Gradient descent is an optimization method that calculates the gradient of the objective function and updates the parameters along the negative gradient direction, gradually adjusting the parameters to minimize the objective function value, thereby finding the optimal solution. This process is repeated until the convergence condition is met. Therefore, formula (26) can be solved by the gradient descent algorithm for the variable Q A The closed solution of , and after (j+1) iterations, can be converted to:

[0114]

[0115] Step 3: When optimizing Θ, fix Φ and Q A , convert the objective function in formula (3.14) into:

[0116]

[0117] After (j+1) iterations, Θ j+1 Can be optimized to:

[0118]

[0119] The singular value thresholding (SVT) operator is a matrix denoising technique that reduces noise and redundant information by retaining the main singular values ​​of the matrix and setting the remaining singular values ​​to zero, thereby achieving data compression and denoising. Therefore, formula (29) can be solved using the SVT operator to obtain the final optimized variable Θ.

[0120] When the optimal variable Φ is obtained, The clustering envelope sample set can be obtained after maintaining the local and global consistency of the sample data structure information with OSS, where d' represents the sample dimension after TLICM. TLICM enables the algorithm to maintain the local and global consistency of the sample data structure information while constructing a multi-layer sample space, thereby achieving effective sample conversion. The pseudo code of the TLICM algorithm is shown in Table 3.

[0121] Table 3 Pseudo code of TLICM

[0122]

[0123]

[0124] For the consistency maintenance module between the second and third levels, the two-layer sample space MSPE and MDC are obtained by the MDC method. As mentioned above, the distribution difference between the samples before and after clustering is not considered in the clustering process. In order to eliminate this distribution difference, TLICM is introduced between the two sample spaces. The principle is as follows Figure 6 As shown. This method has been described in detail above, so it will not be repeated here. Through TLICM processing, the sample distribution in the two sample spaces of MDC and MSPC is made consistent, thereby improving the model performance. First, the matrix in each sample space is transposed (i.e., X MDC ∈R q×u , X MSPE ∈R 2d×n ). This paper considers that MDC can generate the transfer matrix Q B Generate an intermediate sample space ISS similar to the MSPC distribution and introduce an implicit transformation To represent the training sets of MSPE, MDC and ISS, and are defined as and Introduce the linear combination Ψ, that is Where X CM =[X MDC ,X MSPE ], Φ represents the linear combination coefficient matrix. After TLICM algorithm processing, the optimized variables Φ1, Q B and Θ1, and obtain the envelope sample set in the sample space through these optimized variables

[0125] For the embedded stacked autoencoder (ESAE), unlike the traditional SAE, the key part of the ESAE designed by the present invention introduces the input layer features of the autoencoder (i.e., the embedding unit) between two adjacent hidden layers, such as Figure 7 As shown:

[0126] Assume that the input sample matrix of the ESAE network is X OS , where the output matrix of the hidden layer in the kth encoder is d k represents the number of hidden layer neurons in the kth encoder. Then the embedding unit can be expressed as:

[0127] L(V)=G T (V) (30)

[0128] In formula (30), V represents the feature matrix, where It means connecting the original input feature φ(x) of the encoder with the hidden layer output feature h, and G is the corresponding transformation matrix composed of 0 and 1. After G transformation, the hidden layer output vector of the kth encoder in ESAE can be expressed by the following formula:

[0129]

[0130] Where f represents the activation function, that is, f(x) = 1 / (1+exp(-x)). k1 represents the connection weight matrix between the input layer and the hidden layer in the kth autoencoder, b k1 Represents the bias vector between the input layer and the hidden layer in the kth autoencoder. The purpose of the embedding unit is to introduce certain original information constraints in the layer-by-layer training process of the encoder network and filter out some hidden layer outputs with weak feature representation capabilities. The objective function of the proposed embedded unit is expressed as:

[0131]

[0132] Calculate the covariance matrix of the feature matrix V Sort its diagonal elements in descending order and take the first 2d values ​​to form a vector s∈R 2d The transformation matrix G is obtained by formula (33):

[0133]

[0134] in is the covariance matrix The i-th diagonal element of , s jis the jth element of the vector, i∈[1,n+2d], j∈[1,2d]. Combining formulas (21) to (24), we can obtain the hidden layer output of the kth encoder. Then, the decoding function of the kth encoder can be rewritten as:

[0135]

[0136] Where L'(V k-1 ) is the data obtained by reconstructing the output of the embedding unit, W k2 and b k2 are the connection weight matrix and bias vector between the hidden layer and the output layer in the kth encoder, respectively.

[0137] The KL divergence is introduced to enable the hidden layer of the encoder to learn sparse representation. The formula of KL divergence is:

[0138]

[0139] where f j (x i ) is the activation value of the i-th input vector of the j-th neuron in the hidden layer, ρ j is the average activation of the jth neuron in the hidden layer, and ρ is the target average activation. After introducing the embedded unit in the autoencoder structure and the sparse criterion in the training process, the optimization objective function of the kth encoder of ESAE is written as:

[0140]

[0141] Among them, λ∈{0.00001,0.0001,0.001} and β∈{1,2,3,4,5,6} are the penalty parameters of the ESAE regularization term and the sparsity criterion term, respectively.

[0142] After two stages of pre-training and fine-tuning of the autoencoder network, the matrix for the i-th input vector can be expressed as X iOS =[φ(x i1 ),φ(x i2 ),…,φ(x i2d )], each hidden layer in the network outputs a new feature vector, representing information at different feature levels.

[0143] Let q represent the number of neurons in the last hidden layer, and take the output of the last hidden layer as the deep features learned by the autoencoder network, which can be expressed as:

[0144] X DF =[x1',x2',…,x d ']∈R n×q (37)

[0145] For the three-layer space integration mechanism module (TLSIM), the three-layer envelope sample space: OSS, MSPE and MDC are constructed by the method introduced by TLICM, and the generated three-layer envelope sample sets are used as training data of ESAE in their respective sample spaces to train the base classifiers. Multiple base classifiers are integrated into the strong classifier for final prediction. The present invention improves the complementarity of the three-layer sample space by using the three-layer space integration mechanism TLSIM to fuse the ESAE models under each sample space. TLSIM fuses the results of the base classifiers under each layer of sample space to obtain the final prediction result. The TLSIM will involve two commonly used integration methods: weighted fusion (Weighted Fusion, WF) and majority voting (Majorityvoting, MV). The traditional classification algorithm based on SAE directly inputs the deep features learned by SAE into the classifier to obtain the final result. In contrast, the patented algorithm outputs three layers of deep features by constructing three layers of envelope samples. In order to improve the complementarity between the three layers of features, the three-layer space integration mechanism fuses the classification results of the three layers of ESAE and realizes the collaborative conversion between samples and features.

[0146] Assuming that the prediction result of the model in OSS is y0, the prediction result of the model in MSPE is y1, and the prediction result of the model in MDC is y2, the implementation process of TLSIM is as follows:

[0147] (1) Weighted fusion: The key to weighted fusion is to determine the weight w of each base classifier model in each sample space, which requires the use of a validation set. Assuming that the prediction result of each encoder on the validation set is r = (r0, r1, r2), the optimal weight vector is w = (w0, w1, w2), which can be obtained by the following formula:

[0148]

[0149] where Λ v Represented as the number of validation sets, function Assume y = (y0, y1, y2). The prediction result Y of the final test sample can be expressed as:

[0150] Y=round(w T y) (39)

[0151] (2) Majority voting: Assume that the data set has C categories. For each predicted sample, a statistical vector count is given, and the initial value is:

[0152]

[0153] The prediction result of the classifier model in each sample space is y i,So:

[0154]

[0155] The final classification label Y is the subscript j corresponding to the largest component in count, that is, Y = j, where n j =max(count). The pseudo code of DLSIM algorithm is shown in Table 4:

[0156] Table 4 TLSIM algorithm pseudo code

[0157]

[0158] The three-layer sample space, namely OSS, MSPE and MDC, is constructed by the method introduced by TLJE, and three training sample sets are obtained. Each training set is used as the training data of ESAE, and each layer of training data is used to train the base classifier. Multiple base classifiers are integrated into the strong classifier for final prediction. The three-layer space integration mechanism TLSIM is used to fuse the autoencoder models under each sample space to improve the complementarity of the multi-layer sample space. The construction of the training set and test set of each layer of sample space of this patent algorithm and the specific training and testing methods are described as follows:

[0159] (1) Construction of training and test sets for each layer of sample space: First, by training the MSPE module, we obtain the manifold closest sample pair X from the original training data X MSPE As mentioned above, the MDC and TLICM modules construct two layers of envelope samples and obtain the optimal cluster center M, projection matrix P and linear combination coefficient matrix Φ. Finally, the three layers of envelope samples are input into each ESAE layer for training. After training the model, the performance of TLJE-ESAE is verified by the test data. First, the test data X Test Input the trained MSPE module and get At the same time X Test is also the first layer sample. Input into the trained MDC module and get in Again, X Test and Input into the trained TLICM module to obtain the second layer samples and Input the trained TLICM module to get the third layer samples in Finally, three envelope samples (X Test , and ), extract the deep hybrid features F1, F2 and F3. The results of the three-layer envelope samples are fused to get the final result.

[0160] (2) Training and testing of each layer of sample space: TLJE-ESAE outputs three layers of features (samples), and these three layers of samples are trained with three base classifiers respectively. The proposed TLSIM fuses the three layers of results and outputs the final result. The traditional classification algorithm directly inputs the deep features learned by SAE into the classifier to obtain the final classification result. In contrast, TLJE-ESAE outputs three layers of deep features F1, F2 and F3, and finally uses the TLSIM fusion mechanism to fuse the classification results of the three layers of ESAE, realizing the collaborative conversion of samples and features and extracting the correlation information between the sample data layer and the information layer. The training process of TLJE-ESAE is divided into two stages: ① Forward transmission: construct the first layer of samples X based on the training samples X; construct X through MSPE MSPE , and then construct the second layer of samples through the TLICM module Cluster sample X MDC Constructed by MDC modules. MSPE and X MDC As input, the third layer of samples is constructed by the TLICM module Input X to ESAE1 and the output is the training error 1. Input to ESAE2, the output is training error 2. Input to ESAE3, output is training error 3;② Back propagation: Based on training error 1, optimize the weight of ESAE1 through gradient descent algorithm. Based on training error 2, optimize the weight of ESAE2 using gradient descent algorithm. Based on training error 3, optimize the weight of ESAE3 using gradient descent algorithm. TLJE-ESAE testing process: For test sample X Test , input TLJE-ESAE to get three test samples X Test , and X Test , and The trained classifiers 1, 2, and 3 are input respectively to obtain three predicted class labels. The final result is obtained through a three-layer fusion mechanism.

[0161] The three-level joint envelope embedded stack autoencoder for structured data classification proposed in the present invention mainly includes MSPE, MDC, TLICM, ESAE and TLSIM. Therefore, in order to verify the effectiveness of the present invention, the algorithm TLJE-ESAE proposed in this patent is compared with the existing representative feature learning algorithm and the representative SAE model.

[0162] The experimental platform of the present invention is a 64-bit Windows 10 operating system, the hardware part is a computer CPU @ 2.8GHZ processor with Inter (R) CORE (TM) CPU i5-8400 and 8GB RAM, and the development tool Matlab R2018 (b), and the toolbox used in the experiment is libsvm 3.20 version. The performance of the patent algorithm is verified on 15 representative public data sets. These data sets have large samples and small samples, the number of samples ranges from 90 to 20000, the number of categories ranges from 2 to 26, and the number of features ranges from 8 to 561. The main information of the data set used in the present invention is shown in Table 5.

[0163] In the following experiments, since the sample size of some data sets is not large, the number of layers of all SAEs is set to 3. In the experiment, the number of hidden layer neurons of each autoencoder is determined according to the sample size and feature dimension of the data set, and the optimal structure is determined by the grid search method. The experiment adopts the hold-out cross-validation method to divide each data set into three equal parts, of which 1 / 3 samples are used as test sets, 1 / 3 samples are used as validation sets, and the remaining samples are used as training sets. Each data set is repeated five times to take the average value and standard deviation as the classification result of the experiment of this patent. This method can help this article evaluate the performance of the algorithm more accurately. By repeating the experiment many times and calculating the average value and standard deviation, more stable and reliable experimental results can be obtained, eliminating the influence of randomness on the experiment of this patent. In the experiment, classification accuracy ACC, MACRO-F1 (F1), AUC and average precision (AP) are used to measure the model performance of different models. The higher the value of any evaluation index, the better the performance of the algorithm. The best results in the table experiment are shown in bold. In addition, the classifiers used in the experiments of this patent without explicit instructions are all support vector machines (SVM).

[0164] Table 5 Basic information of the dataset

[0165]

[0166]

[0167] Comparison with the most advanced stacked autoencoder model: In order to verify the advantages of the algorithm TLJE-ESAE proposed in this patent over the existing SAE, some representative SAE algorithms are considered for comparison, including WGLAE, DSAE and SGAE. The experimental results are shown in Tables 6-8. The experimental results show in detail the performance of all compared algorithms on the three evaluation indicators ACC, F1 and AUC.

[0168] Observing the comparison of the results of different stacked autoencoder algorithms on ACC in Table 6, it is found that TLJE-ESAE outperforms other comparison algorithms in most cases, and the results of 10 of the 15 data sets are the best, showing a significant advantage. This shows that the TLJE-ESAE algorithm can achieve competitive performance levels in most cases and performs well in data processing and classification tasks. Further observation shows that TLJE-ESAE obtains lower variance on most data sets after multiple runs. This means that compared with other methods, TLJE-ESAE has stronger robustness and stability. Through multiple experimental runs, TLJE-ESAE can maintain a relatively consistent performance and reduce result fluctuations, thereby enhancing the algorithm's adaptability to different data sets and experimental conditions. This highly stable performance further proves the reliability and effectiveness of TLJE-ESAE in processing data and improving model performance, and provides strong support for its feasibility in practical applications.

[0169] In addition, the same results can be found by observing the other three evaluation indicators. Based on the above analysis, the outstanding performance of the TLJE-ESAE algorithm in multiple evaluation indicators and its stability under multiple runs show its potential and advantages as a cutting-edge feature learning method. This shows that the algorithm is of great significance in solving complex problems, improving prediction performance and promoting research in related fields. This shows that the patented algorithm can better capture the correlation between similar samples during the model training process by considering the structural information between samples, thereby improving the generalization ability and classification performance of the model. This method makes the model more robust when making predictions, effectively avoids overfitting problems, and can more accurately distinguish samples of different categories, thereby improving the performance of ACC, F1 and AU indicators.

[0170] Secondly, the use of dimensionality reduction clustering technology can reduce the dimension of the feature space while retaining data features, improving the computational efficiency of the model and reducing the computational complexity. By maintaining dimensionality reduction clustering, we can better discover the potential connections between samples, reduce noise interference, and thus improve the performance of the classifier. This processing method helps to improve the classification accuracy, F1 score and AUC of the model, thereby improving the overall classification effect. Finally, building multi-layer samples can help the model better understand the hierarchical structure and feature information of the data, thereby improving the model's abstract representation and generalization capabilities of the data. Fusion of sample representations at different levels through the TLSIM mechanism can capture the different features and attributes of the original samples between the information layer and the data layer, which helps the model to understand the data set more comprehensively and perform more accurate classification. Therefore, building multi-layer samples can enhance the adaptability and expression ability of the model when processing complex data sets, and improve the performance of ACC, F1 and AUC. Finally, by combining generated samples and SAE modeling, the model can learn more representative feature representations.

[0171] Table 6 Comparison of ACC results of different SAE algorithms (mean ± variance)

[0172]

[0173] Table 7 Comparison of F1 results of different SAE algorithms (mean ± variance)

[0174]

[0175] Table 8 Comparison of AUC results of different SAE algorithms (mean ± variance)

[0176]

[0177] In summary, it can be seen that the existing SAE mainly considers minimizing the error between each input sample and the output sample by using the reconstructed deep features, but ignores the associated structural information between samples. For low-quality sample sets, the performance is often unsatisfactory.

[0178] Aiming at the problem that the existing low-quality samples restrict the improvement of SAE performance, this patent proposes a three-level joint envelope embedded stack autoencoder. This scheme considers the local correlation and global correlation of the manifold between similar samples at the same time, seamlessly integrates the advantages of clustering, dimensionality reduction and neighborhood structure retention, and removes the redundant features of samples in the clustering process. First, a three-level envelope mechanism TLJE is designed to obtain a three-layer envelope sample space for subsequent SAE. The mechanism consists of three parts: first, the MSPE module is used to extract the local correlation of the manifold; second, the MDC module is used to extract the global correlation, seamlessly integrates the advantages of clustering, dimensionality reduction and neighborhood structure retention, and removes the redundant features of samples in the clustering process; third, the three-level consistency maintenance module TLICM is used to reduce the sample distribution differences in different levels of sample space. Finally, TLJE is introduced into the preprocessing part of ESAE, and a new SAE is constructed to extract deep features of the data and information layers. Different from the existing SAE, TLJE-ESAE extracts the correlation between similar samples and converts it into new samples, thereby extracting deep features of the relationship between similar samples. The algorithm was applied to the classification task of 15 data sets. The experimental results show that the TLJE-ESAE algorithm proposed in this paper has good feature learning ability. Compared with other classic feature learning methods and other SAE algorithms, the TLJE-ESAE algorithm can obtain higher results and more stable model performance on different model evaluation indicators.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A three-stage joint envelope embedded stacked autoencoder, characterized in that: It includes a three-level joint envelope module, an embedded stack autoencoder module and a three-layer spatial integration mechanism module, where: The three-level joint envelope module is provided with a manifold sample pair envelope module, a dimension reduction clustering module and a three-level consistency maintenance module; the manifold sample pair envelope module is used to obtain the inherent structure of manifold-similar adjacent samples in the sample envelope space; the dimension reduction clustering module is used to project samples into a low-dimensional space during the clustering process; the three-level consistency maintenance module is used to maintain the consistency of sample distribution; After the input sample enters the three-level joint envelope module, it is jointly processed by the manifold sample pair envelope module, the dimension reduction clustering module and the three-level consistency maintenance module to generate a three-layer sample space; the embedded stack autoencoder module performs embedded stack autoencoder modeling in parallel on the three-layer sample space, and finally the three-layer space integration mechanism module fuses them and obtains the output sample.

2. The three-stage joint envelope embedded stacked autoencoder according to claim 1, characterized in that: The three-layer sample space includes an OSS layer sample space, an MSPE layer sample space and an MDC layer sample space, and the three-level consistency maintenance module includes a first-level and second-level layer consistency maintenance module and a second-level and third-level layer consistency maintenance module, wherein: The OSS layer sample space is the original sample space; the MSPE layer sample space is the sample space in which the input sample is processed by the manifold sample pair envelope module and the first and second layer consistency maintaining module is used to make it consistent with the distribution of the OSS layer sample space; the MDC layer sample space is the sample space in which the input sample is first processed by the manifold sample pair envelope module, then processed by the dimension reduction clustering module, and finally made consistent with the distribution of the MSPE layer sample space by the second and third layer consistency maintaining module.

3. The three-stage joint envelope embedded stacked autoencoder according to claim 1 or 2, characterized in that: The manifold sample pair envelope module calculates the geodesic distance between the original sample and all other samples. The sample with the smallest geodesic distance is considered as the nearest neighbor sample, and then concatenates the original sample with its respective multiple manifold nearest neighbor samples to generate a manifold nearest sample pair.

4. The three-stage joint envelope embedded stacked autoencoder according to claim 3, characterized in that: The dimension reduction-preserving clustering module finally generates a clustered sample space by finding the best cluster center and the best projection matrix, wherein the best cluster center and the best projection matrix are found by optimizing with the goal of minimizing clustering error.

5. The three-stage joint envelope embedded stacked autoencoder according to claim 4, characterized in that: When optimizing with the minimum clustering error as the goal, the objective function also includes a projection matrix constraint term and a projection matrix regularization application term. The projection matrix constraint term constructs an affinity matrix through a local linear embedding method to effectively retain the local structural characteristics of the data sample in the high-dimensional space, thereby better mining the relationship and similarity between the data; The projection matrix regularization application enhances the interpretability of the selected features in the projected space by using the L2,1 norm.

6. The three-stage joint envelope embedded stacked autoencoder according to claim 4 or 5, characterized in that: In the dimension reduction clustering module, K-means is first used as a cluster center initialization method to initialize the cluster center.

7. The three-stage joint envelope embedded stacked autoencoder according to claim 2, characterized in that: The first and second layer consistency maintenance module optimizes the objective function composed of local distribution difference loss, global distribution difference loss and low-rank constraint regularization to obtain the first mapping function, and then generates a clustering envelope sample set that maintains local and global consistency of sample data structure information with the original sample based on the OSS layer sample space, the MSPE layer sample space and the first mapping function.

8. The three-stage joint envelope embedded stacked autoencoder according to claim 2, characterized in that: The second and third level layer consistency maintenance module optimizes the objective function composed of local distribution difference loss, global distribution difference loss and low-rank constraint regularization to obtain the second mapping function, and then generates a clustering envelope sample set that maintains local and global consistency with the sample data structure information of the MSPE layer sample space according to the MSPE layer sample space, the MDC layer sample space and the second mapping function.

9. The three-stage joint envelope embedded stacked autoencoder according to claim 7 or 8, characterized in that: In the embedded stacked autoencoder module, an embedding unit is arranged between two adjacent hidden layers, and the embedding unit introduces the input layer features of the autoencoder and connects with the output features of the previous hidden layer.

10. The three-stage joint envelope embedded stacked autoencoder according to claim 9, characterized in that: The three-layer space integration mechanism module uses the generated three-layer envelope sample sets as training data for the embedded stacked autoencoder in their respective sample spaces to train base classifiers. The three base classifiers are integrated into a strong classifier for final prediction and finally fused to obtain output samples.

Images