Data compression method based on ratio minimization and sparse representation learning model

Abstract

The invention discloses a data compression method based on ratio minimization and sparse representation learning, and the method comprises the steps: obtaining a training data set, carrying out the normalization processing of a data sample in the training data set, and carrying out the dimension reduction processing of the data sample through a principal component analysis method; applying the selected training set to a robust sparse representation learning model based on ratio minimization, and updating a projection matrix and a class cluster center vector until an objective function is converged by continuously iteratively optimizing the learning model, so as to obtain an optimal data projection matrix; and for the to-be-compressed data, the to-be-compressed data is linearly mapped to the low-dimensional space through the learned data projection matrix, so that data compression is realized. According to the method, a sparse regular term based on l1, 2 norms is designed as a measurement criterion, so that the characterization capability of the proposed model is effectively improved; when high-dimensional data is compressed and represented, convergence can be carried out more quickly, and relatively stable processing performance can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a linear compression characterization method for high-dimensional data containing outliers and noise. Background Art

[0002] As an important data characterization technology, Fisher linear discriminant analysis (LDA) has been successfully applied in many scientific fields in recent years, including data compression transmission and information denoising. Fisher linear discriminant analysis is a subspace analysis method for learning the low-dimensional structure of high-dimensional data. It mainly learns a set of vectors that maximize the Fisher discriminant criterion through training data, and then uses these vectors to reduce the dimensionality of the original data.

[0003] The traditional Fisher linear discriminant analysis method adopts an objective function based on trace ratio, which is converted into an eigenvalue optimization problem to obtain a closed-form solution to the original problem model. However, the solution to this optimization problem requires an inverse operation on the intra-class scatter matrix. Therefore, when the intra-class scatter matrix is ​​singular, the Fisher linear discriminant analysis model is ill-conditioned and cannot be solved. However, in high-dimensional feature spaces or feature spaces with highly correlated features, it is common to have an insufficient rank intra-class scatter matrix, such as image / video classification, gene expression, etc. In this case, the traditional solution is usually to use principal component analysis (PCA) as a preprocessing step to remove the relevant feature null space of the overall scatter matrix before performing dimensionality reduction representation. In addition, to solve the null space problem, many variants of Fisher linear discriminant analysis have been proposed. Recently, the Fisher linear discriminant analysis model based on trace ratio has been further studied [1,2] and has shown excellent representation performance.

[0004] However, it is well known that both the traditional principal component analysis and Fisher linear discriminant analysis methods use the least squares estimation objective function, that is, the l2 norm is used as the measurement criterion. However, this norm can easily lead to the problem of computational bias. This is because the use of the square norm measurement makes the error caused by outliers or noise play a dominant role in the overall error loss, making the model generally not robust to noise and outliers. From a statistical point of view [3], the robustness of a method is defined as the characteristic of being insensitive to outliers. In the literature [4,5], in order to improve the robustness of the projection subspace, a variety of variant methods based on the Fisher linear discriminant analysis model are proposed, including methods based on data resampling and subspace search. In addition, in the field of data representation learning, in order to fundamentally improve the robustness of the method, it is easy to think of changing the l2 norm in the loss function to a more appropriate sparsity-induced norm. Based on this idea, some methods improve the robustness of the Fisher linear discriminant analysis method by using the l1 norm, such as the literature [6,7]. But it is precisely because of the introduction of the l1 norm that the model becomes more difficult to optimize. This is because all Fisher linear discriminant analysis methods, including the ratio method and the difference method, need to minimize the intra-class divergence and maximize the inter-class divergence at the same time. Existing sparse learning optimization algorithms such as the gradient projection method, the homotopy algorithm, the iterative shrinkage threshold method, and the augmented Lagrangian method cannot solve the objective function based on the l1 norm. This makes it difficult for the model to improve its robustness to outliers and noise in the data from itself.

[0005] [1] F.Nie, Z.Wang, R.Wang, and X.Li.Submanifold-preserving discriminantanalysis with an auto-optimized graph. IEEE transactions on cybernetics, 2019.

[0006] [2] Y. Jia, F. Nie, and C. Zhang. Trace ratio problem revisited. IEEE Transactions on Neural Networks, vol. 20, no. 4, pp. 729–735, 2009.

[0007] [3]PJHuber, Robust Statistics. Wiley, 1981.

[0008] [4]S.Fidler and A.Leonardis.Robust lda classification bysubsampling.Conference on Computer Vision and Pattern Recognition Workshop,p.97,2003

[0009] [5]H.Zhao,Z.Wang,and F.Nie.A new formulation of linear discriminantanalysis for robust dimensionality reduction.IEEE Transactions on Knowledgeand Data Engineering,vol.31,no.4,pp.629–640,2018.

[0010] [6]C.-N.Li,Y.-H.Shao,and N.-Y.Deng.Robust l1-norm two-dimensionallinear discriminant analysis.Neural Networks,vol.65,pp.92–104,2015.

[0011] [7]Q.Ye,J.Yang,F.Liu,C.Zhao,N.Ye,and T.Yin.L1-norm distance lineardiscriminant analysis based on an effective iterative algorithm.IEEETransactions on Circuits and Systems for Video Technology,vol.28,no.1,pp.114–129,2016.

[0012] [8]C.Ding,D.Zhou,X.He,and H.Zha.R1-pca:Rotational invariant l1-normprincipal component analysis for robust subspace factorization.Int’lConf.Machine Learning,2006.

[0013] [9] F.Nie, J.Yuan, and H.Huang. Optimal mean robust principal component analysis. In Proceedings of the 31st International Conference on Machine Learning (ICML), 2014, pp.1062–1070.

[0014]

[10] F. Nie, H. Huang, X. Cai, and C. Ding. Efficient and robust features selection via joint-norms minimization. In NIPS, 2010. Summary of the invention

[0015] The purpose of the present invention is to provide a data compression method based on ratio minimization and sparse representation learning model, so as to essentially solve the problem that the traditional Fisher linear discriminant analysis method is sensitive to outliers.

[0016] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0017] A data compression method based on ratio minimization and sparse representation learning, comprising:

[0018] Obtain a training data set, normalize the data samples in the training data set, and then perform dimensionality reduction on the data samples using the principal component analysis method;

[0019] The selected training set is applied to the robust sparse representation learning model based on ratio minimization. By continuously iteratively optimizing the learning model, the projection matrix and cluster center vector are updated until the objective function converges, thereby obtaining the optimal data projection matrix.

[0020] For the data to be compressed, the data to be compressed is linearly mapped to a low-dimensional space through the learned data projection matrix, thereby achieving data compression.

[0021] Furthermore, the training set is processed as follows:

[0022] Given a high-dimensional training data set X = [x1,...,x n ]∈R d×n And the corresponding cluster label matrix Y∈{0,1} n×c , and define the i-th class sample set according to the cluster matrix Y where x n represents the nth data sample in the data set, xn i is the nth data sample of the i-th sample set in the data set, n i represents the total number of data samples of the i-th category, d is the data dimension, R represents the real number set, c is the number of data categories, 1≤i≤c; firstly, the data samples in the original data set are normalized, and the high-dimensional data are preprocessed by the principal component analysis method to eliminate the feature null space in the original data.

[0023] Furthermore, the robust sparse representation learning model based on ratio minimization is expressed as:

[0024]

[0025] Where W∈R d×m is the projection matrix to be learned, the superscript T represents the transpose of the matrix, X represents the training data set, and X i represents the i-th sample set, d and m represent the data dimension and projection dimension respectively, c is the number of data categories, and R is a real number set; n i dimensional column vector whose elements are all 1, n i Represents the total number of data samples of the i-th category; Indicates i The average value of the class samples, The jth data sample in the i-th class sample set, ||·|| 1，2 Indicates l 1,2 norm, and I represents the identity matrix.

[0026] Furthermore, the optimal data projection matrix can be obtained by continuously iterating and optimizing the learning model, updating the projection matrix and the cluster center vector until the objective function converges, including:

[0027] Step 1: First, use the robust sparse representation learning model based on ratio minimization to construct the following optimization problem for solution:

[0028]

[0029] Among them, Tr represents the trace of the matrix; λ represents the penalty parameter, which is calculated and updated by the following formula:

[0030]

[0031] D i (1≤i≤c) is a diagonal matrix, where the j-th diagonal element is:

[0032]

[0033] ||·||2 represents the l2 norm;

[0034] D is also a diagonal matrix, and its i-th diagonal element is:

[0035]

[0036] Among them, x i Represents the i-th data sample in the training data set;

[0037] Through further deduction, formula (2) can be transformed into the following problem:

[0038]

[0039] For the above problem, it is about variables W and μ i They are all convex optimization problems, so they have closed-form solutions;

[0040] Step 2, calculate the parameter λ according to formula (3);

[0041] Step 3: Update and calculate the diagonal matrix D according to formula (4) and formula (5) respectively i (i=1,2,...,c) and diagonal matrix D;

[0042] Step 4: For the i-th data sample, by using equation (6) about μ i Taking the derivative and setting it to 0, we can get μ i The optimal solution is as follows:

[0043]

[0044] Step 5: Substitute the optimal solution μ obtained by equation (7) into i Substituting into equation (6), we can obtain the following optimization problem for the projection matrix W:

[0045]

[0046] The matrix A is expressed as follows:

[0047]

[0048] Here, the optimal solution W of the problem in equation (8) is a matrix composed of the eigenvectors corresponding to the first m smallest eigenvalues ​​of the matrix A;

[0049] Step 6: Repeat steps 2 to 5 in the training data set to learn the optimal projection matrix W.

[0050] Furthermore, for the data to be compressed X, the optimal projection matrix W is calculated by using the learned T X, to achieve data compression.

[0051] A terminal device comprises a processor, a memory and a computer program stored in the memory; when the processor is executed by a computer, the data compression method based on ratio minimization and sparse representation learning is implemented.

[0052] A computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the data compression method based on ratio minimization and sparse representation learning is implemented.

[0053] Compared with the prior art, the present invention has the following technical features:

[0054] The method of the present invention establishes a robust sparse representation learning model based on ratio minimization to deal with the data compression problem in high-dimensional space. It essentially solves the problem that the traditional Fisher linear discriminant analysis method is sensitive to outliers and noise by introducing the sparse induced norm. For a given training data set, the Fisher linear discriminant analysis model based on ratio minimization is first introduced to learn the optimal projection matrix under the constraint of maximizing the distance between samples of different classes while minimizing the distance between samples of different classes. Furthermore, in order to improve the processing performance of the model in the environment of noise and abnormal samples, the present invention designs a method based on l 1,2 The sparse regularization term of the norm is used as the measurement criterion, which effectively improves the representation ability of the proposed model; in addition, the present invention designs an optimization algorithm based on the reweighting method to solve the proposed model. When compressing and representing high-dimensional data, the proposed algorithm can converge more quickly and achieve relatively stable processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of data projection representation of the robust sparse representation learning model (RLDA) based on ratio minimization proposed in the present invention, including a comparison of results with the traditional Fisher linear discriminant analysis method (LDA);

[0056] Figure 2 The figure is a flow chart of a compression method in one embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention provides a new robust sparse representation learning model based on ratio minimization, which can utilize l 1,2 The norm essentially solves the problem that the traditional Fisher linear discriminant analysis method is sensitive to outliers. The proposed model can use the l1 norm between data points and the l2 norm on data features as metric calculations; through the l1 norm, the proposed model will greatly reduce the impact of outliers in the data, thereby effectively improving the robustness of the model. However, the existing optimization algorithms cannot be used to effectively solve the l2 norm proposed in this invention. 1,2norm objective function; for this reason, in the present invention, an efficient optimization algorithm based on the re-weighted method is proposed to solve the problem.

[0058] A data compression method based on ratio minimization and sparse representation learning, the main principles of which are as follows:

[0059] First, the given data samples are normalized, and then the high-dimensional data are preprocessed through the principal component analysis method (PCA) to eliminate the feature null space in the original data; since the Fisher linear discriminant analysis algorithm is a supervised learning method, we need to select some samples from the original data set as the training set to train the proposed model, and the remaining part as the test set.

[0060] The selected training set is applied to the robust sparse representation learning model based on ratio minimization proposed in the present invention. By continuously iteratively optimizing the objective function, the projection matrix and the cluster center vector are updated until the objective function converges, thereby obtaining the optimal data projection matrix. The test set data is linearly mapped to the corresponding low-dimensional space through the learned data projection matrix, thereby realizing the compressed representation process of the test data set.

[0061] Due to the use of l 1,2 The proposed model can effectively improve the robustness to high-dimensional data noise and outliers based on the linear dimensionality reduction representation framework of the norm. Compared with other sparse norms such as the l1 norm, the proposed model is easier to optimize, so when processing real data, the proposed model can converge faster and obtain better results.

[0062] 1. Robust sparse representation learning model based on ratio minimization

[0063] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0064] The present invention provides a robust sparse representation learning model based on ratio minimization to achieve effective compression of high-dimensional data. The basic implementation process is as follows:

[0065] Given a training data set X = [x1,...,x n ]∈R d×n ,definition As the set of data belonging to the i-th cluster, where x n represents the nth data sample in the data set, x n iis the nth data sample of the i-th sample set in the data set, d is the data dimension, R represents the real number set, c is the number of data categories, 1≤i≤c. Here, the data samples in the original data set are first normalized, and the high-dimensional data are preprocessed by the principal component analysis method to eliminate the feature null space in the original data. Finally, 40% of the data samples are randomly selected from the preprocessed data set as the training set to learn the optimal projection matrix in the model; the remaining samples are used as test data sets to verify the compression representation performance of the algorithm after training.

[0066] By minimizing the distance between data points within a class and maximizing the distance between data points between classes, the traditional Fisher linear discriminant analysis method can learn a projection matrix W∈R from the training set. d×m , the formula is as follows:

[0067]

[0068] in, represents the average value of the i-th category data sample in the data set, is the average value of all data samples. Combining the above definition and the formula in the optimization problem (1), the following equation holds:

[0069]

[0070] Based on the above equation, the optimization problem (1) is equivalent to the following problem:

[0071]

[0072] From problem (2), we can see that by maximizing the distance between samples in different classes and minimizing the distance between samples in different classes, we can learn the optimal projection matrix W. Without loss of generality, we can assume that all sample data have been centered, that is, μ = 0. Therefore, problem (2) is equivalent to optimizing the following problem:

[0073]

[0074] Note that the constraint W in problem (3) T W=I, so the following equation holds:

[0075]

[0076] Therefore, combined with the above equation, problem (3) can be rewritten as follows:

[0077]

[0078] Note that, given a training dataset, is a constant, and It can be regarded as the reconstruction error. Therefore, it can be seen that problem (4) is to minimize the projection distance of samples within the class in the numerator and minimize the reconstruction error of samples in the denominator. Therefore, the present invention uses l 1,2 The square of the l2 norm is used to improve the robustness of the model to outliers and noise. Here, l 1,2 The purpose of using the L1 norm instead of the L1 norm is to avoid the problem of feature loss caused by sparse constraints on sample features, which can be verified in the robust principal component analysis method [8]. Here, L 1,2 The norm is to improve the robustness of the algorithm by using the l1 norm in the process of summing between data samples. The influence of outliers and noise is actually reduced by the l1 norm, but it does not involve the characteristics of the data. Therefore, the robust sparse representation learning model proposed in the present invention is as follows:

[0079]

[0080] Notice that It is based on the optimal mean under the squared l2 norm, not under l 1,2 Therefore, in this model, it is also necessary to calculate the center vector of each cluster. Perform optimization updates, so the final optimization model can be written as:

[0081]

[0082] It can be written in the following matrix form:

[0083]

[0084] Among them, ||·|| 1，2 Indicates l 1,2 Norm, first take the l1 norm for each column vector of the matrix X, and then take the l2 norm for the row vector; W∈R d×m is the projection matrix to be learned, d and m represent the data dimension and projection dimension respectively; n i dimensional column vector, whose elements are all 1, where n i Identifies the total number of data samples of the i-th category; in addition, represents the average value of the i-th data sample in the data. From formula (7), we can see that this scheme introduces l 1,2 The norm is used as the metric, but the norm is a non-smooth function, and the objective function in formula (7) is also non-convex, which makes the proposed formula (7) difficult to optimize and solve. To this end, the present invention proposes an efficient optimization algorithm based on reweighting to solve formula (7).

[0085] 2. Design of optimization algorithm

[0086] In order to solve the robust sparse representation learning model as shown in formula (7), we first need to transform the problem into the following problem for optimization:

[0087]

[0088] Here, the parameter λ update calculation formula is as follows:

[0089]

[0090] According to the formula derivation, problem (8) can be further transformed into the following optimization problem:

[0091]

[0092] Combining the reweighted optimization algorithm proposed in the literature [9,10], we can alternately solve the following problems to obtain an optimal solution to problem (9):

[0093]

[0094] Where D i (1≤i≤c) is a diagonal matrix, where the j-th diagonal element is:

[0095]

[0096] D is also a diagonal matrix, and its i-th diagonal element is:

[0097]

[0098] By deduction, problem (10) can be transformed into the following problem:

[0099]

[0100] Problem (11) has a closed-form solution. For the i-th sample, by solving problem (11) about μ i Taking the derivative and setting it to 0, we can get μ i The optimal solution is as follows:

[0101]

[0102] The optimal solution μ obtained by formula (12) i Substituting into the optimization problem (11), we can get the following optimization problem about the projection matrix W:

[0103]

[0104] The matrix A is expressed as follows:

[0105]

[0106] The optimal solution W of problem (13) is a matrix composed of the eigenvectors corresponding to the first m smallest eigenvalues ​​of matrix A, where m is the dimension of the projection space.

[0107] By continuously iterating and optimizing the original problem (7) through the above steps, the optimal mapping matrix W can be finally obtained. According to the learned projection matrix W, it can be applied to the original high-dimensional data for linear dimensionality reduction representation, and the data compression process can be realized by projecting the high-dimensional data into the low-dimensional space.

[0108] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data compression method based on ratio minimization and sparse representation learning, characterized in that: include: Obtain a training data set, normalize the data samples in the training data set, and then perform dimensionality reduction on the data samples using the principal component analysis method; The selected training set is applied to the robust sparse representation learning model based on ratio minimization. By continuously iteratively optimizing the learning model, the projection matrix and cluster center vector are updated until the objective function converges, thereby obtaining the optimal data projection matrix. For the data to be compressed, the data to be compressed is linearly mapped to a low-dimensional space through the learned data projection matrix, thereby achieving data compression.

2. The data compression method based on ratio minimization and sparse representation learning according to claim 1, characterized in that: The processing process of the training data set is as follows: Given a high-dimensional training data set X = [x1,...,x n ]∈R d×n And the corresponding cluster label matrix Y∈{0,1} n×c , and define the i-th class sample set according to the cluster matrix Y where x n represents the nth data sample in the data set, x n i is the nth data sample of the i-th sample set in the data set, n i represents the total number of data samples of the i-th category, d is the data dimension, R represents the real number set, c is the number of data categories, 1≤i≤c; firstly, the data samples in the original data set are normalized, and the high-dimensional data are preprocessed by the principal component analysis method to eliminate the feature null space in the original data.

3. The data compression method based on ratio minimization and sparse representation learning according to claim 1, characterized in that: The robust sparse representation learning model based on ratio minimization is expressed as: Where W∈R d×m is the projection matrix to be learned, the superscript T represents the transpose of the matrix, X represents the training data set, and X i represents the i-th sample set, d and m represent the data dimension and projection dimension respectively, c is the number of data categories, and R is a real number set; n i dimensional column vector whose elements are all 1, n i Represents the total number of data samples of the i-th category; Indicates i The average value of the class samples, The jth data sample in the i-th class sample set, ||·|| 1，2 Indicates l 1,2 norm, and I represents the identity matrix.

4. The data compression method based on ratio minimization and sparse representation learning according to claim 1, characterized in that: The optimal data projection matrix can be obtained by continuously iterating and optimizing the learning model, updating the projection matrix and the cluster center vector until the objective function converges, including: Step 1: First, use the robust sparse representation learning model based on ratio minimization to construct the following optimization problem for solution: Among them, Tr represents the trace of the matrix; λ represents the penalty parameter, which is calculated and updated by the following formula: D i (1≤i≤c) is a diagonal matrix, where the j-th diagonal element is: ||·||2 represents the l2 norm; D is also a diagonal matrix, and its i-th diagonal element is: Among them, x i Represents the i-th data sample in the training data set; Through further deduction, formula (2) can be transformed into the following problem: For the above problem, it is about variables W and μ i They are all convex optimization problems, so they have closed-form solutions; Step 2, calculate the parameter λ according to formula (3); Step 3: Update and calculate the diagonal matrix D according to formula (4) and formula (5) respectively i (i=1,2,...,c) and diagonal matrix D; Step 4: For the i-th data sample, by using equation (6) about μ i Taking the derivative and setting it to 0, we can get μ i The optimal solution is as follows: Step 5: Substitute the optimal solution μ obtained by equation (7) into i Substituting into equation (6), we can obtain the following optimization problem for the projection matrix W: The matrix A is expressed as follows: Here, the optimal solution W of the problem in equation (8) is a matrix composed of the eigenvectors corresponding to the first m smallest eigenvalues ​​of the matrix A; Step 6: Repeat steps 2 to 5 in the training data set to learn the optimal projection matrix W.

5. The data compression method based on ratio minimization and sparse representation learning according to claim 1, characterized in that: For the data to be compressed X, we calculate W by using the optimal projection matrix W learned T X, to achieve data compression.

6. A terminal device comprising a processor, a memory and a computer program stored in the memory; characterized in that: When the processor is executed by a computer, the data compression method based on ratio minimization and sparse representation learning according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium, wherein a computer program is stored in the medium; characterized in that: When the computer program is executed by a processor, the data compression method based on ratio minimization and sparse representation learning according to any one of claims 1 to 5 is implemented.

Images