Efficient algorithm for large-scale sparse matrix decomposition

Abstract

The invention provides an efficient algorithm for large-scale sparse matrix decomposition, and the algorithm comprises the steps: finding out two low-rank matrixes W < Rm * k and H < Rk * n for a given large-scale sparse matrix A < Rm * n, and enabling A to be approximately equal to WH; the method specifically comprises the following steps: generating a random matrix R belonging to Rn * l; calculating a matrix # imgabs0 # after dimension reduction to construct an optimization problem with sparse constraint; solving by adopting an alternating direction multiplier method; and post-processing the result. According to the method, random projection dimensionality reduction is performed on the original sparse matrix, so that the calculation complexity is reduced; a sparse regularization item is introduced, and the sparsity of the matrix is kept; and solving by adopting an alternating direction multiplier method, and accelerating convergence. According to the method, the calculation and storage cost can be greatly reduced while the decomposition precision is guaranteed, and the method is suitable for large-scale data processing in the fields of machine learning, data mining and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer software algorithms, and in particular to an efficient algorithm for large-scale sparse matrix decomposition, which is specifically applied to the fields of machine learning, data mining, signal processing and scientific computing. Background Art

[0002] In the era of big data, processing and analyzing large-scale sparse matrices has become a key challenge. Traditional matrix decomposition algorithms, such as singular value decomposition (SVD) and non-negative matrix factorization (NMF), have the following problems when processing large-scale sparse matrices:

[0003] (1) High computational complexity: The computational complexity of these algorithms is usually P(mnmin(m,n)), which is too expensive for large matrices.

[0004] (2) Large memory consumption: The entire matrix and intermediate results need to be stored, which has high memory requirements and is not suitable for resource-constrained environments.

[0005] (3) Loss of sparsity: Traditional algorithms may destroy the sparse structure during the calculation process, resulting in additional storage and computational burdens.

[0006] Therefore, it is of great significance to develop an efficient sparse matrix decomposition algorithm that reduces computational complexity and memory consumption while ensuring accuracy. Summary of the invention

[0007] The present invention aims to provide an efficient algorithm for large-scale sparse matrix decomposition to solve the problems of high computational complexity, large memory consumption, loss of sparsity, etc. existing in the prior art.

[0008] To achieve the above object, the present invention proposes the following technical solutions:

[0009] An efficient algorithm for large-scale sparse matrix decomposition. For a given large-scale sparse matrix A∈R m×n , find two low-rank matrices W∈R m×k and H∈R k×n , so that A≈WH;

[0010] The specific steps include:

[0011] Step 1, random projection dimensionality reduction: Generate a random matrix R∈R n×l ; Calculate the matrix after dimension reduction

[0012] Step 2: Construct an optimization problem with sparse constraints.

[0013] Step 3: Solve using the alternating direction multiplier method: transform the original problem into an equivalent constrained optimization problem; introduce the Lagrange multiplier U W and U H , construct the augmented Lagrangian function; update W, H, Z W , Z H , Lagrange multiplier U W and U H ; Check convergence conditions and calculate the original residual r prim and the dual residual r dual , if r prim <∈ prim And r dual <∈ dual , then stop the iteration; where, ∈ prim ,∈ dual The allowable errors for the primal residual and the dual residual, respectively.

[0014] Step 4, result post-processing: After the algorithm converges, output the decomposition result W * and H * , perform sparsity adjustment and reconstruction error evaluation on the results.

[0015] The algorithm of the present invention can significantly reduce the computational complexity and memory consumption while ensuring the decomposition accuracy, and is suitable for processing ultra-large-scale sparse matrices. The beneficial effects are specifically reflected in the following aspects:

[0016] (1) The present invention uses random projection technology to map high-dimensional sparse matrices to low-dimensional space, thereby reducing computational complexity.

[0017] (2) The present invention introduces a sparse regularization term in the decomposition process to maintain the sparsity of the matrix and reduce storage requirements.

[0018] (3) The present invention adopts the alternating direction multiplier method to efficiently optimize the algorithm to solve the decomposition problem and accelerate the convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The figure is an overall flow chart of the efficient algorithm for large-scale sparse matrix decomposition described in the present invention. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] 1. Embodiments of the technical solution of the present invention

[0022] like Figure 1 As shown, an efficient algorithm for large-scale sparse matrix decomposition according to the present invention comprises the following steps:

[0023] Step 1: Random Projection Dimensionality Reduction

[0024] To reduce the computational complexity, the matrix A is first randomly projected to obtain a low-dimensional representation.

[0025] (1) Generate a random matrix R∈R n×l , where l<<n, l is usually of the order of log(n), and the elements of R can be obtained from the standard normal distribution Medium sampling.

[0026] (2) Calculate the matrix after dimensionality reduction

[0027]

[0028] Step 2: Construct an optimization problem with sparse constraints

[0029] Construct the following optimization problem:

[0030]

[0031] Where: |·|F is the Frobenius norm, defined as Used to measure the difference between matrices; |·|1 is the L1 norm at the element level, defined as |X|1=∑i,j|Xij|, which is used for sparse regularization to promote the sparsity of the solution; λ W and λ H are sparse regularization parameters, which are used to control the sparsity of W and H respectively.

[0032] Step 3: Solve using the alternating direction method of multipliers (ADMM)

[0033] In order to facilitate the solution, the original problem is converted into an equivalent constrained optimization problem:

[0034]

[0035] Introducing the Lagrange multiplier U W and U H , construct the augmented Lagrangian function:

[0036]

[0037] Among them, ρ>0 is the penalty parameter,<X,Y> represents the matrix inner product, Z W ∈R m×k and Z H ∈R k×l Both are auxiliary variables used to decompose the original optimization problem into easy-to-handle sub-problems; U W ∈R m×k and U H ∈R k×lBoth are Lagrange multipliers; used in ADMM to handle the constraint condition W = Z W and H = Z H .

[0038] Alternating update steps:

[0039] (1) Update W:

[0040]

[0041] Expanding and arranging the above quadratic form, we obtain the matrix equation for W:

[0042]

[0043] Solving the linear equations, we get W (t+1) . Where t represents the number of iterations, starting from 0, and t increases by 1 each time a complete iteration process is completed.

[0044] (2) Update H:

[0045]

[0046] Similarly, we get the matrix equation about H:

[0047]

[0048] Solving this linear equation system, we get H (t+1) .

[0049] (3) Update Z W and Z H :

[0050] Z W :

[0051]

[0052] Apply a soft thresholding function element-wise:

[0053]

[0054] Z H :

[0055]

[0056] Apply a soft thresholding function element-wise:

[0057]

[0058] Soft threshold function definition:

[0059]

[0060] The soft threshold function is used to update Z W and Z H It promotes sparsity.

[0061] (4) Update the Lagrange multiplier U W and U H :

[0062]

[0063] (5) Check convergence conditions:

[0064] Compute the primal and dual residuals:

[0065]

[0066]

[0067] If r prim <∈ prim And r dual <∈ dual , then the iteration stops. Among them, ∈ prim ,∈ dual The allowable errors of the primal residual and the dual residual are used as convergence thresholds to control the convergence accuracy.

[0068] Step 4: Post-processing of results

[0069] After the algorithm converges, the decomposition result W is output * and H * , the results can be processed as follows:

[0070] (1) Sparsity adjustment: If the result sparsity does not meet the requirements, you can adjust λ W and λ H Rerun the algorithm.

[0071] (2) Reconstruction error evaluation: Calculate the original matrix A and the reconstructed matrix W * H * The error between:

[0072]

[0073] 2. Application Experiment of the Invention

[0074] An actual large-scale sparse matrix is ​​taken as an example to verify the effectiveness of this algorithm.

[0075] 1. Dataset:

[0076] Matrix size: m = 10 6 , n = 10 5 .

[0077] Sparsity: The proportion of non-zero elements is about 0.01%.

[0078] 2. Parameter settings:

[0079] Low rank: k = 50. Random projection dimension: l = 500. Sparse regularization parameter: λ W =0.1,λ H =0.1. Penalty parameter: ρ = 1. Maximum number of iterations: T max =100. Convergence threshold: ∈ prim =∈ dual =10 -4 .

[0080] 3. Experimental results:

[0081] (1) Number of iterations: The algorithm converges after 30 iterations.

[0082] (2) Sparsity: The sparsity of W and H obtained by decomposition is 95% and 90% respectively.

[0083] (3) Reconstruction error: E = 0.05, indicating that the decomposition accuracy is high.

[0084] (4) Computation time: Compared with the traditional SVD algorithm, the computation time is reduced by about 85%.

[0085] 4. Conclusion:

[0086] Experimental results verify the efficiency and accuracy of this algorithm in large-scale sparse matrix decomposition.

[0087] 3. The following factors have been considered in the selection of parameters during the research and development of the present invention:

[0088] 1. k (target low rank):

[0089] The selection depends on the specific application scenario and is generally determined through experiments or cross-validation. The smaller k is, the lower the computational complexity is, but it may lead to an increase in reconstruction error.

[0090] 2. Lambda W and λ H (sparse regularization parameter):

[0091] Controls the sparsity. The larger the value, the sparser the solution. There is a trade-off between sparsity and reconstruction error.

[0092] 3. ρ (penalty parameter):

[0093] Affects the convergence speed and accuracy. It can be fixed or adaptively adjusted according to the residual during the iteration process.

[0094] 4. l (random projection dimension):

[0095] l<<n is selected to reduce the computational complexity while ensuring acceptable information loss. Generally, l=O(log n) or a slightly larger value is selected to improve the approximation accuracy.

[0096] 5. T max (Maximum number of iterations):

[0097] Prevent the algorithm from falling into an infinite loop. If the algorithm is in T max If convergence has not occurred after iterations, the parameter settings or algorithm implementation need to be checked.

[0098] 6.∈ prim and ∈ dual (convergence threshold):

[0099] Set to a small positive number, such as 10 -4 or 10 -5 If it is too large, it may stop prematurely, and if it is too small, it may cause the calculation to take too long.

Claims

1. An efficient algorithm for large-scale sparse matrix decomposition, characterized in that: For a given large-scale sparse matrix A∈R m×n , find two low-rank matrices W∈R m×k and H∈R k×n , so that A≈WH; The specific steps include: Step 1, random projection dimensionality reduction: Step 2: Construct an optimization problem with sparse constraints: Step 3, use the alternating direction multiplier method to solve: Step 4, post-processing of results: Where A is the original large-scale sparse matrix, m is the number of rows of matrix A, and n is the number of columns of matrix A; k is the preset target low rank, satisfying k<<min(m,n), which means that matrix A is decomposed into two low-rank matrices W and H, where k is the decomposition rank; W∈R m×k is the basis matrix, which is the variable to be optimized and represents the basis of the new feature space; H∈R k×l is the coefficient matrix, the variables to be optimized, and the coefficients expressed on the basis W; R∈R n×l is a random projection matrix, l represents the dimension after random projection, and satisfies l<<n. The elements of R are usually from the standard normal distribution. Independent sampling.

2. The efficient algorithm for large-scale sparse matrix decomposition according to claim 1, characterized in that: The specific steps of step 1 are as follows: (1) Generate a random matrix R∈R n×l ; (2) Calculate the matrix after dimensionality reduction 3. The efficient algorithm for large-scale sparse matrix decomposition according to claim 1, characterized in that: The step 2 constructs the following optimization problem: Where |·|F is the Frobenius norm, defined as |·|1 is the element-level L1 norm, defined as |X|1=∑i,j|Xij|; λ W and λ H is the sparsity regularization parameter.

4. The efficient algorithm for large-scale sparse matrix decomposition according to claim 1, characterized in that: The step 3 comprises the following steps: (1) Convert the original problem into an equivalent constrained optimization problem: s.t.W=Z W ,H=Z H (2) Introducing the Lagrange multiplier U W and U H , construct the augmented Lagrangian function: Among them, ρ>0 is the penalty parameter,<X,Y> represents the matrix inner product, Z W ∈R m×k and Z H ∈R k×l are auxiliary variables, U W ∈R m×k and U H ∈R k×l are all Lagrange multipliers.

5. The efficient algorithm for large-scale sparse matrix decomposition according to claim 4, characterized in that: The step 3 further includes the following steps: when the number of iterations t is less than the maximum number of iterations T max , perform the following alternating update steps: (1) Update W: Expanding and arranging the above quadratic form, we obtain the matrix equation for W: Solving the linear equations, we get W (t+1) ; Where t represents the number of iterations, starting from 0, and t increases by 1 after each complete iteration; (2) Update H: Similarly, we get the matrix equation about H: Solving this linear equation system, we get H (t+1) ; (3) Update Z W and Z H : right Apply a soft thresholding function element-wise: right Apply a soft thresholding function element-wise: Soft threshold function definition: The soft threshold function is used to update Z W and Z H When promoting sparsity; (4) Update the Lagrange multiplier U W and U H : (5) Check convergence conditions: Compute the primal and dual residuals: If r prim <∈ prim And r dual <∈ dual , then stop the iteration; where, ∈ prim ,∈ dual The allowable errors for the primal residual and the dual residual, respectively.

6. The efficient algorithm for large-scale sparse matrix decomposition according to claim 1, characterized in that: After the algorithm converges, step 4 outputs the decomposition result W * and H * , and process the results as follows: (1) Sparsity adjustment: If the result sparsity does not meet the requirements, adjust λ W and λ H Re-run the algorithm; (2) Reconstruction error evaluation: Calculate the original matrix A and the reconstructed matrix W * H * The error between: reconstruction error

Images