An Image Processing Method and System Based on Data Dimensionality Reduction Compression

Abstract

The present invention discloses an image processing method and system based on data dimensionality reduction and compression, which belongs to the field of data processing. First, a random Gaussian matrix and a subspace orbit method are used on a distributed agent, i.e., a subagent, to perform rapid matrix decomposition on distributed data, and the decomposition result is intercepted with a set dimension; then, the intercepted decomposition result is transmitted to the main agent, i.e., the central agent, through the subagent; finally, the central agent uses the received subspace matrix to reconstruct the global data matrix and obtain the final approximate reconstruction result. The present invention uses a distributed method to complete the principal component analysis process, and the data transmission pressure is smaller than that of the traditional distributed principal component analysis algorithm, and the operation speed is faster than that of the traditional distributed principal component analysis algorithm, while the operation result has high accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of data processing and relates to an image processing method and system based on data dimensionality reduction and compression. Background Art

[0002] Principal component analysis (PCA) can be used to extract the subspace that best describes high-dimensional data. Projecting the original data onto a low-dimensional subspace can capture the greatest data variation. PCA plays a crucial role in signal processing and machine learning, including face recognition, Poisson denoising in image processing, tensor decomposition, dictionary learning, signal subspace extraction, low-rank optimization, real-time online fault monitoring, and missing data interpolation in gene expression analysis. With the advancement of sensor technology and device storage, massive amounts of data are now collected and stored on geographically distributed data storage devices. A master agent server collects and synchronizes data from decentralized databases, namely, subagent servers, to perform centralized PCA. In this context, distributed PCA holds significant research value. Distributed PCA techniques enable agent networks to implement or approximate centralized PCA solutions, with each agent performing simple computations and exchanging information without transmitting raw data.

[0003] The design of the distributed principal component analysis algorithm depends on the network topology and data partitioning model. The main network topologies are mesh networks and star networks (master-slave structures). In a mesh network, all agents play equal roles; in a star network, the master agent coordinates other sub-agents in the star network. Data partitioning models mainly include distributed column observation (DCO) and distributed row observation (DRO) models. For these models, mature algorithms have been developed and applied, such as the extension of traditional power methods to distributed environments and distributed Oja implementations. However, for large-scale local data matrices, running a complete singular value decomposition on each slave agent requires certain computing power of the agent, and the calculation speed is slow, which cannot complete fast data processing. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art image processing methods, which have the disadvantage of slow data processing, and to provide an image processing method and system based on data dimensionality reduction and compression.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An image processing method based on data dimensionality reduction and compression comprises the following steps:

[0007] Step 1) Obtain sampled data in a star-structured distributed system, classify the sampled data into several sub-agents, and build a local data matrix based on the sub-agents;

[0008] Step 2) performing matrix power method iterative operations on the local data matrix until the number of operations reaches a preset value, thereby obtaining a plurality of temporary matrices;

[0009] Step 3) Decompose the temporary matrix multiple times to obtain a subspace matrix, perform dimension truncation on the subspace matrix, and the subagent transmits the truncation subspace matrix to the master agent of the star-structured distributed system;

[0010] Step 4) Based on the data transmitted by the subagents, the master agent reconstructs the local data matrices of each subagent and splices the reconstruction results into a global data matrix. The global data matrix is then decomposed and dimensionally truncated in sequence. The truncated subspace matrices are then used to reconstruct a new global matrix. Based on the reconstructed new global matrix, the image processing is completed.

[0011] Preferably, step 2) is specifically:

[0012] Step 21) The algorithm is initialized and the subagent randomly generates a random Gaussian matrix Φ with a dimension of d×l;

[0013] Step 22) Perform the first loop operation to construct temporary matrices T1 and T2, and calculate the local information X i Perform autocorrelation processing.

[0014] T1=X i Φ,

[0015]

[0016] Step 23) Perform the second and subsequent loop operations until the number of loops reaches the threshold value q.

[0017] T1=X i T2,

[0018]

[0019] Preferably, step 3) is specifically:

[0020] Step 31) Perform QR decomposition on the temporary matrices T1 and T2 to obtain two unitary matrices Q1 and Q2.

[0021] T1=Q1R1,

[0022] T2=Q2R2.

[0023] Step 32) Use the unitary matrices Q1, Q2 obtained in step 31) and the local data matrix X i , construct the matrix Z i ,

[0024]

[0025] Step 33) For the matrix Z i Do singular value decomposition,

[0026]

[0027] Step 34) performs r-dimension interception on the matrix decomposed in step 33).

[0028]

[0029] V i =[V i，r |V i，d-r ],

[0030] ∑ i =[∑ i，r |∑ i，d-r ]

[0031] Step 35) Use the unitary matrices Q1 and Q2 obtained in step 31) and the left and right singular vector matrices intercepted in the previous step to construct new left and right singular vector matrices.

[0032]

[0033]

[0034] Step 36) The matrix obtained in step 35) and the ∑ obtained in step 34) i，r Transmit to the master agent.

[0035] Preferably, step 4) is specifically:

[0036] Step 41) Reconstruct the local data matrix of each sub-agent,

[0037]

[0038] Where i=1,...,s.

[0039] Step 42) All reconstructed sub-matrices are spliced into a global matrix.

[0040]

[0041] Step 43) Perform singular value decomposition on the spliced global matrix and perform r-dimensional interception on the decomposition result.

[0042]

[0043] U g =[U g，r |U g，m-r ],

[0044] Vg =[V g，r |V g，d-r ],

[0045] ∑ g =[∑ g，r |∑ g，d-r ]

[0046] Step 44) Reconstruct the global matrix using the truncated subspace.

[0047]

[0048] Preferably, the random Gaussian matrix Φ has a dimension of 784×30.

[0049] Preferably, the subagent cycle number threshold is q=2.

[0050] Preferably, in the r-dimensional truncation performed on the subspace matrix result obtained by decomposition, the truncation dimension r=15.

[0051] An image processing system based on data dimensionality reduction and compression, comprising:

[0052] The sampling module is used to obtain sampled data in a star-structured distributed system, classify the sampled data into several sub-agents, and build a local data matrix based on the sub-agents;

[0053] An iteration module is used to perform matrix power method iterative operations on the local data matrix until the number of operations reaches a preset value, thereby obtaining a number of temporary matrices;

[0054] The decomposition module is used to perform multiple decompositions on the temporary matrix to obtain a subspace matrix, perform dimension truncation on the subspace matrix, and the subagent transmits the truncation subspace matrix to the main agent of the star-structured distributed system;

[0055] In the reconstruction module, based on the data transmitted by the sub-agents, the main agent reconstructs the local data matrices of each sub-agent, splices the reconstruction results into a global data matrix, and then decomposes and truncates the global data matrix in turn. The truncation subspace matrix is then used to reconstruct a new global matrix. Based on the reconstructed new global matrix, the image processing is completed.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This invention discloses an image processing method based on data dimensionality reduction and compression. First, distributed agents (subagents) use random Gaussian matrices and the subspace orbit method to perform rapid matrix decomposition on distributed data, truncating the decomposition results to a set dimension. The subagents then transmit the truncated decomposition results to the master agent (central agent). Finally, the central agent uses the received subspace matrix to reconstruct the global data matrix and obtain a final approximate reconstruction result. This method uses a distributed approach to complete the principal component analysis process, resulting in less data transmission pressure and faster computation speed than traditional distributed principal component analysis algorithms, while also achieving high computational accuracy. Compared to conventional centralized principal component analysis algorithms, this method can operate on large amounts of distributed data, overcoming the limitations of centralized algorithms that are limited to single-machine operations. This method accelerates traditional distributed algorithms by using the subspace orbit method and, by setting a Gaussian matrix of appropriate dimensions, increases computational speed while maximizing computational accuracy. This method also uses a dimensional truncation method, setting a truncation scheme of appropriate dimensions, to reduce data transmission pressure from subagents to the master agent while ensuring computational accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of the present invention;

[0059] Figure 2 A schematic diagram showing the difference in computing speed between the method of the present invention and the traditional method;

[0060] Figure 3 A schematic diagram showing the difference in accuracy between the method of the present invention and the traditional method;

[0061] Figure 4 Figure (a) shows the image before and after the treatment using the method of the present invention (b). DETAILED DESCRIPTION

[0062] The present invention is described in further detail below with reference to the accompanying drawings:

[0063] Example 1

[0064] An image processing method based on data dimensionality reduction and compression comprises the following steps:

[0065] Step 1) Obtain sampled data in a star-structured distributed system, classify the sampled data into several sub-agents, and build a local data matrix based on the sub-agents;

[0066] Step 2) performing matrix power method iterative operations on the local data matrix until the number of operations reaches a preset value, thereby obtaining a plurality of temporary matrices;

[0067] Step 3) Decompose the temporary matrix multiple times to obtain a subspace matrix, perform dimension truncation on the subspace matrix, and the subagent transmits the truncation subspace matrix to the master agent of the star-structured distributed system;

[0068] Step 4) Based on the data transmitted by the subagents, the master agent reconstructs the local data matrices of each subagent and splices the reconstruction results into a global data matrix. The global data matrix is then decomposed and dimensionally truncated in sequence. The truncated subspace matrices are then used to reconstruct a new global matrix. Based on the reconstructed new global matrix, the image processing is completed.

[0069] Example 2

[0070] An image processing method based on data dimensionality reduction and compression, such as Figure 1 As shown, the following steps are included:

[0071] Step 1: Data acquisition and construction

[0072] First, collect the local data matrix X of dimension m×d from subagent i i , the form is as follows,

[0073]

[0074] Each column of data is the result collected by the sub-agent through a single sampling.

[0075] In the subsequent processing, the matrices collected by the s sub-agents are Perform the operation.

[0076] Step 2: Process the sub-agent local data matrix, construct matrices T1 and T2, perform matrix power method iterations, update the results of each step, and reduce the weights of the spatial directions corresponding to small singular values until the number of operations reaches the preset value, that is:

[0077] j=q,

[0078] Among them, j is the iteration number, and q is the preset maximum number of iterations.

[0079] Step 2-1: Initialize the algorithm. The subagent randomly generates a random Gaussian matrix Φ with a dimension of d×l, where the dimension of the random Gaussian matrix Φ is 784×30.

[0080] Step 2-2: Perform the first loop operation, construct temporary matrices T1 and T2, and perform autocorrelation processing on local information as follows:

[0081] T1=X i Φ,

[0082]

[0083] Step 2-3: Perform the second and subsequent loop operations until the number of loops reaches a threshold value q, where the subagent loop number threshold value is q=2.

[0084] T1=X i T2,

[0085]

[0086] Step 3: The subagent performs multiple decompositions on the temporary matrices T1 and T2, truncating the resulting subspace matrices by the r-dimensionality, where r is a manually defined variable. Finally, the subagent transmits the truncated subspace matrices to the master agent via the network.

[0087] Step 3-1: Perform QR decomposition on T1 and T2 to obtain two unitary matrices Q1 and Q2.

[0088] T1=Q1R1,

[0089] T2=Q2R2.

[0090] Step 3-2: Use the unitary matrices Q1, Q2 and the local data matrix X obtained in the previous step i , construct the matrix Z i ,

[0091]

[0092] Step 3-3: Matrix Z i Do singular value decomposition,

[0093]

[0094] Step 3-4: Truncate the matrix decomposed in the previous step in r dimension, with the truncation dimension r=15.

[0095]

[0096] V i =[V i，r |V i，d-r ],

[0097] ∑ i =[∑ i，r |∑ i，d-r ]

[0098] Step 3-5: Use the unitary matrices Q1, Q2 obtained in step 3-1 and the left and right singular vector matrices intercepted in the previous step to construct new left and right singular vector matrices.

[0099]

[0100]

[0101] Step 3-6: Substitute the matrix obtained in step 3-5 and the ∑ obtained in step 3-4 i，r Transmit to the master agent.

[0102] Step 4: The master agent receives the data transmitted by the subagents, first reconstructs the local data matrix of each subagent, then splices the reconstruction results into a global data matrix, performs singular value decomposition on the global data matrix, and performs r-dimensional truncation on the decomposition result. Finally, the truncation subspace matrix is used to reconstruct the global matrix.

[0103] Step 4-1: Reconstruct the local data matrix of each sub-agent,

[0104]

[0105] Where i=1,...,s.

[0106] Step 4-2: Concatenate all reconstructed sub-matrices into a global matrix.

[0107]

[0108] Step 4-3: Perform singular value decomposition on the spliced global matrix and perform r-dimensional interception on the decomposition result.

[0109]

[0110] U g =[U g，r |U g，m-r ],

[0111] V g =[V g，r |V g，d-r ],

[0112] ∑ g =[∑ g，r |∑ g，d-r ].

[0113] Step 4-4: Use the truncated subspace to reconstruct the global matrix.

[0114]

[0115] Example 3

[0116] This example conducts experiments on the MNIST dataset. The dataset includes 60,000 handwritten digit images from 0 to 9, and the resolution of each image is 28x28 (pixels); the dataset can be represented by a matrix with a dimension of 60,000x784. The present invention randomly and evenly divides the dataset into s=20 distributed sub-agents for calculation, and compares the final reconstructed matrix with the reconstructed matrix obtained using centralized principal component analysis and traditional distributed principal component analysis. The present invention compares the F norm of the error between the matrices reconstructed by the three methods and the true global data matrix.

[0117] Example 4

[0118] An image processing system based on data dimensionality reduction and compression, comprising:

[0119] The sampling module is used to obtain sampled data in a star-structured distributed system, classify the sampled data into several sub-agents, and build a local data matrix based on the sub-agents;

[0120] An iteration module is used to perform matrix power method iterative operations on the local data matrix until the number of operations reaches a preset value, thereby obtaining a number of temporary matrices;

[0121] The decomposition module is used to perform multiple decompositions on the temporary matrix to obtain a subspace matrix, perform dimension truncation on the subspace matrix, and the subagent transmits the truncation subspace matrix to the main agent of the star-structured distributed system;

[0122] In the reconstruction module, based on the data transmitted by the sub-agents, the main agent reconstructs the local data matrices of each sub-agent, splices the reconstruction results into a global data matrix, and then decomposes and truncates the global data matrix in turn. The truncation subspace matrix is then used to reconstruct a new global matrix. Based on the reconstructed new global matrix, the image processing is completed.

[0123] Depend on Figure 2 、 Figure 3 and Figure 4 It can be seen that the reconstructed matrix obtained by the present invention is extremely close to the centralized principal component analysis and the traditional distributed principal component analysis, and has high accuracy. At the same time, the introduction of random matrices and the subspace orbit method can accelerate the centralized principal component analysis, while improving the speed and ensuring the accuracy of the calculation. Therefore, the method of the present invention is efficient and highly practical.

[0124] In summary, the image processing method based on data dimensionality reduction and compression of the present invention aims to reduce the dimensionality of data stored on multiple agents. This method approximates a centralized principal component analysis solution within an agent network structure, where each agent performs simple computations and exchanges information without transmitting the original data. Furthermore, it applies matrix random low-rank approximation techniques to accelerate singular value decomposition operations with acceptable accuracy loss.

[0125] The above content is only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. An image processing method based on data dimensionality reduction and compression, characterized in that: The following steps are involved: Step 1) Obtain sampled data in a star-structured distributed system, classify the sampled data into several sub-agents, and build a local data matrix based on the sub-agents; Step 2) Perform matrix power method iterations on the local data matrix until the number of operations reaches a preset value, and obtain several temporary matrices, specifically: Step 21) The algorithm is initialized and the subagent randomly generates a dimension A random Gaussian matrix ; Step 22) Perform the first loop operation to build a temporary matrix and , for the local data matrix Perform autocorrelation processing. Step 23) Perform the second and subsequent loop operations until the number of loops reaches the threshold , ； Step 3) Decompose the temporary matrix multiple times to obtain the subspace matrix, and perform dimension truncation on the subspace matrix. The subagent transmits the truncation subspace matrix to the master agent of the star-structured distributed system. Specifically: Step 31) Temporary matrix and Do QR decomposition and get two unitary matrices and , Step 32) Use the unitary matrix obtained in step 31) , and the local data matrix , construct the matrix , Step 33) Matrix Do singular value decomposition, Step 34) Perform the matrix decomposed in step 33) Dimension interception: Step 35) Use the unitary matrix obtained in step 31) , Construct new left and right singular vector matrices with the left and right singular vector matrices intercepted in the previous step, Step 36) The matrix obtained in step 35) , and the one obtained in step 34) Transmit to the master agent; Step 4) Based on the data transmitted by the subagents, the master agent reconstructs the local data matrices of each subagent and splices the reconstruction results into a global data matrix. The global data matrix is then decomposed and dimensionally truncated. The truncated subspace matrices are then used to reconstruct a new global matrix. Based on the reconstructed new global matrix, the image is processed as follows: Step 41) Reconstruct the local data matrix of each sub-agent, in ; Step 42) All reconstructed sub-matrices are concatenated into a global matrix. Step 43) Perform singular value decomposition on the spliced global matrix and perform Dimension interception; ， Step 44) Use the truncated subspace to reconstruct the global matrix. 。 2. The image processing method based on data dimensionality reduction and compression according to claim 1, characterized in that: Random Gaussian matrix The dimension is .

3. The image processing method based on data dimensionality reduction and compression according to claim 1, characterized in that: The subagent cycle threshold is .

4. The image processing method based on data dimensionality reduction and compression according to claim 1, characterized in that: The decomposition of the subspace matrix results Dimension truncation, truncation dimension .

5. An image processing system based on data dimensionality reduction and compression, based on the image processing method based on data dimensionality reduction and compression according to any one of claims 1 to 4, characterized in that: include: The sampling module is used to obtain sampled data in a star-structured distributed system, classify the sampled data into several sub-agents, and build a local data matrix based on the sub-agents; An iteration module is used to perform matrix power method iterative operations on the local data matrix until the number of operations reaches a preset value, thereby obtaining a number of temporary matrices; The decomposition module is used to perform multiple decompositions on the temporary matrix to obtain a subspace matrix, perform dimension truncation on the subspace matrix, and the subagent transmits the truncation subspace matrix to the main agent of the star-structured distributed system; In the reconstruction module, based on the data transmitted by the sub-agents, the main agent reconstructs the local data matrices of each sub-agent, splices the reconstruction results into a global data matrix, and then decomposes and truncates the global data matrix in turn. The truncation subspace matrix is then used to reconstruct a new global matrix. Based on the reconstructed new global matrix, the image processing is completed.

Images