Sparse tensor joint completion method and device based on differential privacy

Abstract

The present invention discloses a sparse tensor joint completion method based on differential privacy: (1) grouping training set data according to data dimensions; (2) establishing a sparse tensor joint completion model based on differential privacy, initializing the global factor matrix of the central server; (3) using the elastic average stochastic gradient descent method to train the model, and perturbing the output results through differential privacy technology; (4) each client completes the missing data according to the model. Compared with the existing technology using differential privacy, the present invention can accurately complete the data completion task while ensuring user privacy, achieves excellent results in efficiency, completion accuracy and versatility, and has strong practical application value. The present invention also provides a corresponding sparse tensor joint completion device based on differential privacy.

Description

Technical Field

[0001] The present invention belongs to the field of privacy protection technology, and more specifically, relates to a sparse tensor joint completion method and device based on differential privacy. Background Art

[0002] With the rapid development of the internet, data has exploded. This data is large in scale, multi-dimensional, and complex in structure, containing a wealth of high-value information. This data not only presents unprecedented opportunities for scientific research but also has a significant impact on the rapid development of society and changes in people's lifestyles. However, due to issues with collection, transmission, or the data itself, much of this data is often incomplete, significantly impacting subsequent data processing and analysis. To address the issue of missing high-dimensional data, tensor completion technology can extract features from the original data while preserving its high-dimensional internal structure, obtaining a low-dimensional representation and subsequently completing the missing data. In recent years, sparse tensor completion technology has made significant progress and has been widely applied in fields such as recommendation systems, medical diagnosis, network analysis, and spatiotemporal data prediction. For example, in the recommendation field, tensor completion can sort through disorganized personal information, uncover potential connections between consumers and products, and help consumers find products of interest. However, to provide the highest quality service possible, tensor completion requires the collection of large amounts of user data to build models, which raises serious privacy concerns.

[0003] Privacy protection is a major challenge in tensor completion technology. The challenge lies in ensuring data completion quality while simultaneously defending against a growing array of attacks and preventing the leakage of user privacy data. On the one hand, if user privacy is not adequately protected, users will distrust the system and, in turn, refuse to provide personal information to service providers or use the corresponding services. On the other hand, overly stringent privacy protection policies can reduce data availability, hindering service providers from providing high-quality user services and potentially leading to large-scale user churn. Therefore, it is crucial to ensure privacy protection in tensor completion while ensuring data availability. While existing privacy-preserving tensor completion methods can protect user privacy data to a certain extent, they introduce excessive noise, resulting in insufficient model training accuracy. Furthermore, they do not consider joint tensor completion in multi-server scenarios, making them less versatile. Summary of the Invention

[0004] In response to the above defects or improvement needs of the existing technology, the present invention provides a sparse tensor joint completion method based on differential privacy, which aims to combine differential privacy technology and use targeted perturbation means to interfere with the algorithm output so that attackers cannot infer any specific record in the database based on the query results, prevent differential attacks, and avoid sensitive information leakage; solve the sparse tensor joint completion model in a multi-server scenario through the elastic average stochastic gradient descent method, and introduce a parallel tensor decomposition mechanism to address the update conflict problem of the factor matrix, thereby solving the technical problem of sparse tensor joint completion based on differential privacy.

[0005] To achieve the above objectives, according to one aspect of the present invention, a sparse tensor joint completion method based on differential privacy is provided, the method comprising the following steps:

[0006] (1) Number each client node t, group each client node data set according to data dimension information and introduce parallel tensor decomposition method to obtain a numbered training data set;

[0007] (2) According to the completion task requirements, a loss function is designed, and the Laplace mechanism is adopted to establish a sparse tensor joint completion model based on differential privacy, and initialize the global factor matrix of the central server. and the local factor matrix A of each client node [t] ,B [t] ,C [t] ;

[0008] (3) Using the numbered training dataset in step (1), the elastic average stochastic gradient descent method is used for training to extract high-dimensional data features and obtain a sparse tensor joint completion model based on differential privacy;

[0009] (4) Use the above-trained tensor decomposition model to complete the missing data.

[0010] In one embodiment of the present invention, step (1) specifically includes:

[0011] Split the training dataset into multiple independent blocks Z p ,and in is the original tensor data. x is the tensor An observation item of , and Then for any observation x∈Z p ,x′∈Z p′ In terms of x i ≠x i ′ and x j ≠x j '. Therefore, any independent block Z p ,Z p′The first and second dimensions are non-overlapping. The non-overlapping independent blocks that meet the above conditions are defined as group s, and the number of independent blocks in each group is T, that is, The local tensor data of client i is The grouping rule formula is as follows:

[0012]

[0013] j s,i =(i+s)mod d

[0014] Among them, independent blocks The coordinate range is b i =(1,I [i] ),b j =(j[J / q],(j+1)[J / q]), k =(1,K). For each group, the independent blocks of each client do not overlap to avoid update conflicts during training.

[0015] In one embodiment of the present invention, step (2) specifically includes:

[0016] (2.1) The number of client nodes is known to be T, and the local data of each client is The completed data is Its loss function can be expressed as the sum of T client tensor decomposition loss functions, expressed as follows:

[0017]

[0018] (2.2) For the global optimization problem, all clients share the non-user global dimension factor matrix. Therefore, the global optimal problem is as follows:

[0019]

[0020] Among them, B [1] =B [2] =…=B [T] , C [1] =C [2] =…=C [T] ;

[0021] (2.3) Decompose the global optimization problem into T local subproblems, set a quadratic penalty between the local factor matrix and the global factor matrix, and add the target perturbation. Therefore, the local optimal problem is as follows:

[0022]

[0023]

[0024] Among them, Ω is the tensor observation term, They are the local factor matrices A [t] The i-th row vector of B [t] The j-th row vector of C [t] The kth row vector of λ A ,λ B ,λ C is the regularization coefficient of the factor matrix, is the transpose of the k-th row vector of the target disturbance noise matrix N;

[0025] (2.4) Initialize the local factor matrix and the global factor matrix according to the Gaussian distribution, and use the norm upper bound m to constrain the vectors in the factor matrix.

[0026] In one embodiment of the present invention, step (3) specifically includes:

[0027] (3.1) The central server sends training tasks to each client node based on the training data set grouped in step (2);

[0028] (3.2) Each client node trains the local data through the sparse tensor joint completion model according to the training task in step (3.1) and updates the local factor matrix A [t] ,B [t] ,C [t] ;

[0029] (3.3) After completing the training task, each client node sends the updated local factor matrix to the central server;

[0030] (3.4) The central server collects the local factor matrix B of each client [t] ,C [t] , for the global factor matrix Make updates;

[0031] (3.5) The central server repeats step (3.1) to continue issuing training tasks until each set of training set data has been trained.

[0032] In one embodiment of the present invention, step (4) specifically includes:

[0033] (4.1) The central server will train the global factor matrix Send it to each client node;

[0034] (4.2) Each client is based on the local factor matrix A [t] and the global factor matrix Calculate the approximate value of the missing data and then complete the sparse tensor joint completion task based on differential privacy.

[0035] In one embodiment of the present invention, the step (3.2) specifically includes:

[0036] (3.2.1) Each client traverses the observation items according to the training task and updates the local factor matrix A [t] ,B [t] . Local factor matrix A [t] The i-th row vector partial derivative of for:

[0037]

[0038] Update factor matrix A [t] The calculation formula for the i-th row vector is as follows:

[0039]

[0040]

[0041] Where η is the learning rate of the local factor matrix and m is the upper bound constraint of the norm.

[0042] Local factor matrix B [t] The j-th row vector partial derivative of The calculation formula is as follows:

[0043]

[0044] Update factor matrix B [t] The calculation formula for the j-th row vector is as follows:

[0045]

[0046]

[0047] (3.2.2) Each client traverses the observation items according to the training task and updates the local factor matrix C [t] . Local factor matrix C [t] The rules for updating the k-th row vector are as follows:

[0048]

[0049] in, is the local factor matrix C [t] The kth row vector gradient sum, The calculation formula is as follows:

[0050]

[0051] n k: is the kth row vector of the target perturbation noise matrix N, which obeys the Laplace distribution. The element n of the noise matrix N iskr The probability density distribution formula is as follows:

[0052]

[0053] Where ∈ is the privacy budget, R is the rank of the tensor, and Δ is the L1 sensitivity of the training dataset.

[0054] In one embodiment of the present invention, the step (3.4) specifically includes:

[0055] The central server collects the local factor matrix B of T clients [t] ,C [t] , update the global factor matrix , the calculation formula is as follows:

[0056]

[0057]

[0058] in, is the learning rate of the global factor matrix.

[0059] According to another aspect of the present invention, a sparse tensor joint completion device based on differential privacy is also provided, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the sparse tensor joint completion method based on differential privacy.

[0060] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0061] (1) High efficiency: The invention adopts the elastic average gradient descent method and introduces a parallel tensor decomposition mechanism to avoid update conflicts. It can quickly converge for global optimization problems and improve the efficiency of sparse tensor federation completion.

[0062] (2) High security: This method uses differential privacy technology to prevent attackers from inferring any specific record in the database based on the sparse tensor federated completion model, thereby protecting user privacy.

[0063] (2) High completion accuracy: This method uses the target perturbation method to perturb the factor matrix, which reduces the impact of unnecessary noise on the data, improves data availability, and enhances the data completion effect.

[0064] (3) Strong versatility: The differentially private sparse tensor joint completion method proposed in this invention is a universal method that is applicable to a variety of application scenarios. The sparse tensor joint completion model used in this invention can be used in different fields, networks, and different data completion tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flow chart of a sparse tensor joint completion method based on differential privacy in an embodiment of the present invention;

[0066] Figure 2 Schematic diagram of the overall structure of the sparse tensor joint completion model based on differential privacy in an embodiment of the present invention;

[0067] Figure 3 This is an example of the result of data completion in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0069] The following first explains and illustrates the technical terms of the present invention:

[0070] Tensor CP decomposition: decomposes a tensor into the sum of multiple rank tensors. The rank of a tensor is the minimum number of rank tensors required to generate the original tensor. The calculation formula of its CP decomposition is as follows:

[0071]

[0072] Among them, A, B and C are called factor matrices, which are vector combinations of rank tensors.

[0073] Differential privacy is a strict privacy protection technology that uses noise to interfere with the output of the algorithm, making it impossible for attackers to infer any specific record in the database based on the query results. This prevents differential attacks and avoids the leakage of sensitive information.

[0074] L1 sensitivity: Also known as global sensitivity, it measures the magnitude of a single record that can change the function f under any circumstances. Sensitivity is independent of the dataset and only related to the output of the function f. For the function The calculation formula of its L1 sensitivity is as follows:

[0075]

[0076] Laplace mechanism: It is a privacy protection mechanism in differential privacy technology. For the function The Laplace mechanism is defined as:

[0077]

[0078] Among them, Y i is a random variable that obeys the Laplace distribution Lap(Δf / ∈).

[0079] like Figure 1 As shown, the sparse tensor joint completion method based on differential privacy in this aspect includes the following steps:

[0080] (1) Each client node is numbered t, and the client node data set is grouped according to the data dimension information and the parallel tensor decomposition method is introduced to obtain a numbered training data set:

[0081] Split the training dataset into multiple independent blocks Z p ,and in is the original tensor data. x is the tensor An observation item of , and Then for any observation x∈Z p ,x′∈Z p′ In terms of x i ≠x i ′ and x j ≠x j '. Therefore, any independent block Z p ,Z p′ The first and second dimensions are non-overlapping. The non-overlapping independent blocks that meet the above conditions are defined as group s, and the number of independent blocks in each group is T, that is, The local tensor data of client i is The grouping rule formula is as follows:

[0082]

[0083] j s,i =(i+s)mod d

[0084] Among them, independent blocks The coordinate range is b i =(1,I [i] ),b j =(j[J / q],(j+1)[J / q]),b k =(1,K). For each group, the independent blocks of each client do not overlap to avoid update conflicts during training.

[0085] (2) According to the completion task requirements, a loss function is designed, and the Laplace mechanism is adopted to establish a sparse tensor joint completion model based on differential privacy, and initialize the global factor matrix of the central server. and the local factor matrix A of each client node [t] ,B [t] ,C [t] , specifically including:

[0086] (2.1) The number of client nodes is known to be T, and the local data of each client is The completed data is Its loss function can be expressed as the sum of T client tensor decomposition loss functions, expressed as follows:

[0087]

[0088] (2.2) For the global optimization problem, all clients share the non-user global dimension factor matrix. Therefore, the global optimal problem is as follows:

[0089]

[0090] Among them, B [1] =B [2] =…=B [T] , C [1] =C [2] =…=C [T] ;

[0091] (2.3) Decompose the global optimization problem into T local subproblems, set a quadratic penalty between the local factor matrix and the global factor matrix, and add the target perturbation. Therefore, the local optimal problem is as follows:

[0092]

[0093] Among them, Ω is the tensor observation term, They are the local factor matrices A [t] The i-th row vector of B [t] The j-th row vector of C [t] The kth row vector of λ A ,λ B ,λ C is the regularization coefficient of the factor matrix, is the transpose of the k-th row vector of the target disturbance noise matrix N;

[0094] (2.4) Initialize the local factor matrix and the global factor matrix according to the Gaussian distribution, and use the norm upper bound m to constrain the vectors in the factor matrix.

[0095] (3) Using the numbered training dataset in step (1), the elastic average stochastic gradient descent method is used for training to extract high-dimensional data features and obtain a sparse tensor joint completion model based on differential privacy, as shown in Figure 2 As shown, specifically including:

[0096] (3.1) The central server sends training tasks to each client node based on the training data set grouped in step (2);

[0097] (3.2) Each client node trains the local data through the sparse tensor joint completion model according to the training task in step (3.1) and updates the local factor matrix A [t] ,B [t] ,C [t] , specifically including the following sub-steps;

[0098] (3.2.1) Each client traverses the observation items according to the training task and updates the local factor matrix A [t] ,B [t] . Local factor matrix A [t] The i-th row vector partial derivative of for:

[0099]

[0100] Update factor matrix A [t] The calculation formula for the i-th row vector is as follows:

[0101]

[0102]

[0103] Where η is the learning rate of the local factor matrix and m is the upper bound constraint of the norm.

[0104] Local factor matrix B [t] The j-th row vector partial derivative of

[0105]

[0106] Update factor matrix B [t] The calculation formula for the j-th row vector is as follows:

[0107]

[0108]

[0109] (3.2.2) Each client traverses the observation items according to the training task and updates the local factor matrix C [t] . Local factor matrix C [t]The rules for updating the k-th row vector are as follows:

[0110]

[0111] in, is the local factor matrix C [t] The kth row vector gradient sum, The calculation formula is as follows:

[0112]

[0113] n k: is the kth row vector of the target perturbation noise matrix N, which obeys the Laplace distribution. The element n of the noise matrix N is kr The probability density distribution formula is as follows:

[0114]

[0115] Where ∈ is the privacy budget, R is the rank of the tensor, and Δ is the L1 sensitivity of the training dataset.

[0116] (3.3) After completing the training task, each client node sends the updated local factor matrix to the central server;

[0117] (3.4) The central server collects the local factor matrix B of each client [t] ,C [t] , for the global factor matrix To update:

[0118] The central server collects the local factor matrix B of T clients [t] ,C [t] , update the global factor matrix , the calculation formula is as follows:

[0119]

[0120]

[0121] in, is the learning rate of the global factor matrix.

[0122] (3.5) The central server repeats step (3.1) to continue issuing training tasks until all training set data are trained;

[0123] (4) Use the trained tensor decomposition model to complete the missing data, which includes the following sub-steps:

[0124] (4.1) The central server will train the global factor matrix Sent to each client node

[0125] (4.2) Each client is based on the local factor matrix A [t] and the global factor matrix Calculate the approximate value of the missing data and then complete the sparse tensor joint completion task based on differential privacy.

[0126] like Figure 3 The data completion result example in the embodiment of the present invention is shown in FIG.

[0127] Furthermore, the present invention also provides a sparse tensor joint completion device based on differential privacy, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the sparse tensor joint completion method based on differential privacy.

[0128] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sparse tensor joint completion method based on differential privacy, characterized by: The following steps are involved: (1) Number each client node ,According to the data dimension information and introducing the parallel tensor decomposition method, the client node data set is grouped to obtain a numbered training data set; (2) According to the completion task requirements, a loss function is designed, and the Laplace mechanism is adopted to establish a sparse tensor joint completion model based on differential privacy, initializing the global factor matrix of the central server and the local factor matrix of each client node; (3) Using the numbered training data set in step (1), the elastic average stochastic gradient descent method is used for training, high-dimensional data features are extracted, and a sparse tensor joint completion model based on differential privacy is obtained; the step (3) specifically includes: (3.1) the central server sends a training task to each client node according to the grouped training data set in step (1); (3.2) each client node trains the local data through the sparse tensor joint completion model according to the training task in step (3.1), and updates the local factor matrix ; (3.3) After each client node completes the training task, it sends the updated local factor matrix to the central server; (3.4) The central server collects the local factor matrix of each client , for the global factor matrix Update; (3.5) the central server repeats step (3.1) to continue issuing training tasks until each set of training set data is trained; the step (3.4) specifically includes: the central server collects T The local factor matrix of each client , update the global factor matrix , the calculation formula is as follows: in, is the learning rate of the global factor matrix; (4) Use the above-trained sparse tensor joint completion model to complete the missing data.

2. The sparse tensor joint completion method based on differential privacy according to claim 1, characterized in that: The step (1) specifically includes: Split the training dataset into multiple independent chunks ,and ,in is the original tensor data; is a tensor An observation item of , and ; For any observation , In terms of and ; Any independent block There is no overlap in the first and second dimensions; the non-overlapping independent blocks that meet the above conditions are defined as groups , the number of independent blocks in each group is ,Right now ; Client i The local tensor data is , the grouping rule formula is as follows: Among them, independent blocks The coordinate range is ; For each group, the independent blocks of each client do not overlap to avoid update conflicts during training.

3. The sparse tensor joint completion method based on differential privacy according to claim 1 or 2, characterized in that: The step (2) specifically includes: (2.1) The number of known client nodes is T , each client's local data is , the completed data is ; The loss function is expressed as T The sum of the client tensor decomposition loss functions is expressed as follows: (2.2) For the global optimization problem, all clients share the non-user global dimension factor matrix; the global optimal problem is as follows: in, , ; (2.3) Decompose the global optimization problem into T A local subproblem is solved by setting a quadratic penalty between the local factor matrix and the global factor matrix and adding target perturbation. The local optimal problem is as follows: in, is the tensor observation term, are local factor matrices No. i row vector, No. j row vector, No. k row vector; is the regularization coefficient of the factor matrix, is the target perturbation noise matrix The transpose of the k-th row vector of ; (2.4) Initialize the local factor matrix and the global factor matrix according to Gaussian distribution, and use the norm upper bound m to constrain the row vectors in the factor matrix.

4. The sparse tensor joint completion method based on differential privacy according to claim 1 or 2, characterized in that: The step (4) specifically includes: (4.1) The central server will train the global factor matrix Send it to each client node; (4.2) Each client uses the local factor matrix and the global factor matrix , calculate the approximate value of the missing data, and then complete the sparse tensor joint completion task based on differential privacy.

5. The sparse tensor joint completion method based on differential privacy according to claim 3, characterized in that: The step (3.2) specifically includes: (3.2.1) Each client traverses the observation items according to the training task and updates the local factor matrix ; Local factor matrix No. i Row vector partial derivatives for: Update factor matrix No. i The calculation formula for row vector is as follows: in, is the learning rate of the local factor matrix, is the upper bound constraint of the norm; Local factor matrix No. j Row vector partial derivatives The calculation formula is as follows: Update factor matrix No. j The calculation formula for row vector is as follows: (3.2.2) Each client traverses the observation items according to the training task and updates the local factor matrix ; Local factor matrix No. k The rules for updating row vectors are as follows: in, is the local factor matrix No. k The row vector gradient sum, The calculation formula is as follows: is the target perturbation noise matrix No. k row vector, the noise matrix Obeying the Laplace distribution, the noise matrix Elements The probability density distribution formula is as follows: ， in, Budget for privacy, is the rank of the tensor, For the training data set L 1Sensitivity.

6. A sparse tensor joint completion device based on differential privacy, characterized by: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the sparse tensor joint completion method based on differential privacy according to any one of claims 1 to 5.

Images